Problems

[vagetablechicken]

Setup

• Dir /data just a readme doc, and /dataset is empty. Not good. And the way downloading data wget -r -np -R "index.html*" http://43.138.115.238/download/data/; cp -r <dataset directory> ./dataset is not good too. Use wget -r -np -R "index.html*" http://43.138.115.238/download/data/ -P ./dataset

• Give the openmldb quickstart url, users don't know how to start OpenMLDB cluster.
  

  febench/README.md

  Lines 55 to 56 in fc0d88b

  	Step 3: Start the cluster and enter the *OpenMLDB* folder

• OpenMLDB cluster settings

  • https://openmldb.ai/docs/zh/main/maintain/faq.html#java-lang-outofmemoryerror-java-heap-space default setting is not enough, set bigger
  • zk config maxClientCnxns default 60, need to set bigger?

• Give a script to convert. And mention that we should generate a new file named 'conf.properties' from 'conf.properties.template'.
  

  febench/README.md

  Line 58 in fc0d88b

  Step 4: Update the settings (the OpenMLDB cluster, data/query locations) in the configuration file (./conf/conf.properties)

  
  e.g. sed '' xx.template > xx

• Comment about the DATASET_ID will be set by compile_test.sh/test.sh.
  

  febench/OpenMLDB/conf/conf.properties.template

  Line 4 in f63a480

  DATASET_ID=4

• default thread is 1? The result will show poor performance.
  

  febench/OpenMLDB/src/src/main/java/com/_4paradigm/benchmark/OpenMLDBPerfBenchmark_Cluster.java

  Line 56 in f63a480

  @Threads(1)

Test Code

• even we do sync load data, but if failed, won't stop the whole test.
  

  febench/OpenMLDB/src/src/main/java/com/_4paradigm/benchmark/OpenMLDBPerfBenchmark_Cluster.java

  Line 208 in f63a480

  Util.executeSQL_block(loadDataSQL, executor);

• readParquetFolder is for insert, give the explaination about it, how many file we used in each dataset, and why? e.g. why dataset 0 reads 40 files, but dataset 1,2,3... only read 1 file.
  

  febench/OpenMLDB/src/src/main/java/com/_4paradigm/benchmark/OpenMLDBPerfBenchmark_Cluster.java

  Lines 511 to 512 in f63a480

  	String filePath;
  	for(int i=0; i<40; ++i){

• what's this?
  

  febench/OpenMLDB/src/src/main/java/com/_4paradigm/benchmark/OpenMLDBPerfBenchmark_Cluster.java

  Line 542 in f63a480

  int[] fs_number=new int[]{1};

error handle

• if create table throws exception, abort the test, don't just print error.
  

  febench/OpenMLDB/src/src/main/java/com/_4paradigm/benchmark/OpenMLDBPerfBenchmark_Cluster.java

  Line 103 in f63a480

  e.printStackTrace();

• check returned value, if false, there's no deployment, test can't continue. Make sure deployments exist.
  

  febench/OpenMLDB/src/src/main/java/com/_4paradigm/benchmark/OpenMLDBPerfBenchmark_Cluster.java

  Line 171 in f63a480

  Util.executeSQL("DEPLOY " + deployName + " " + builder.toString(), executor);

[vagetablechicken]

Update 23/08/07

Doc

• Step 4 怎么修改不清晰，配置项也不同步
• maven http有特殊要求，版本推荐3.8.0，3.8.1级以上可能需要特殊配置，问题在 https://stackoverflow.com/a/67002852

下载使用tar压缩包，分文件下载能下14h+
Q3等dataset必然要使用超过1g默认memory配置，文档写明测试前必须要检查spark配置，而不是错误了再修正配置。local需要改driver memory，yarn需要改execute memory

Usage

• logs目录要提前配置，否则报错，也不应该叫logs，result/report更合适，并且代码或者repo中建好目录

  febench/flink/src/src/main/java/org/example/join/BenchmarkConfig.java

  Line 89 in 5b2f7c2

  logfile = "logs/" + System.currentTimeMillis() + ".json";

• 脚本中途失败应该推出，不要继续跑，加set -ex，不要合并cmd https://github.com/decis-bench/febench/blob/91dac5f16a5c076d136b1299d118e07927775cab/OpenMLDB/compile_test.sh
• flink测试不可以配置flink集群？为什么还会有openmldb配置https://github.com/decis-bench/febench/blob/5facee2f136645e6cf2c0ac4b89bb5b0b2b51ec0/flink/conf/conf.properties.template
• 增加日志系统，不要只stdout，日志不分级非常影响调试
  • init期可能因为无db而删除db失败，打印异常很干扰，适合加入info级日志

Code Style

• 代码命名不好，java有下划线，全改为驼峰

[XuanheZhou]

Setup

• Dir /data just a readme doc, and /dataset is empty. Not good. And the way downloading data wget -r -np -R "index.html*" http://43.138.115.238/download/data/; cp -r <dataset directory> ./dataset is not good too. Use wget -r -np -R "index.html*" http://43.138.115.238/download/data/ -P ./dataset

• Give the openmldb quickstart url, users don't know how to start OpenMLDB cluster.
  

  febench/README.md

  Lines 55 to 56 in fc0d88b

  	Step 3: Start the cluster and enter the *OpenMLDB* folder

• OpenMLDB cluster settings

  • https://openmldb.ai/docs/zh/main/maintain/faq.html#java-lang-outofmemoryerror-java-heap-space default setting is not enough, set bigger
  • zk config maxClientCnxns default 60, need to set bigger?

• Give a script to convert. And mention that we should generate a new file named 'conf.properties' from 'conf.properties.template'.
  

  febench/README.md

  Line 58 in fc0d88b

  Step 4: Update the settings (the OpenMLDB cluster, data/query locations) in the configuration file (./conf/conf.properties)

  e.g. sed '' xx.template > xx

• Comment about the DATASET_ID will be set by compile_test.sh/test.sh.
  

  febench/OpenMLDB/conf/conf.properties.template

  Line 4 in f63a480

  DATASET_ID=4

• default thread is 1? The result will show poor performance.
  

  febench/OpenMLDB/src/src/main/java/com/_4paradigm/benchmark/OpenMLDBPerfBenchmark_Cluster.java

  Line 56 in f63a480

  @Threads(1)

Test Code

• even we do sync load data, but if failed, won't stop the whole test.
  

  febench/OpenMLDB/src/src/main/java/com/_4paradigm/benchmark/OpenMLDBPerfBenchmark_Cluster.java

  Line 208 in f63a480

  Util.executeSQL_block(loadDataSQL, executor);

• readParquetFolder is for insert, give the explaination about it, how many file we used in each dataset, and why? e.g. why dataset 0 reads 40 files, but dataset 1,2,3... only read 1 file.
  

  febench/OpenMLDB/src/src/main/java/com/_4paradigm/benchmark/OpenMLDBPerfBenchmark_Cluster.java

  Lines 511 to 512 in f63a480

  	String filePath;
  	for(int i=0; i<40; ++i){

• what's this?
  

  febench/OpenMLDB/src/src/main/java/com/_4paradigm/benchmark/OpenMLDBPerfBenchmark_Cluster.java

  Line 542 in f63a480

  int[] fs_number=new int[]{1};

error handle

• if create table throws exception, abort the test, don't just print error.
  

  febench/OpenMLDB/src/src/main/java/com/_4paradigm/benchmark/OpenMLDBPerfBenchmark_Cluster.java

  Line 103 in f63a480

  e.printStackTrace();

• check returned value, if false, there's no deployment, test can't continue. Make sure deployments exist.
  

  febench/OpenMLDB/src/src/main/java/com/_4paradigm/benchmark/OpenMLDBPerfBenchmark_Cluster.java

  Line 171 in f63a480

  Util.executeSQL("DEPLOY " + deployName + " " + builder.toString(), executor);

updated :)

[vagetablechicken]

docker needs update after code improved

[vagetablechicken]

febench/README.md

Line 194 in 523d304

5. Run the benchmark according to Step 5 in *<a href="#-quickstart">QuickStart</a>*.

not step 5, start from step 4(modify conf file)

sbin style for openmldb cluster deploy

导入需要注意物理内存，测试Q3 local 32会吃多少内存，以此为参考，用户物理内存少的，必须调整local核数