Skip to content

Add the answer check and the report generator - #13

Merged
tamnd merged 1 commit into
mainfrom
report
Oct 7, 2026
Merged

tamnd merged 1 commit into
mainfrom
report

Conversation

@tamnd

@tamnd tamnd commented Oct 7, 2026

Copy link
Copy Markdown
Owner

Part of #1.

What changes

  • src/answers.rs holds the answer set format and the answer check of spec/20 section 20.5. An answer set has one <query>.tsv file for each query: a header line, then the rows in the text format of PostgreSQL COPY, with \N for NULL. The check compares two answers as multisets of rows. It sorts both by all columns, with numbers sorted by value. Floats match within a relative 1e-9.
  • The check also reads the files of the TPC-H answer set (dbgen/answers/q*.out) and applies TPC-H clause 2.1.3.5. Column values and counts must be equal. Sums may differ by 100. Averages may differ by 1 percent after rounding to 2 decimals. Ratios of sums (Q8, Q14, Q17) must pass both rules. sum(l_quantity) (Q1, Q18) must be equal (comment 4 of the clause). The column name in the answer file gives the kind of value.
  • rupg-bench answers --expected PATH --actual PATH compares a file or a directory and prints one line for each query. A wrong answer makes the command fail.
  • src/report.rs is the report generator. It writes reports/<date>/<commit>-<machine>-<suite>.json and a Markdown file from the JSON file only. A smoke run gets -smoke in its name and a line at the top that says that its numbers are not baselines. Text that the style check does not allow in Markdown goes into a code block or is escaped.
  • rupg-bench report --result FILE writes the two files from the --json output of a command. --suite, --machine, --commit and --smoke fill the fields that the file does not have.
  • src/json.rs gets a parser with a depth limit, and the writer now escapes every character that is not ASCII.
  • The README documents both commands.

Test

srvtest passed on server3: clippy is clean and 34 tests pass.

Answer check on server3, 7 October 2026. This was a test of the checker and not a baseline. I generated TPC-H SF1 in memory with the tpch extension of DuckDB 1.5.6. Then I wrote the 22 answers with COPY (<query>) TO 'actual/qN.tsv' (FORMAT csv, DELIMITER '\t', HEADER true, NULLSTR '\N', QUOTE '', ESCAPE ''). The query texts came from tpch_queries(). Then I compared them to the 22 answer files of duckdb/duckdb-tpch at 0743d9b3.

rupg-bench answers --expected expected --actual actual
q1       ok, 4 rows
q10      WRONG: sorted row 1 column 6 (c_address): got "rsR9lRxyTdHbDOVt8nYbwjK5vAWH9sB", expected "Az9RFaut7NkPnc5zSD2PwHgVwr4jRzq"
q13      WRONG: sorted row 1 column 2 (custdist): got "50004", expected "50005"
q15      WRONG: sorted row 1 column 3 (s_address): got "5BXWsJERA2mP5OyO4", expected "Wp34zim9qYFbVctdW"
q2       WRONG: sorted row 1 column 6 (s_address): got "Z1,hkHIw,Z3,,Comv6kLxIiPJtoNt", expected "2Z0JGkiv01Y00oCFwUGfviIbhzCdy"
q20      WRONG: sorted row 1 column 2 (s_address): got "JtPqm19E7tF 152Rl1wQZ8j0H", expected "iybAE,RmTymrZVYaFZva2SH,j"
rupg-bench: 5 of 22 answers are wrong

The other 17 queries pass. The 5 failures are real differences in the data. The built-in generator of DuckDB makes other address strings and other order comments than dbgen 3.0.1. Its own answer table, tpch_answers(), also gives 50004 for the first row of Q13, where the official set gives 50005. So the TPC-H driver must load the data of the pinned dbgen and not the data of a built-in generator. The first try also showed that the CSV writer of DuckDB quotes values that contain #. That is why the export uses QUOTE ''.

Report on server3, 7 October 2026. rupg-bench measure --unit cron.service --idle --json measured 10.0 s with 100 samples. Then rupg-bench report --result idle-cron.json --suite idle-base --machine server3 --smoke wrote 2026-10-07/236a8e18-server3-idle-base-smoke.md and .json. check.py reports 0 issues on both files.

@tamnd tamnd added kind/feature It does not do something that it must do. area/report Per-query tables, gate evaluation and the transport column. labels Oct 7, 2026
@tamnd tamnd self-assigned this Oct 7, 2026
@tamnd
tamnd merged commit 5e501c0 into main Oct 7, 2026
2 checks passed
@tamnd
tamnd deleted the report branch October 7, 2026 08:53
@tamnd tamnd mentioned this pull request Oct 7, 2026
5 of 17 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/report Per-query tables, gate evaluation and the transport column. kind/feature It does not do something that it must do.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant