Skip to content

Add AIG Benchmark Suite and Regression Test Script with Non-Determinism Detection - #9

Open
Max-astro wants to merge 2 commits into
cuhk-eda:mainfrom
Max-astro:testsuite
Open

Add AIG Benchmark Suite and Regression Test Script with Non-Determinism Detection#9
Max-astro wants to merge 2 commits into
cuhk-eda:mainfrom
Max-astro:testsuite

Conversation

@Max-astro

Copy link
Copy Markdown

This PR adds AIG cases from the mockturtle project and a regression testing script (test_all.sh) for validating the correctness and determinism of the GPU-accelerated AIG optimization algorithms (e.g., resyn2rs).

During development and testing, I observed that the resyn2rs command crashes or produces non-deterministic results in some cases (running the same benchmark with the same input can yield different AND-gate counts and logic levels across runs).

I'm using CUDA 13.1 on a 5070ti GPU. The crash and ND issues will likely occur in other GPU environments as well.


Usage of the test_all.sh:

  1. Basic regression run (default resyn2rs command)
./test_all.sh

Runs all benchmarks once and reports PASS/FAIL status with AND count and level changes.

  1. Save results as a new baseline
./test_all.sh --save my_baseline.txt
  1. Compare against an existing baseline
# Allow improvements (only warn on degradation):
./test_all.sh --baseline benchmarks/resyn2rs.checkpoint

# Require exact match with baseline:
./test_all.sh --baseline benchmarks/resyn2rs.checkpoint --exact
  1. Detect non-deterministic behavior
# Run all benchmarks 2 times and compare results between runs:
./test_all.sh --deterministic-check

# Run N times for more confidence:
./test_all.sh --deterministic-check 5

This mode automatically saves baseline.1.txt, baseline.2.txt, etc., and compares each iteration against the previous one using exact matching. If any benchmark produces different results across runs, the script reports the mismatched cases and exits with a non-zero status.

  1. Test with a custom command sequence
./test_all.sh --command "rw; rf; b"

For full usage details, run:

./test_all.sh --help

Test results

./test_all.sh

Starting Benchmark Execution...
./test_all.sh: line 126: 30587 Aborted                 (core dumped) $EXEC -c "read $full_path; ps; $TEST_CMD; time; ps" > "$log_file" 2>&1
c17 FAILED
./test_all.sh: line 126: 30895 Segmentation fault      (core dumped) $EXEC -c "read $full_path; ps; $TEST_CMD; time; ps" > "$log_file" 2>&1
c432 FAILED
./test_all.sh: line 126: 31202 Segmentation fault      (core dumped) $EXEC -c "read $full_path; ps; $TEST_CMD; time; ps" > "$log_file" 2>&1
int2float FAILED
./test_all.sh: line 126: 31506 Segmentation fault      (core dumped) $EXEC -c "read $full_path; ps; $TEST_CMD; time; ps" > "$log_file" 2>&1
ctrl FAILED
./test_all.sh: line 126: 31822 Segmentation fault      (core dumped) $EXEC -c "read $full_path; ps; $TEST_CMD; time; ps" > "$log_file" 2>&1
c1908 FAILED
./test_all.sh: line 126: 32127 Segmentation fault      (core dumped) $EXEC -c "read $full_path; ps; $TEST_CMD; time; ps" > "$log_file" 2>&1
c880 FAILED
./test_all.sh: line 126: 32466 Aborted                 (core dumped) $EXEC -c "read $full_path; ps; $TEST_CMD; time; ps" > "$log_file" 2>&1
c499 FAILED

...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant