ci:ctest timing summary for MacOS#3288
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3288 +/- ##
==========================================
- Coverage 78.33% 78.16% -0.18%
==========================================
Files 693 693
Lines 123786 123783 -3
Branches 17196 17191 -5
==========================================
- Hits 96972 96749 -223
- Misses 25891 26114 +223
+ Partials 923 920 -3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
- Add Time field to testresult.Result (Chromium JSON format) - Pass duration from all_tests.go to JSON output - Enable --json-output in CMake run_tests targets - Add test_results_summary.py to parse results and emit GitHub Actions step summary with top 10 slowest tests - Wire up summary step in macOS-x86-FIPS workflow
WillChilds-Klein
left a comment
There was a problem hiding this comment.
This is just a sample; not all jobs that could emit slow test data were set up.
why not just time all tests and only show the 20 slowest in the summary?
Does this data belong in CloudWatch instead/also? Are these metrics actionable ?
i would vote just keeping it here. if you're annoyed with slow runtime on a test/PR you care about you can check the summary right from the PR. timing might fluctuate by run and over time, so adequately tuning some alarm threshold seems brittle.
justsmth
left a comment
There was a problem hiding this comment.
Looks good -- I have just one lingering "uniqueness" concern.
| fmt.Fprintf(os.Stderr, "Warning: could not create GTEST_REPORT_DIR %q: %v; skipping timing report\n", reportDir, err) | ||
| } else { | ||
| binName := filepath.Base(prog) | ||
| reportFile := filepath.Join(reportDir, fmt.Sprintf("%s_shard_%d_%d.json", binName, test.shard, atomic.AddInt64(&reportSeq, 1))) |
There was a problem hiding this comment.
Sorry -- I think the PID would still be needed for uniqueness across iterations:
reportFile := filepath.Join(reportDir, fmt.Sprintf("%s_shard_%d_%d_%d.json",
binName, test.shard, os.Getpid(), atomic.AddInt64(&reportSeq, 1)))
Description of changes:
Our macOS CI jobs run several build configurations sequentially, making it hard to identify which tests are the long poles. This PR adds per-test-case timing by capturing gtest's native JSON reports and surfaces the top 20 slowest tests in the GitHub Actions step summary for both the macOS-x86 (FIPS) and macOS-ARM jobs.
Changes:
util/all_tests.go, whenGTEST_REPORT_DIRis set, passGTEST_OUTPUT=json:<file>to each test binary so gtest emits a per-test-case timing report. Report filenames include the binary name, shard index, and PID so sequential/sharded runs don't overwrite each other.tests/ci/test_results_summary.py, which aggregates all gtest JSON reports in the directory and emits a markdown table of the slowest test cases to the step summary.GTEST_REPORT_DIRand aTest timing summarystep (if: always()) into the macOS-x86 (FIPS) and macOS-ARM jobs inactions-ci.yml.Call-outs:
Testing:
This PR; see an example Summary.

For each arch the python script creates a top-20 slowest-tests table; an in-line example:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and the ISC license.