Skip to content

ci:ctest timing summary for MacOS#3288

Open
dougch wants to merge 9 commits into
aws:mainfrom
dougch:ci/ctest-timing-summary
Open

ci:ctest timing summary for MacOS#3288
dougch wants to merge 9 commits into
aws:mainfrom
dougch:ci/ctest-timing-summary

Conversation

@dougch

@dougch dougch commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Description of changes:

Our macOS CI jobs run several build configurations sequentially, making it hard to identify which tests are the long poles. This PR adds per-test-case timing by capturing gtest's native JSON reports and surfaces the top 20 slowest tests in the GitHub Actions step summary for both the macOS-x86 (FIPS) and macOS-ARM jobs.

Changes:

  • In util/all_tests.go, when GTEST_REPORT_DIR is set, pass GTEST_OUTPUT=json:<file> to each test binary so gtest emits a per-test-case timing report. Report filenames include the binary name, shard index, and PID so sequential/sharded runs don't overwrite each other.
  • Add tests/ci/test_results_summary.py, which aggregates all gtest JSON reports in the directory and emits a markdown table of the slowest test cases to the step summary.
  • Wire GTEST_REPORT_DIR and a Test timing summary step (if: always()) into the macOS-x86 (FIPS) and macOS-ARM jobs in actions-ci.yml.

Call-outs:

  1. This is just a sample; not all jobs that could emit slow test data were set up.
  2. With the size of these workflows, this table format might not scale/be readable — 20 entries that are ~1 sec apart might not be that useful.
  3. Does this data belong in CloudWatch instead/also? Are these metrics actionable ?

Testing:

This PR; see an example Summary.
For each arch the python script creates a top-20 slowest-tests table; an in-line example:
Screenshot 2026-06-09 at 12 10 28

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and the ISC license.

@dougch dougch temporarily deployed to auto-approve June 4, 2026 22:26 — with GitHub Actions Inactive
@dougch dougch changed the title ci/ctest timing summary ci:ctest timing summary Jun 4, 2026
@dougch dougch changed the title ci:ctest timing summary ci:ctest timing summary for MacOS Jun 4, 2026
@dougch dougch temporarily deployed to auto-approve June 4, 2026 22:33 — with GitHub Actions Inactive
@dougch dougch temporarily deployed to auto-approve June 4, 2026 22:33 — with GitHub Actions Inactive
@dougch dougch temporarily deployed to auto-approve June 4, 2026 22:33 — with GitHub Actions Inactive
@dougch dougch temporarily deployed to auto-approve June 4, 2026 22:33 — with GitHub Actions Inactive
@dougch dougch temporarily deployed to auto-approve June 4, 2026 22:33 — with GitHub Actions Inactive
@dougch dougch temporarily deployed to auto-approve June 4, 2026 22:33 — with GitHub Actions Inactive
@codecov-commenter

codecov-commenter commented Jun 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.16%. Comparing base (84b74e0) to head (9a00d43).
⚠️ Report is 14 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3288      +/-   ##
==========================================
- Coverage   78.33%   78.16%   -0.18%     
==========================================
  Files         693      693              
  Lines      123786   123783       -3     
  Branches    17196    17191       -5     
==========================================
- Hits        96972    96749     -223     
- Misses      25891    26114     +223     
+ Partials      923      920       -3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@dougch dougch temporarily deployed to auto-approve June 4, 2026 23:20 — with GitHub Actions Inactive
@dougch dougch temporarily deployed to auto-approve June 4, 2026 23:20 — with GitHub Actions Inactive
@dougch dougch temporarily deployed to auto-approve June 4, 2026 23:20 — with GitHub Actions Inactive
@dougch dougch temporarily deployed to auto-approve June 4, 2026 23:20 — with GitHub Actions Inactive
@dougch dougch temporarily deployed to auto-approve June 4, 2026 23:20 — with GitHub Actions Inactive
@dougch dougch temporarily deployed to auto-approve June 4, 2026 23:20 — with GitHub Actions Inactive
@dougch dougch temporarily deployed to auto-approve June 4, 2026 23:20 — with GitHub Actions Inactive
@dougch dougch marked this pull request as ready for review June 5, 2026 16:45
@dougch dougch requested a review from a team as a code owner June 5, 2026 16:45
@dougch dougch requested a review from justsmth June 5, 2026 16:46
@dougch dougch marked this pull request as draft June 8, 2026 23:01
@dougch dougch removed the request for review from justsmth June 8, 2026 23:02
- Add Time field to testresult.Result (Chromium JSON format)
- Pass duration from all_tests.go to JSON output
- Enable --json-output in CMake run_tests targets
- Add test_results_summary.py to parse results and emit
  GitHub Actions step summary with top 10 slowest tests
- Wire up summary step in macOS-x86-FIPS workflow
@dougch dougch temporarily deployed to auto-approve June 9, 2026 17:34 — with GitHub Actions Inactive
@dougch dougch temporarily deployed to auto-approve June 9, 2026 17:34 — with GitHub Actions Inactive
@dougch dougch temporarily deployed to auto-approve June 9, 2026 17:34 — with GitHub Actions Inactive
@dougch dougch temporarily deployed to auto-approve June 9, 2026 17:34 — with GitHub Actions Inactive
@dougch dougch temporarily deployed to auto-approve June 9, 2026 17:34 — with GitHub Actions Inactive
@dougch dougch temporarily deployed to auto-approve June 9, 2026 17:34 — with GitHub Actions Inactive
@dougch dougch marked this pull request as ready for review June 10, 2026 17:47
Comment thread util/all_tests.go Outdated
Comment thread tests/ci/test_results_summary.py Outdated
Comment thread util/all_tests.go Outdated

@WillChilds-Klein WillChilds-Klein left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just a sample; not all jobs that could emit slow test data were set up.

why not just time all tests and only show the 20 slowest in the summary?

Does this data belong in CloudWatch instead/also? Are these metrics actionable ?

i would vote just keeping it here. if you're annoyed with slow runtime on a test/PR you care about you can check the summary right from the PR. timing might fluctuate by run and over time, so adequately tuning some alarm threshold seems brittle.

Comment thread .github/workflows/actions-ci.yml Outdated

@justsmth justsmth left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good -- I have just one lingering "uniqueness" concern.

Comment thread util/all_tests.go
fmt.Fprintf(os.Stderr, "Warning: could not create GTEST_REPORT_DIR %q: %v; skipping timing report\n", reportDir, err)
} else {
binName := filepath.Base(prog)
reportFile := filepath.Join(reportDir, fmt.Sprintf("%s_shard_%d_%d.json", binName, test.shard, atomic.AddInt64(&reportSeq, 1)))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry -- I think the PID would still be needed for uniqueness across iterations:

reportFile := filepath.Join(reportDir, fmt.Sprintf("%s_shard_%d_%d_%d.json",
    binName, test.shard, os.Getpid(), atomic.AddInt64(&reportSeq, 1)))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type:ci Continuous Integration related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants