Skip to content

[dev-tool] Surface actual failures in extract-api and test runner#39238

Open
jeremymeng wants to merge 4 commits into
mainfrom
jeremymeng-verify-arm-alertrulerec-api-report
Open

[dev-tool] Surface actual failures in extract-api and test runner#39238
jeremymeng wants to merge 4 commits into
mainfrom
jeremymeng-verify-arm-alertrulerec-api-report

Conversation

@jeremymeng

Copy link
Copy Markdown
Member

Problem

Several dev-tool commands — most notably extract-api — could exit non-zero with no actionable message. Failures surfaced only as the generic Errors occurred. See the output above. (exit 1) or [Internal Error] undefined, which made CI Build libraries failures very hard to diagnose (an extract-api step failing with exit code 1 and no clear error).

This PR makes dev-tool surface the real underlying failure in three spots.

Changes

Gap 1 — extract-api ignored api-extractor's result

commands/run/extract-api.ts previously called extractApi(...) and discarded its return value, so a genuine api-extractor failure only manifested later as a bare ENOENT when reading the (never-written) temp report, and the exports branch kept success = true. Now:

  • extractApi returns { succeeded, errorCount, warningCount }.
  • extractApiForEntry throws with the entry name/runtime, mainEntryPointFilePath, error/warning counts, and the expected report path before the downstream readFile.
  • The non-exports branch folds .succeeded into success.

Gap 4 — top-level catch mishandled non-Error throws

index.ts assumed err was an Error (err.message / console.trace(err.stack)), yielding [Internal Error] undefined for non-Error rejections and printing the callback's own stack rather than the error's. Now a format(value) helper handles Error → stack ?? message, object → JSON.stringify (guarded), else String, prints err.cause when present, and drops the misleading console.trace. Still exits 255.

Gap 5 — vitest runner's concurrently rejection is not an Error

concurrently([...]).result rejects with an array of close-event objects, not an Error, so a failing test run propagated to the top-level catch as [Internal Error] undefined. Added summarizeCloseEvents (in util/testUtils.ts) that extracts the non-zero exitCode events into a readable message (e.g. vitest exited with code 1). Both concurrently(...).result await sites (testVitest.ts non-proxy path and runTestsWithProxyTool) now catch, log an actionable message, and return false (clean exit 1 with a message).

Note for reviewers

The extract-api genuine-failure path now surfaces as exit 255 (a thrown Error routed through the top-level catch) rather than the previous message-less exit 1. This is intentional — the throw carries the actionable context.

Verification

  • pnpm build -F @azure/dev-tool → success; tsc --noEmit in common/tools/dev-tool → clean.
  • Manually forced each failure path against sdk/alertrulerecommendations/arm-alertrulerecommendations and confirmed clear output:
    • Extractor reports errors (temporarily escalating ae-forgotten-export to error): API Extractor failed for entry 'node' (mainEntryPoint: ...\dist\esm\index.d.ts) with 1 errors and 1 warnings. Expected report at ...\review\arm-alertrulerecommendations-node.api.md. → exit 255.
    • Missing dist (prepare throw): the real The "mainEntryPointFilePath" path does not exist: ... error with a proper stack → exit 255.
    • Corrupted .d.ts (internal extractor crash): full real stack under [Internal Error] → exit 255.

Previously several dev-tool commands could exit non-zero with no
actionable message, making CI "Build libraries" failures hard to
diagnose. This surfaces the real errors:

- extract-api ignored api-extractor's result. Now captures
  {succeeded, errorCount, warningCount}, throws with the entry name,
  mainEntryPoint, counts and expected report path before the downstream
  ENOENT, and folds failure into `success`.
- Top-level catch now handles non-Error throws (Error -> stack,
  object -> JSON, else -> String), prints err.cause, and removes the
  misleading console.trace that printed the callback's own stack.
- The vitest runner's concurrently rejection is an array of close-event
  objects, not an Error. Now summarized via summarizeCloseEvents and
  returned as a clean exit-1 with a message.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 10, 2026 18:08
@jeremymeng jeremymeng requested a review from a team as a code owner July 10, 2026 18:08
@github-actions github-actions Bot added the dev-tool Issues related to the Azure SDK for JS dev-tool label Jul 10, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Improves dev-tool diagnostics by surfacing underlying API extraction and test-runner failures.

Changes:

  • Propagates API Extractor results and detailed failures.
  • Formats non-Error top-level rejections and causes.
  • Summarizes failed Vitest subprocesses and returns failure status.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
common/tools/dev-tool/src/util/testUtils.ts Summarizes subprocess failures and preserves proxy cleanup.
common/tools/dev-tool/src/index.ts Formats top-level errors and causes.
common/tools/dev-tool/src/commands/run/testVitest.ts Reports non-proxy Vitest failures cleanly.
common/tools/dev-tool/src/commands/run/extract-api.ts Validates and propagates API Extractor results.

Comment thread common/tools/dev-tool/src/index.ts Outdated
Comment thread common/tools/dev-tool/src/util/testUtils.ts Outdated
Intermittent Windows nightly failures showed extract-api crashing with no
message and no turbo summary, dying at the final merged-API writeFile in
buildMergedApiJson. Wrap the FS operations in writeRuntimeApiFiles and
buildMergedApiJson with a transient-error retry helper (EBUSY/EPERM/ENOENT/
EMFILE/EAGAIN/UNKNOWN), and guard the two call sites so a post-retry failure
sets success = false and logs a file-qualified error. This lets extract-api
exit non-zero gracefully so turbo can attribute the failing task, and lets
most transient flakes self-heal.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
jeremymeng and others added 2 commits July 10, 2026 13:00
- index.ts: gate the error-cause log on `!== undefined` so valid falsy
  causes (0, false, "", null) are still surfaced.
- testUtils.ts: concurrently CloseEvent.exitCode is string|number where a
  string is the terminating signal; type it accordingly and label signal
  terminations distinctly from numeric exit codes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dev-tool Issues related to the Azure SDK for JS dev-tool

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants