What happened
On PR #3259, the pingidentity workspace had backstage.json set to 1.54.5 and source.json repo-backstage-version set to 1.54.5, but both metadata files used supportedVersions: 1.54.4 and OCI tags with bs_1.54.4. The review agent caught this in 2 of 3 runs on the same code state (Sep 3 runs 8 and 10 flagged it; run 9 approved without mention). The inconsistency was merged into main at 18:18 UTC.
What could go better
The review agent has no explicit instruction in AGENTS.md to cross-check version fields within a workspace. Whether the backstage.json-vs-metadata mismatch gets flagged depends on whether the LLM happens to notice it during a given run, explaining the 67% detection rate. The AGENTS.md Package YAML section lists key fields (spec.packageName, spec.dynamicArtifact, spec.version, spec.backstage.role, spec.support, spec.appConfigExamples) but omits spec.backstage.supportedVersions entirely. The /override-backstage documentation says it "creates backstage.json and rewrites metadata OCI tags" but does not state that these must stay in sync when modified manually. This is distinct from #3445 (which covers backstage.json vs the repo-level versions.json, a different pair) and #2970 (which covers the CI /publish check catching mismatches, not review-time guidance). Confidence: high — the gap is directly observable in AGENTS.md and the agent's inconsistent behavior correlates with the missing guidance.
Proposed change
Add a version-consistency cross-check rule to AGENTS.md, either in the existing "Working with Workspaces" section or a new "Review Checklist" subsection. The guidance should state:
When a PR modifies any version-carrying file in a workspace (backstage.json, source.json, or metadata/*.yaml), verify that these four values agree within the workspace:
backstage.json version (if the file exists — it is an optional override)source.json repo-backstage-versionmetadata/*.yaml spec.backstage.supportedVersions- The
bs_ prefix in metadata/*.yaml spec.dynamicArtifact OCI tags (e.g., bs_1.54.4__0.4.0)
A mismatch indicates either a partially applied /override-backstage or a manual editing error. Also add spec.backstage.supportedVersions to the Package YAML key fields list, which currently omits it.
Validation criteria
On the next 3 PRs that modify workspace version files (backstage.json, source.json, or metadata version/OCI fields), the review agent should consistently flag any per-workspace version mismatch in every run, rather than the 67% rate (2 of 3 runs) observed on PR #3259.
Generated by retro agent from #3259
What happened
On PR #3259, the
pingidentityworkspace hadbackstage.jsonset to1.54.5andsource.jsonrepo-backstage-versionset to1.54.5, but both metadata files usedsupportedVersions: 1.54.4and OCI tags withbs_1.54.4. The review agent caught this in 2 of 3 runs on the same code state (Sep 3 runs 8 and 10 flagged it; run 9 approved without mention). The inconsistency was merged into main at 18:18 UTC.What could go better
The review agent has no explicit instruction in AGENTS.md to cross-check version fields within a workspace. Whether the backstage.json-vs-metadata mismatch gets flagged depends on whether the LLM happens to notice it during a given run, explaining the 67% detection rate. The AGENTS.md Package YAML section lists key fields (
spec.packageName,spec.dynamicArtifact,spec.version,spec.backstage.role,spec.support,spec.appConfigExamples) but omitsspec.backstage.supportedVersionsentirely. The/override-backstagedocumentation says it "creates backstage.json and rewrites metadata OCI tags" but does not state that these must stay in sync when modified manually. This is distinct from #3445 (which covers backstage.json vs the repo-levelversions.json, a different pair) and #2970 (which covers the CI/publishcheck catching mismatches, not review-time guidance). Confidence: high — the gap is directly observable in AGENTS.md and the agent's inconsistent behavior correlates with the missing guidance.Proposed change
Add a version-consistency cross-check rule to AGENTS.md, either in the existing "Working with Workspaces" section or a new "Review Checklist" subsection. The guidance should state:
When a PR modifies any version-carrying file in a workspace (
backstage.json,source.json, ormetadata/*.yaml), verify that these four values agree within the workspace:backstage.jsonversion(if the file exists — it is an optional override)source.jsonrepo-backstage-versionmetadata/*.yamlspec.backstage.supportedVersionsbs_prefix inmetadata/*.yamlspec.dynamicArtifactOCI tags (e.g.,bs_1.54.4__0.4.0)A mismatch indicates either a partially applied
/override-backstageor a manual editing error. Also addspec.backstage.supportedVersionsto the Package YAML key fields list, which currently omits it.Validation criteria
On the next 3 PRs that modify workspace version files (backstage.json, source.json, or metadata version/OCI fields), the review agent should consistently flag any per-workspace version mismatch in every run, rather than the 67% rate (2 of 3 runs) observed on PR #3259.
Generated by retro agent from #3259