feat: populate assessment plan and step identity in evaluation logs by hbraswelrh · Pull Request #579 · complytime/complyctl

hbraswelrh · 2026-06-11T14:50:21Z

Description

The evaluation log YAML emitted by complyctl scan had two identity gaps after PR #575 fixed the structural step closure bug:

The plan field on gemara.AssessmentLog was never populated, breaking the requirement-to-plan traceability chain
Steps serialized as Go function pointer names (github.com/complytime/complyctl/internal/output.providerStepsToGemara.providerStepToGemara.func1)instead of meaningful identifiers linking back to the checks that were run.

Notes

The step summary included the Go function pointer names rather than the identifiers from the .rego files in the complypack complytime-mapping.json associate the gemara requirement-ids (in the Policy assessment plan) to its .rego namespace id equivalent (i.e., the assessment plan testable condition related to its actual testable .rego file executable checks). By pointing to the correct step execution link they can be accessed by their sha256 once updated in subsequent versions.

evaluations:
- name: force-push-protection
  result: Passed
  message: 1 of 1 repositories passed
  control:
    reference-id: policies/test-branch-protection
    entry-id: force-push-protection
  assessment-logs:
  - requirement:
      reference-id: policies/test-branch-protection
      entry-id: block-force-push
    description: 1 of 1 repositories passed
    result: Passed
    message: 1 of 1 repositories passed
    applicability:
    - default
    steps:
    - github.com/complytime/complyctl/internal/output.providerStepsToGemara.providerStepToGemara.func1 # using the pointer here
    steps-executed: 1
    start: "2026-06-10T01:44:46Z"
    confidence-level: High

Changes

Add reqToPlan and complypackRef fields to Evaluator struct. Thread plan ID reverse map and complypack OCI ref from scan pipeline through processScanOutput/buildEvaluators into NewEvaluator
Populate Plan (*EntryMapping) in providerToGemaraAssessment using reqToPlan lookup from the dependency graph
Define shadow structs mirroring gemara.EvaluationLog hierarchy with Steps []string instead of Steps []AssessmentStep (function type)
Add formatStepIdentity() to produce '{oci-ref}#{step-name}' format with bare step name fallback when no complypack is configured
Store step names parallel to closures in stepNames map keyed by AssessmentLog pointer; toSerializable() converts to shadow struct
Update Write() to marshal shadow struct instead of gemara struct
Add reverseMap() with collision warning logging
Add resolveComplypackRef() with debug logging on cache errors

Unchanged

proto changes
no go-gemara changes

Note: Provider-side Step.Name population is tracked separately in complytime-providers.

Related Issues

Closes #578
Supersedes #573

Review Hints

Testing Steps
Prerequisites: A GitHub Codespace from the opsx/eval-log-plan-step-identity branch on hbraswelrh/complyctl.

Rebuild complyctl

cd /workspaces/complyctl
make build

Rebuild OPA provider with synthetic step fix

PROVIDERS_TMP=$(mktemp -d)
git clone --depth 1 -b opsx/fix-synthetic-step-names \
    https://github.com/hbraswelrh/complytime-providers.git \
    "$PROVIDERS_TMP/complytime-providers"
make -C "$PROVIDERS_TMP/complytime-providers" build
cp -f "$PROVIDERS_TMP/complytime-providers/bin/complyctl-provider-opa" \
    ~/.complytime/providers/
rm -rf "$PROVIDERS_TMP"

Clear stale cache and resync

cd ~/test-workspace
rm -f ~/.complytime/state.json
rm -rf .complytime/scan .complytime/opa
complyctl get

Run scan and inspect the evaluation log

complyctl scan --policy-id test-opa-bp
cat .complytime/scan/evaluation-log-policies-test-opa-policy-*.yaml

Verify the output
Check each assessment-logs entry in the YAML for:

plan: field is present with reference-id and entry-id (was previously missing)
steps: array contains a string like "complypacks/test-opa-complypack@sha256:...#test-deployment.yaml" (was previously steps: [] or a Go function pointer name like ...providerStepToGemara.func1)
Example of correct output:

assessment-logs:
- requirement:
    reference-id: policies/test-opa-policy
    entry-id: check-run-as-nonroot
  plan:
    reference-id: policies/test-opa-policy
    entry-id: check-run-as-nonroot
  steps:
  - "complypacks/test-opa-complypack@sha256:...#test-deployment.yaml"
  steps-executed: 1
  confidence-level: High

Note: Step 2 requires the companion PR hbraswelrh/complytime-providers#opsx/fix-synthetic-step-names (https://github.com/hbraswelrh/complytime-providers/compare/opsx/fix-synthetic-step-names) which populates steps on synthetic passing assessments in the OPA provider. Without it, steps will be steps: [] for requirements that passed all checks.

Assisted-by: Claude (Anthropic, Claude Opus 4.6)

hbraswelrh · 2026-06-11T17:19:42Z

@jpower432 @gvauter Testing result following the Review Hints in the PR description

Compliance Scan Report: policies/test-opa-policy

Generated: 2026-06-11T17:17:34Z

Control: container-resource-limits

Result: Passed
Message: 1 of 1 targets passed

check-resource-limits

Confidence: High
Result: Passed
Message: 1 of 1 targets passed
Steps Executed: 1

Control: container-run-as-nonroot

Result: Passed
Message: 1 of 1 targets passed

check-run-as-nonroot

Confidence: High
Result: Passed
Message: 1 of 1 targets passed
Steps Executed: 1

Evaluation Log

metadata:
  id: policies/test-opa-policy
  type: EvaluationLog
  gemara-version: v1.0.0
  description: Compliance scan evaluation log
  author:
    id: complytime
    name: complytime
    type: Software
    uri: https://github.com/complytime/complyctl
result: Passed
evaluations:
- name: container-resource-limits
  result: Passed
  message: 1 of 1 targets passed
  control:
    reference-id: policies/test-opa-policy
    entry-id: container-resource-limits
  assessment-logs:
  - requirement:
      reference-id: policies/test-opa-policy
      entry-id: check-resource-limits
    plan:
      reference-id: policies/test-opa-policy
      entry-id: check-resource-limits
    description: 1 of 1 targets passed
    result: Passed
    message: 1 of 1 targets passed
    applicability:
    - default
    steps:
    - "complypacks/test-opa-complypack@sha256:18260d6f99bf60dd7fba9c68fe8b1a0fd28c6db2dd44423f42511262b5e21f65#test-deployment.yaml"
    steps-executed: 1
    start: "2026-06-11T17:17:34Z"
    confidence-level: High
- name: container-run-as-nonroot
  result: Passed
  message: 1 of 1 targets passed
  control:
    reference-id: policies/test-opa-policy
    entry-id: container-run-as-nonroot
  assessment-logs:
  - requirement:
      reference-id: policies/test-opa-policy
      entry-id: check-run-as-nonroot
    plan:
      reference-id: policies/test-opa-policy
      entry-id: check-run-as-nonroot
    description: 1 of 1 targets passed
    result: Passed
    message: 1 of 1 targets passed
    applicability:
    - default
    steps:
    - "complypacks/test-opa-complypack@sha256:18260d6f99bf60dd7fba9c68fe8b1a0fd28c6db2dd44423f42511262b5e21f65#test-deployment.yaml"
    steps-executed: 1
    start: "2026-06-11T17:17:34Z"
    confidence-level: High
target:
  id: test-k8s-deployment
  name: test-k8s-deployment
  type: Software

gvauter

DevPod E2E verified: full OPA pipeline (get → generate → scan) produces correct evaluation log output with plan: field populated and steps: containing digest-pinned complypack identity strings. All CI checks pass.

One minor consistency note flagged inline — non-blocking.

This review was generated by /review-pr (AI-assisted).

hbraswelrh · 2026-06-12T13:14:09Z

@jpower432 @gvauter see the output here with the feedback addressed.

Compliance Scan Report: policies/test-opa-policy

Generated: 2026-06-12T13:12:47Z

Control: container-resource-limits

Result: Passed
Message: 1 of 1 targets passed

check-resource-limits

Confidence: High
Result: Passed
Message: 1 of 1 targets passed
Steps Executed: 1

Control: container-run-as-nonroot

Result: Passed
Message: 1 of 1 targets passed

check-run-as-nonroot

Confidence: High
Result: Passed
Message: 1 of 1 targets passed
Steps Executed: 1

Evaluation Log

metadata:
  id: policies/test-opa-policy
  type: EvaluationLog
  gemara-version: v1.0.0
  description: Compliance scan evaluation log
  author:
    id: complytime
    name: complytime
    type: Software
    uri: https://github.com/complytime/complyctl
result: Passed
evaluations:
- name: container-resource-limits
  result: Passed
  message: 1 of 1 targets passed
  control:
    reference-id: policies/test-opa-policy
    entry-id: container-resource-limits
  assessment-logs:
  - requirement:
      reference-id: policies/test-opa-policy
      entry-id: check-resource-limits
    plan:
      reference-id: policies/test-opa-policy
      entry-id: check-resource-limits
    description: 1 of 1 targets passed
    result: Passed
    message: 1 of 1 targets passed
    applicability:
    - default
    steps:
    - "complypacks/test-opa-complypack@sha256:18260d6f99bf60dd7fba9c68fe8b1a0fd28c6db2dd44423f42511262b5e21f65#test-deployment.yaml"
    steps-executed: 1
    start: "2026-06-12T13:12:47Z"
    confidence-level: High
- name: container-run-as-nonroot
  result: Passed
  message: 1 of 1 targets passed
  control:
    reference-id: policies/test-opa-policy
    entry-id: container-run-as-nonroot
  assessment-logs:
  - requirement:
      reference-id: policies/test-opa-policy
      entry-id: check-run-as-nonroot
    plan:
      reference-id: policies/test-opa-policy
      entry-id: check-run-as-nonroot
    description: 1 of 1 targets passed
    result: Passed
    message: 1 of 1 targets passed
    applicability:
    - default
    steps:
    - "complypacks/test-opa-complypack@sha256:18260d6f99bf60dd7fba9c68fe8b1a0fd28c6db2dd44423f42511262b5e21f65#test-deployment.yaml"
    steps-executed: 1
    start: "2026-06-12T13:12:47Z"
    confidence-level: High
target:
  id: test-k8s-deployment
  name: test-k8s-deployment
  type: Software

gvauter

LGTM

marcusburghardt

Well-structured PR with thorough test coverage, complete OpenSpec artifacts, and clean CI. The shadow struct approach for step identity serialization is well-motivated and documented in the design doc. Two items worth noting below.

Commit trailers (MEDIUM): Commit c0fde116 is missing the Signed-off-by trailer required by the constitution. Commit 73853dd6 has a name mismatch between author ("Hannah Braswell") and sign-off ("Heather Braswell"). These can be fixed with an interactive rebase.

This review was generated by /review-pr (AI-assisted).

hbraswelrh · 2026-06-12T14:38:47Z

Well-structured PR with thorough test coverage, complete OpenSpec artifacts, and clean CI. The shadow struct approach for step identity serialization is well-motivated and documented in the design doc. Two items worth noting below.

Commit trailers (MEDIUM): Commit c0fde116 is missing the Signed-off-by trailer required by the constitution. Commit 73853dd6 has a name mismatch between author ("Hannah Braswell") and sign-off ("Heather Braswell"). These can be fixed with an interactive rebase.

This review was generated by /review-pr (AI-assisted).

Thanks for this catch. I will fix this immediately.

marcusburghardt

Commit trailer issues from the prior review round are resolved. All GitHub Actions CI checks pass. Two MEDIUM items remain as follow-up suggestions (swapped warning labels in reverseMap, growing parameter count on processScanOutput) -- neither affects correctness.

This review was generated by /review-pr (AI-assisted).

gvauter

All 5 sub-issues from the spec are addressed with comprehensive test coverage.
The resolvedMappings struct cleanly prevents parameter-transposition bugs.
Both prior reviewer concerns (gvauter, jpower432) are resolved in the code.
All CI checks pass.

This review was generated by /review-pr (AI-assisted).

The scan command builds gemara.EvaluationLog from provider.ScanResponse, but the conversion from provider.Step (a data struct) to gemara.AssessmentStep (a function type) was missing. The evaluation log YAML always showed steps: [] regardless of what providers sent. Add providerStepToGemara() to wrap each provider.Step into a gemara.AssessmentStep closure that returns the step's pre-computed result, message, and the assessment-level confidence. Add providerStepsToGemara() to convert a slice of provider steps. Update providerToGemaraAssessment() to populate the Steps field on the returned gemara.AssessmentLog. Signed-off-by: Hannah Braswell <hbraswel@redhat.com>

The evaluation log YAML emitted by complyctl scan had two identity gaps after PR complytime#575 fixed the structural step closure bug: 1. The plan field on gemara.AssessmentLog was never populated, breaking the requirement-to-plan traceability chain 2. Steps serialized as Go function pointer names instead of meaningful identifiers linking back to the checks that were run No proto changes, no go-gemara changes. Provider-side Step.Name population is tracked separately in complytime-providers. Closes: complytime#578 Assisted-by: Claude (Anthropic, Claude Opus 4.6) Signed-off-by: Hannah Braswell <hbraswel@redhat.com>

Signed-off-by: Hannah Braswell <hbraswel@redhat.com>

Addresses review feedback from jpower432: the previous resolveComplypackRef() re-derived the OCI reference from cache state and assumed one complypack per evaluator. Replace with per-evaluator ref map threaded from config+state. Changes: - Replace resolveComplypackRef() with buildComplypackRefs() returning map[string]string (evaluatorID -> repo@digest) built from state.Complypacks - Add extractReqToEvaluator() to map requirementID -> evaluatorID from evaluator groups, enabling per-assessment complypack ref resolution - Thread both maps through executeScanPhase -> runScanAndReport -> processScanOutput -> buildEvaluators -> NewEvaluator - Change Evaluator.complypackRef string to complypackRefs map + reqToEvaluator map with resolveComplypackRef(requirementID) method for per-assessment lookup - Update toSerializable() to resolve complypack ref per assessment log This correctly handles multiple evaluators with different complypacks per policy without re-reading cache state. Signed-off-by: Hannah Braswell <hbraswel@redhat.com>

Signed-off-by: Hannah Braswell <hbraswel@redhat.com>

Replace two separate maps (complypackRefs evaluatorID->ref and reqToEvaluator reqID->evaluatorID) with a single pre-resolved reqToComplypackRef map (reqID->ref). The resolution chain is composed in buildReqToComplypackRef() in the scan pipeline, keeping the Evaluator free of evaluator-ID routing concerns. This simplifies the Evaluator constructor from 6 to 5 params, reduces resolveComplypackRef to a direct map lookup, and adds a collision warning when multiple complypacks share an evaluator-id. Assisted-by: Claude (Anthropic, Claude Opus 4.6) Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>

…refs refactor: pre-resolve complypack refs upstream of Evaluator

jpower432

LGTM

gvauter

Lgtm

hbraswelrh requested review from gvauter and jpower432 June 11, 2026 14:50

hbraswelrh force-pushed the opsx/eval-log-plan-step-identity branch 3 times, most recently from 8f8f4d4 to 0279571 Compare June 11, 2026 15:14

hbraswelrh mentioned this pull request Jun 11, 2026

fix(opa): populate steps on synthetic passing assessments complytime/complytime-providers#70

Merged

hbraswelrh force-pushed the opsx/eval-log-plan-step-identity branch 2 times, most recently from 7813ef1 to 2a7f806 Compare June 11, 2026 17:05

hbraswelrh marked this pull request as ready for review June 11, 2026 18:06

hbraswelrh requested a review from a team as a code owner June 11, 2026 18:06

gvauter previously approved these changes Jun 11, 2026

View reviewed changes

Comment thread internal/cache/state.go

hbraswelrh dismissed gvauter’s stale review via 7232297 June 11, 2026 20:52

This comment was marked as duplicate.

Sign in to view

jpower432 requested changes Jun 11, 2026

View reviewed changes

Comment thread cmd/complyctl/cli/scan.go Outdated

hbraswelrh mentioned this pull request Jun 12, 2026

fix: convert provider Steps to gemara AssessmentStep closures #575

Closed

hbraswelrh force-pushed the opsx/eval-log-plan-step-identity branch from 7232297 to 0645189 Compare June 12, 2026 12:41

hbraswelrh requested a review from jpower432 June 12, 2026 13:02

hbraswelrh force-pushed the opsx/eval-log-plan-step-identity branch from 363f43b to 6158b29 Compare June 12, 2026 13:12

gvauter self-requested a review June 12, 2026 13:41

gvauter previously approved these changes Jun 12, 2026

View reviewed changes

marcusburghardt reviewed Jun 12, 2026

View reviewed changes

Comment thread cmd/complyctl/cli/scan.go Outdated

Comment thread cmd/complyctl/cli/scan.go Outdated

hbraswelrh force-pushed the opsx/eval-log-plan-step-identity branch from 6158b29 to 82c16c8 Compare June 12, 2026 14:57

hbraswelrh dismissed gvauter’s stale review via 9ebddc8 June 12, 2026 15:06

marcusburghardt previously approved these changes Jun 12, 2026

View reviewed changes

gvauter previously approved these changes Jun 12, 2026

View reviewed changes

hbraswelrh added 2 commits June 12, 2026 13:47

hbraswelrh added 4 commits June 12, 2026 13:49

fix: addresses review comment

12f24f8

Signed-off-by: Hannah Braswell <hbraswel@redhat.com>

fix: address feedback on warning labels

d9b633a

Signed-off-by: Hannah Braswell <hbraswel@redhat.com>

fix: regenerate baseline.json after rebase onto upstream/main

4e3ebb9

hbraswelrh dismissed stale reviews from gvauter and marcusburghardt via 4e3ebb9 June 12, 2026 17:54

hbraswelrh force-pushed the opsx/eval-log-plan-step-identity branch from 9ebddc8 to 4e3ebb9 Compare June 12, 2026 17:54

jpower432 mentioned this pull request Jun 12, 2026

refactor: pre-resolve complypack refs upstream of Evaluator hbraswelrh/complyctl#5

Merged

5 tasks

Merge pull request #5 from jpower432/refactor/pre-resolve-complypack-…

f13c845

…refs refactor: pre-resolve complypack refs upstream of Evaluator

jpower432 approved these changes Jun 12, 2026

View reviewed changes

hbraswelrh requested a review from gvauter June 12, 2026 20:09

gvauter approved these changes Jun 12, 2026

View reviewed changes

hbraswelrh merged commit c4cfc1a into complytime:main Jun 12, 2026
27 of 30 checks passed

hbraswelrh deleted the opsx/eval-log-plan-step-identity branch June 12, 2026 20:18

jpower432 mentioned this pull request Jun 12, 2026

Generation state does not track complypack digests #583

Open

hbraswelrh mentioned this pull request Jun 12, 2026

Scan command does not convert provider Steps to Gemara AssessmentStep closures #573

Closed

Conversation

hbraswelrh commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Notes

Changes

Unchanged

Related Issues

Review Hints

Uh oh!

hbraswelrh commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Compliance Scan Report: policies/test-opa-policy

Control: container-resource-limits

check-resource-limits

Control: container-run-as-nonroot

check-run-as-nonroot

Evaluation Log

Uh oh!

gvauter left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment was marked as duplicate.

Uh oh!

Uh oh!

hbraswelrh commented Jun 12, 2026

Compliance Scan Report: policies/test-opa-policy

Control: container-resource-limits

check-resource-limits

Control: container-run-as-nonroot

check-run-as-nonroot

Evaluation Log

Uh oh!

gvauter left a comment

Choose a reason for hiding this comment

Uh oh!

marcusburghardt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hbraswelrh commented Jun 12, 2026

Uh oh!

marcusburghardt left a comment

Choose a reason for hiding this comment

Uh oh!

gvauter left a comment

Choose a reason for hiding this comment

Uh oh!

jpower432 left a comment

Choose a reason for hiding this comment

Uh oh!

gvauter left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

hbraswelrh commented Jun 11, 2026 •

edited

Loading

hbraswelrh commented Jun 11, 2026 •

edited

Loading