Map controls directly to Azure Policy instead of a 10-control MCSB stub - #56
Merged
Conversation
compliance-pipeline/ was a fourth, fully standalone mapping stack with its own policy_mapper.py hardcoding a 64-GUID menu (32 overlapping the backend's pre-B1 34-GUID menu) - the same defect B1 fixed in the backend pipeline, at larger scale, in a copy nothing in app/ imports or references (confirmed: no code or test under app/ touches it). Deleted the whole directory rather than reworking it to delegate to AIMappingService: it has its own requirements.txt and entry point, never shared a runtime with the backend, and nothing depends on it. Hardcoding a policy shortlist is exactly the failure mode the AI mapping engine exists to avoid - a curated list caps recall at whatever its author thought of, against a catalogue of 2,467 shipped definitions. Removed its two remaining references in README.md and docs/FUNCTIONAL_SPEC.md. There is now exactly one mapping engine in the repository (AIMappingService), reached by both the services path and the pipeline path (B1). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 39bc839d-c643-40f2-b8a7-a7e524d91ed6
…SB stub
The MCSB intermediate taxonomy (MCSBService) never shipped its real ~200-
control benchmark file in any deployment (docs/BACKLOG.md B4) - every run
silently fell through to a hardcoded 10-control demo set, capping mapping
accuracy at whatever those 10 buckets could approximate regardless of how
good retrieval or the model were.
- Removed MCSBService, MCSBControl, and the mcsb_control_id/name/domain
fields from ControlMapping/ControlPolicyMapping. Controls now map
directly against the full ~2,467-definition Azure Policy catalog, which
retrieval already used - only the MCSB label was ever gating anything.
Added a deterministic, server-computed policy_category field (from the
catalog's own category of the selected policies) as the replacement
grouping label in CSV exports, the manual register, and the frontend.
- Rewrote the confidence-scoring rubric in SYSTEM_PROMPT to grade
confidence_score against how closely the selected Azure Policy
definition's own text matches the control's literal wording, anchored
with real worked examples from the NCSP gold mapping workbook, instead
of abstract "0.9-1.0: exact match" bullets scored against MCSB fit.
- Surfaced built-in initiatives (Microsoft cloud security benchmark, ASC
Default, other Defender for Cloud initiatives) as retrieval context via
policy_catalog_service.initiatives_containing(), so Defender's own
policies/configs/initiatives are considered as legitimate mapping
candidates. defender_recommendations stays forced-empty (no live
subscription to verify free-text recommendations against).
- Removed the GET /mapping/mcsb/{controls,domains} endpoints and the MCSB
gate on the /health status field.
Verified: full backend pytest suite (405/405) and frontend suite (846
pass) both regression-clean against baseline (git stash diff showed an
identical pre-existing 14-test failure set on both, none introduced here).
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 322e5e54-ab2e-43b0-956c-5ec01cd955e7
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
The MCSB intermediate taxonomy (
MCSBService) never shipped its real ~200-control benchmark file in any deployment (docs/BACKLOG.mdB4) — every run silently fell through to a hardcoded 10-control demo set, capping mapping accuracy at whatever those 10 buckets could approximate, regardless of how good retrieval or the model were.Removed the MCSB bottleneck entirely. Deleted
MCSBService,MCSBControl, themcsb_control_id/name/domainfields onControlMapping/ControlPolicyMapping, and theGET /mapping/mcsb/{controls,domains}endpoints. Controls now map directly against the full ~2,467-definition Azure Policy catalog — which retrieval already searched, only the MCSB label was ever gating anything. Added a deterministic, server-computedpolicy_categoryfield (from the catalog's own category of the selected policies) as the replacement grouping label in CSV exports, the manual register, and the frontend.Rewrote the confidence-scoring rubric.
confidence_scoreis now graded against how closely the selected Azure Policy definition's own text matches the control's literal wording, anchored with real worked examples pulled from the NCSP gold mapping workbook (HYOK→CMK policies ≈0.95, multi-resource encryption ≈0.75, MFA-via-Entra-Config ≈0.5-0.6, pure governance = 0.0) — instead of abstract "0.9-1.0: exact match" bullets scored against MCSB fit.Surfaced Defender for Cloud's built-in initiatives. The catalog's 150 built-in initiatives (Microsoft cloud security benchmark, ASC Default, other Defender-backing initiatives) were loaded but never retrievable.
_search_azure_policiesnow callsinitiatives_containing()per candidate so the model sees which initiative(s) already bundle it.defender_recommendationscorrectly stays forced-empty (no live subscription to verify free-text recommendations against).Verification
Ran the full pytest suite before/after via
git stashdiff on both backend and frontend paths:No regressions introduced.
Co-authored-by: Copilot App 223556219+Copilot@users.noreply.github.com
Copilot-Session: 322e5e54-ab2e-43b0-956c-5ec01cd955e7