Build-DeploymentPlans -BuildExemptionsOnly fails on policydefinition that has been removed on a newer version of the initiative

[SebastianVeile]

Describe the bug
We have pinned the version of the "Microsoft Cloud Security Benchmark V2" policy initiative to version 1.3.*-preview in our policy assignment.

{
  "$schema": "https://raw.githubusercontent.com/Azure/enterprise-azure-policy-as-code/main/Schemas/policy-assignment-schema.json",
  "nodeName": "root/Deploy-MCSB2-Monitoring",
  "assignment": {
    "name": "Deploy-MCSB2-Monitoring",
    "displayName": "Microsoft Cloud Security Benchmark v2",
    "description": "Microsoft Cloud Security Benchmark v2 policy initiative."
  },
  "definitionEntry": {
    "displayName": "Microsoft Cloud Security Benchmark v2",
    "policySetId": "/providers/Microsoft.Authorization/policySetDefinitions/e3ec7e09-768c-4b64-882c-fcada3772047"
  },
  "definitionVersion": "1.3.*-preview",
  "enforcementMode": "Default",
  "parameters": {},
  "nonComplianceMessages": [
    {
      "message": "Microsoft Cloud Security Benchmark v2 must be met."
    }
  ],
  "scope": {
    "epac-dev": [
      "/providers/Microsoft.Management/managementGroups/epac-dev"
    ]
  }
}

Everything works fine. We have deployed it without issues and the reference is 1.3.*-preview in the portal.

This policy initiative has since been updated to 1.4.* (We are not ready to upgrade to 1.4.* yet)
In this 1.4.* version of the policy initiative the following definition has been removed:
remove Policy Configure Azure AI Services resources to disable local key access (disable local authentication) (55eff01b-f2bd-4c32-9203-db285f709d30)
But it still exist in version 1.3.*-preview

If we then go ahead and create an exemption on the Microsoft Cloud Security Benchmark V2 on that specific policy, the Build-DeploymentPlans fails as the policy reference does no longer exist (Even though we have set the version to 1.3.*-preview where the policy reference should exist). See screenshot below.

My thoughts are that it does not respect the pinned version we have assigned and instead plans against the latest version of the policy initiative.

To Reproduce

1. Create an assignment on Microsoft Cloud Security Benchmark V2 with version pinned to 1.3.*-preview
2. Create an exemption on the following policy: "Configure Azure AI Services resources to disable local key access (disable local authentication)"
3. Execute command: Build-DeploymentPlans -PacEnvironmentSelector ${{ parameters.pacEnvironmentSelector }} -BuildExemptionsOnly -DevOpsType "ado" -InformationAction Continue
4. Error

Expected behavior
Pipeline should succeed and exemption and plan show exemption to be created

Screenshots

EPAC Version
Latest

[anwather]

You're correct - we only sync the latest version from the Graph, for this policy its actually v1.6.0. What we need to do is pull all the other versions for each of the policies as well so we can compare to the specified version - it's actually going to be a fair bit of work. In the meantime what if you just deployed that single policy as its own assignment, and repointed the exemption back to the new assignment?

[SebastianVeile]

Thank you for taking this on. I really appreciate all the effort you put into improving and maintaining this project. EPAC is a godsend and has helped us a lot!

Looking at the suggested workaround, I'm not sure it would solve our issue. My understanding is that the policy would still be included in the original initiative assignment, meaning we'd still need to exempt that same policy from the initiative itself. In that case, creating a separate assignment wouldn't seem to reduce the number of required exemptions. But perhaps I'm misunderstanding the proposed approach?

Besides that, we're fortunately able to create this exemption through the portal for now, so there's no immediate need for a fix on our side. We're happy to wait with syncing exemptions until this has been resolved.

[anwather]

I've got a fix for this in at the moment - not deployed yet but under aw/issue1349 branch. If you are able to test to see if it solves your problem that would be great. Note that if you have deployed policy set definitions that contain a definitionVersion key it will require extra time to look these up so there will be a small impact to the total time for the run.

[anwather]

Sebastian Veile (@SebastianVeile) - let me know if you have some time to test this please

[SebastianVeile]

Hi Anthony Watherston (@anwather),

Thanks for the update and for working on the fix.

I’m no longer assigned to the project where this issue occurred, so I’m not able to test the branch myself. I’ll reach out to my colleagues currently working on that project and ask if they can test it on their side.

Thanks again for looking into this!

[anwather]

Thanks let me know please