Skip to content

refactor(compliance): consolidate contributor check CLIs - #3876

Open
Carlos Hernandez (carloshvp) wants to merge 2 commits into
microsoft:mainfrom
carloshvp:refactor-contributor-check-source
Open

refactor(compliance): consolidate contributor check CLIs#3876
Carlos Hernandez (carloshvp) wants to merge 2 commits into
microsoft:mainfrom
carloshvp:refactor-contributor-check-source

Conversation

@carloshvp

@carloshvp Carlos Hernandez (carloshvp) commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Summary

  • make agent_compliance.cli the single source for contributor and credential checks
  • preserve the repository commands as 27-line compatibility shims and ship the allowlist as package data
  • add import and CLI parity tests, then include the package source in the contributor-check sparse checkout

The canonical contributor checker keeps the package's retry and future-timestamp fixes and gains the repository checker's org-aware and domain-aware hardening. cluster_detect.py and contributor_check_action.py remain repository-local tools.

Validation

  • 132 repository contributor-check tests pass on Python 3.11 and 3.12
  • 20 package retry, timestamp, and shim-parity tests pass on Python 3.11 and 3.12
  • 443 dependency-independent compliance tests pass; the existing CI matrix supplies the native ACS and Agent OS dependencies for the remaining cases
  • 84 CI structure tests pass, including a clean sparse-checkout simulation of the contributor-check action
  • Ruff, Pyright, formatting, changed-line spelling, wheel build, installed console commands, and packaged allowlist checks pass

Closes #3571

Signed-off-by: Carlos Hernandez-Vaquero <carloshvp@gmail.com>
Signed-off-by: Carlos Hernandez-Vaquero <carloshvp@gmail.com>
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@github-actions github-actions Bot added dependencies Pull requests that update a dependency file tests scripts/ci/cd size/XL Extra large PR (500+ lines) labels Sep 2, 2026
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

PR Review Summary

Check Status Details
🔍 Code Review ⚠️ Missing No current-run comment
🛡️ Security Scan ⚠️ Missing No current-run comment
🔄 Breaking Changes ⚠️ Missing No current-run comment
📝 Docs Sync ⚠️ Missing No current-run comment
🧪 Test Coverage ⚠️ Missing No current-run comment

Verdict: ⚠️ AI review incomplete; ready for human review

AI review comments are untrusted advisory output. The summary reports workflow-generated completion status only, not model-authored pass/fail claims.

@carloshvp

Copy link
Copy Markdown
Contributor Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Commenter does not have sufficient privileges for PR 3876 in repo microsoft/agent-governance-toolkit

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consolidating these is clearly right and long overdue. One thing about the change is not what the
title says it is, and it decides how this should be reviewed.

This is not a move, it is a merge of two divergent implementations. Measured on main:

scripts/contributor_check.py                                 1475 lines
agent_compliance/cli/contributor_check.py                    1076 lines
diff between the two main copies                             ~605 changed lines

and this PR's consolidated file is 1700 lines, with ~1238 changed lines against the packaged copy
it replaces. So the two copies had drifted substantially and the result is a third thing that matches
neither original. That is the correct destination, but "refactor" and "consolidate" both imply
behaviour preservation, and behaviour preservation is the one property a merge of two divergent forks
does not get for free.

Why that matters more here than it would elsewhere. scripts/contributor_check.py is what the
contributor-check workflow runs, through
scripts/contributor_check_action.py, and its risk output becomes the
needs-review:LOW/MEDIUM/HIGH/UNKNOWN label applied to every incoming PR. Those labels are already
imperfect in ways people have noticed. Changing which of two implementations produces them, in a PR
whose stated purpose is deduplication, means any resulting change in labelling would be attributed to
the wrong cause for weeks.

What I would want before merging is not more code but evidence: run both main copies and the
consolidated one against the same set of, say, a dozen real logins spanning LOW through HIGH, and
paste the risk values in a table. If all three agree, the merge is proven and this becomes an easy
approval. If they diverge, that is worth knowing regardless, because it means the workflow and the
agt-contributor-check console script have been giving different answers to the same question.

Two things I want to credit specifically.

Adding agent-governance-python/agent-compliance/src/agent_compliance to the workflow's
sparse-checkout is easy to forget and would have produced a confusing ModuleNotFoundError in CI
only, not locally. Catching it in the same PR is the difference between this landing and this being
reverted.

The shim shape is right. sys.modules[__name__] = _implementation after defining main keeps both
python scripts/contributor_check.py and import credential_audit working, so existing callers and
existing tests do not need touching. test_repository_shim_is_packaged_module asserting
shim is packaged pins exactly the property that makes it safe, which is the identity of the module
object rather than the equality of its behaviour.

Sequencing with #3812. That PR fixes a real pagination bug in _search, editing both
scripts/credential_audit.py and the packaged copy. Your shim deletes the implementation from the
scripts/ one. Landing #3812 first is the cheaper order: it is 160 lines against 3,200, and you then
drop its now-redundant scripts/ hunk on rebase. The other way round, #3812 has to be re-cut.

One small gap. The [tool.setuptools.package-data] addition covers cli/*.json, which picks up
contributor_check_allowlist.json. Worth confirming that the packaged wheel actually contains it, for
example with python -m build and a unzip -l on the result, since a missing data file in a wheel
fails at runtime for installed users and never in CI, where the source tree is present.

Nothing blocking on the direction.

@carloshvp

Copy link
Copy Markdown
Contributor Author

Follow-up evidence requested in review, run on 2026-09-03 against exact PR head b246a69a and current main at 359a2332.

For each login, I called check_contributor(login, "microsoft/agent-governance-toolkit") from the current-main repository script, the current-main packaged module, and this PR's consolidated module. Each unique live GitHub API response was fetched once per login and deep-copied to all three implementations, so they scored the same snapshot. The only runtime adjustment shortened the maintainer-lookup pacing delay; scoring code and thresholds were unchanged. These are checker outputs used for parity testing, not manual judgments about contributors.

Login main repo script main packaged CLI PR consolidated
carloshvp LOW LOW LOW
torvalds LOW LOW LOW
gvanrossum LOW MEDIUM LOW
sindresorhus LOW HIGH LOW
Mr-Neutr0n HIGH HIGH HIGH
DhineshPonnarasan MEDIUM MEDIUM MEDIUM
Ricky-G LOW LOW LOW
AgentGymLeader MEDIUM MEDIUM MEDIUM
EHESPO HIGH HIGH HIGH
JinhaoSong322 HIGH HIGH HIGH
CTWalk HIGH HIGH HIGH
Knapp-Kevin MEDIUM MEDIUM MEDIUM

The consolidated implementation matches the current workflow script in 12 of 12 cases across LOW, MEDIUM, and HIGH. The two mismatches are between the already-divergent main copies: the old packaged CLI rates gvanrossum MEDIUM and sindresorhus HIGH, while the workflow script rates both LOW. This PR therefore preserves workflow label behavior and intentionally changes the installed agt-contributor-check behavior to match the workflow's established-account hardening.

Wheel evidence from the same exact head:

$ python -m build
Successfully built agent_governance_toolkit_compliance-5.0.0.tar.gz and agent_governance_toolkit_compliance-5.0.0-py3-none-any.whl

$ unzip -l dist/agent_governance_toolkit_compliance-5.0.0-py3-none-any.whl | rg contributor_check
    59216  09-03-2026 17:57   agent_compliance/cli/contributor_check.py
      770  09-03-2026 17:57   agent_compliance/cli/contributor_check_allowlist.json

unzip -p ... contributor_check_allowlist.json | jq -e '.users == [] and .orgs == []' also passes, confirming the packaged data is present and readable. #3812 remains the intended first merge; I have not rebased this PR ahead of it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file scripts/ci/cd size/XL Extra large PR (500+ lines) tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

contributor_check and credential_audit have diverged into two copies; CI runs the one without retry, the packaged one lacks all hardening

2 participants