Score any public Git repo for AI-pattern signals and maintainability, then rank repos in a shareable league table. Makes the abstract "is this vibe-coded?" debate concrete, reproducible, and shareable.
Read this first: the AI-pattern score is a heuristic proxy, not a lie detector. It measures style, structure, and git-workflow patterns that anecdotally correlate with template-heavy / machine-assisted / scaffold-dumped code. Humans write code that trips these signals too. A high score is a conversation starter and a code-smell flag — never proof that a model wrote the code. The tool is deliberately calibrated to clear mature, hand-crafted libraries rather than false-flag them (see the sample league below).
Given a repo, it clones it locally (full history, so commit-workflow signals work), walks the authored source (skipping vendored/generated dirs), and computes:
- AI-pattern score (0–100, proxy) — weighted blend of five deterministic signals.
- Maintainability score (0–100) — independent: complexity, function length, test presence, comment balance.
- A scorecard (markdown + JSON) with the raw evidence behind every signal.
- A league table across many repos (markdown +
leaderboard.html+ JSON).
Everything that feeds the scores is deterministic from the source + git history. The optional --llm flag adds a qualitative paragraph only — it never moves the numbers.
git clone https://github.com/JDDavenport/vibe-coding-detector.git
cd vibe-coding-detector
pip install -r requirements.txt # only dependency: radon (optional — see below)git must be on PATH. radon is optional: with it, Python complexity uses real cyclomatic complexity; without it, the tool falls back to a language-agnostic block-length proxy and tells you so in the scorecard.
Add a vibe score to any repo's CI — one snippet in .github/workflows/vibe-check.yml:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # full history for the commit-burstiness signal
- name: Vibe check
id: vibe
uses: JDDavenport/vibe-coding-detector@main
- run: echo "Score ${{ steps.vibe.outputs.ai-pattern-score }}/100 — ${{ steps.vibe.outputs.band }}"Action outputs:
| Output | Description |
|---|---|
ai-pattern-score |
AI-pattern score 0–100 |
maintainability-score |
Maintainability score 0–100 |
band |
Human-readable band label |
badge-url |
shields.io badge URL — paste into your README |
scorecard-json |
Path to full JSON scorecard |
README badge — generate one for any repo with --badge:
python -m vibe_detector score psf/requests --badgeOutput:
[](https://github.com/JDDavenport/vibe-coding-detector)
Drop that line into your README to show your score.
Score one repo (accepts a full URL, owner/repo shorthand, or a local path):
python -m vibe_detector score psf/requests
python -m vibe_detector score https://github.com/pallets/flask --json out.json --md out.md
python -m vibe_detector score ./my-local-checkout
python -m vibe_detector score owner/repo --llm # + qualitative read (non-scoring)
python -m vibe_detector score owner/repo --badge # + shields.io badge for the READMEBuild a league table from a list of repos (one per line in a text file):
python -m vibe_detector league examples/repos.txt --out-dir examplesThat writes league.md, leaderboard.html, league.json, and a per-repo scorecard-*.md into the output dir.
| Signal | Weight | What it measures | Honest caveat |
|---|---|---|---|
comment_narration |
1.4 | Comment-to-code ratio + density of LLM-style narrating comments ("This function takes the data and...", "First, we initialize..."). | Verbose teaching codebases and tutorials also narrate. |
commit_burstiness |
1.3 | Big-bang first commit (one commit holding most of the code), sparse history relative to LOC, and many commits within the same minute. | Squash-merge workflows and imported codebases mimic this. Excluded entirely when no git history is available. |
structural_duplication |
1.2 | Fraction of substantive lines repeated 3+ times (per-line contribution capped so one fixture line can't dominate; fixture/test/snapshot dirs excluded). | DRY violations happen for non-AI reasons. |
placeholder_density |
1.0 | Density of genuine unfinished-work markers (TODO/FIXME/"implement here"/"not implemented yet"). Generic words like foo/bar/example.com are deliberately not counted — they fired constantly on real code and destroyed the signal. |
Active projects carry honest TODOs. |
identifier_genericness |
1.0 | Share of generic identifiers (data/result/temp/handler) vs domain-specific names, language keywords excluded. |
Glue code is legitimately generic. |
The score is the weighted mean of these signals, scaled to 0–100. If git history is unavailable (e.g. a tarball download), the commit signal is dropped from the denominator rather than counted as zero, so the repo isn't unfairly deflated.
| Component | What | Healthy = |
|---|---|---|
| Complexity | Average cyclomatic complexity (radon, Python) or block-length proxy | low |
| Function length | p95 block size | small |
| Tests | test-LOC : code-LOC ratio + test-file presence | present |
| Comment balance | comment ratio near a sane sweet spot (~0.18) | balanced |
Thresholds are anchored to measured distributions on real public repos, not guessed. During development, three signals saturated at 1.0 for nearly every repo (zero discriminating power); they were re-calibrated against a measured baseline so that mature, hand-crafted, heavily-reviewed classics land low — which is the honest expectation and the credibility test. See the sample league below: psf/black (124k LOC, heavily reviewed) scores 1.9; the canonical libraries cluster in the 18–28 range.
Bundled in examples/ (regenerate with python -m vibe_detector league examples/repos.txt). Two tiers on purpose: hand-crafted classics that should score low, and starter templates / small newer tools that are scaffold-heavy by design.
| # | Repo | AI-pattern (proxy) | Band | Maintainability |
|---|---|---|---|---|
| 1 | OthersideAI/self-operating-computer | 31.8 | Some scaffold/AI-pattern signals | 64.5 |
| 2 | psf/requests | 27.5 | Some scaffold/AI-pattern signals | 86.0 |
| 3 | vercel/next-learn | 27.3 | Some scaffold/AI-pattern signals | 60.4 |
| 4 | pallets/flask | 26.3 | Some scaffold/AI-pattern signals | 86.0 |
| 5 | pallets/click | 26.0 | Some scaffold/AI-pattern signals | 86.0 |
| 6 | tiangolo/full-stack-fastapi-template | 23.8 | Mostly human-shaped | 90.4 |
| 7 | httpie/cli | 17.9 | Mostly human-shaped | 96.0 |
| 8 | psf/black | 1.9 | Mostly human-shaped | 87.2 |
The headline finding is the absence of false positives: the tool refuses to flag the canonical Python libraries it would be most embarrassing to mislabel. black — the largest repo in the set — reads as the most human, because it's domain-named, lightly-commented, and heavily reviewed. That's the credibility check passing.
- Not proof of AI authorship. It cannot be. It measures patterns, and humans produce those patterns.
- Language coverage is strongest for Python (real cyclomatic complexity); other languages use proxies.
- Squash-merge / imported / migrated repos can look "bursty" through no fault of the code.
- Small repos have noisy signals; treat scores under ~1k LOC as directional only.
- Don't use this to accuse a person. Use it to start a conversation about code health and provenance norms.
python -m unittest discover -s tests # deterministic, no networkMIT — see LICENSE.