Skip to content

Fix LLM consensus (Groq models, vote window) and make deploy/backup workflows safe - #2

Merged
HardbanRecordsLab merged 7 commits into
mainfrom
fix/groq-models-llm-window-2026-09-21
Sep 22, 2026
Merged

HardbanRecordsLab merged 7 commits into
mainfrom
fix/groq-models-llm-window-2026-09-21

Conversation

@HardbanRecordsLab

@HardbanRecordsLab HardbanRecordsLab commented Sep 21, 2026 •

Copy link
Copy Markdown
Owner

Summary

Three groups of changes, all found while running the production analysis on 2026-09-21. Do not merge before reading "Notes for the reviewer" below — the merge itself runs the deploy workflow.

1. LLM consensus (175f9f3)

  • Groq no longer serves llama-3.3-70b-versatile / llama-3.1-8b-instant for our key (HTTP 404 model_not_found, checked with GET /openai/v1/models). Model ids now live in services/groq_models.py (GROQ_MODEL_PRO=openai/gpt-oss-120b, GROQ_MODEL_FLASH=openai/gpt-oss-20b, both overridable). gpt-oss models get reasoning_effort=low and extra max_tokens headroom.
  • Pro mode kept getting a single vote: deepseek-v4-flash (23-45 s) and gemma regularly missed the 40 s vote timeout, and the whole LLM step was capped at ~44 s. New lineup gemini-2.5-flash-lite / mistral-small-24b / gemma-4-31b (measured 3-19 s), vote timeout 75 s (OPENROUTER_VOTE_TIMEOUT_SEC), LLM window 90 s (LLM_MIN_WINDOW_SEC). Result: 3 LLMs per analysis instead of 1.
  • entrypoint.sh no longer prints the database password in the container log; /api/health/diagnose also reports the Groq models and vote timeout.

2. Deploy workflow (cf386ee)

Production runs docker-compose.yml on PostgreSQL (hbrl-postgres/metadata_engine), but "Deploy to VPS" deployed vps.docker-compose.yml (pinned to SQLite) and rewrote the VPS .env from GitHub Secrets (DATABASE_URL=sqlite:///…). A successful run would have started the app on an empty SQLite file.

  • Deploys docker-compose.yml. The VPS .env owns DATABASE_URL: the job refuses to run unless it already points at hbrl-postgres/metadata_engine, never writes the stale secret, and re-checks the assembled file before swapping it in.
  • git fetch + checkout main + merge --ff-only instead of git pull (fails instead of mixing commits).
  • No compose down; the running image is tagged :previous; after the swap the job waits for healthy, verifies the container's DATABASE_URL and rolls back to :previous on failure. Concurrency group so two deploys never overlap.
  • vps.docker-compose.yml is now a deprecated alias (include: docker-compose.yml); resolved config checked identical to docker-compose.yml.

3. Backup workflow + docs (ce36ef3)

backup.yml called the sqlite3 CLI (not installed on the VPS) against a retired SQLite file: it failed daily from 2026-09-14 and no backup was ever written.

  • Now pg_dump -Fc metadata_engine on the VPS, verified (size + pg_restore --list), mode 600, keeps the last 14 dumps of this job (mme-pg-*) only. Not uploaded as an artifact — this repository is public and a dump holds user data.
  • README / ARCHITECTURE / OPERATIONS updated to the real setup (PostgreSQL, guarded deploy with rollback, backup + restore steps).

Verification

  • Backend: 8 new/updated unit tests pass locally (test_groq_models.py, test_openrouter_vote.py); 45 tests pass overall; the remaining local failures/collection errors come from dependencies missing on the dev machine (librosa, tinytag) — the same 7 collection errors occur on main.
  • Live on the VPS: Groq call through the new helper answers in 0.7 s; pro analyses report 3 LLMs; startup log shows the masked URL.
  • Deploy script: extracted from the YAML and run against stubbed docker/git/curl in 7 scenarios (happy path, SQLite in .env, missing .env, container on SQLite, never healthy, no fast-forward, first DATABASE_URL line SQLite). Guard checked against the real VPS .env (pass). Backup script: 4 scenarios (good dump, too small, no table data, rotation).
  • Not verified: a real GitHub Actions run of the new workflows.

Notes for the reviewer

  • Merging to main triggers "Deploy to VPS" — the first run of the new script on real Actions. The guard fails closed before anything is touched, and there is an automatic rollback after the swap. Production currently already runs 175f9f3 from this branch, so the VPS checkout will be on a branch at merge time; the new checkout main + --ff-only step moves it back to main.
  • The new backup job deletes its own dumps beyond the 14 most recent (mme-pg-* only). There is no off-site copy yet.
  • Not part of this PR: resync of the analysis_history id sequence after the SQLite→Postgres migration (a one-off SQL, pending approval) and moving DSP off the event loop (performance).

🤖 Generated with Claude Code

Summary by Sourcery

Harden production data protection and deployment while restoring reliable multi-model LLM consensus.

New Features:

  • Centralize Groq model selection with configurable fallbacks and expose active LLM models and vote settings through the health diagnostics endpoint.
  • Run production database backups as verified, permission-restricted PostgreSQL dumps retained on the VPS.

Bug Fixes:

  • Restore reliable multi-model LLM consensus by replacing retired Groq models, updating the OpenRouter lineup, and extending the vote and LLM time windows.
  • Prevent deployment from switching production to SQLite by enforcing the VPS PostgreSQL configuration and using the production compose setup.
  • Stop exposing database passwords in startup logs and correct CI and test environment setup.

Enhancements:

  • Make deployments fail-safe with serialized execution, fast-forward-only updates, health and database verification, and automatic rollback to the previous image.
  • Update operational documentation to describe the PostgreSQL production architecture, guarded deployments, and backup and restore procedures.

CI:

  • Configure required application environment variables and Python import paths so backend tests run reliably in CI.

Deployment:

  • Replace the SQLite-based backup workflow with verified PostgreSQL backups and remove public artifact uploads of database dumps.

Documentation:

  • Update the README, architecture, and operations documentation for PostgreSQL production storage and the revised deployment and backup workflows.

Tests:

  • Add coverage for Groq model selection, fallback behavior, vote timeout configuration, and revised analysis time-budget behavior.

Chores:

  • Retain the former VPS compose file as a deprecated alias of the production compose configuration.

HardbanRecordsLab and others added 3 commits September 21, 2026 20:10
…stop logging DB password

Verified on the VPS on 2026-09-21 (analyses returned "1 LLMs" / "0 LLMs"):

* Groq no longer serves llama-3.3-70b-versatile / llama-3.1-8b-instant for our key
  (HTTP 404 model_not_found). Model ids now come from services/groq_models.py
  (GROQ_MODEL_PRO=openai/gpt-oss-120b, GROQ_MODEL_FLASH=openai/gpt-oss-20b, both overridable);
  gpt-oss reasoning models get reasoning_effort=low and extra max_tokens headroom.
* Pro mode got a single vote: deepseek-v4-flash (23-45s) and gemma (17-19s) regularly missed the
  40s vote timeout, and the whole LLM step was additionally capped at ~44s. Vote lineup is now
  gemini-2.5-flash-lite / mistral-small-24b / gemma-4-31b (measured 3-19s), vote timeout 75s
  (OPENROUTER_VOTE_TIMEOUT_SEC) and the guaranteed LLM window 90s (LLM_MIN_WINDOW_SEC).
* entrypoint.sh printed DATABASE_URL including the password to the container logs; it is masked now.
* /api/health/diagnose also reports the Groq models and the vote timeout.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…d guards and rollback

On 2026-09-21 "Deploy to VPS" rewrote the VPS .env from GitHub Secrets (DATABASE_URL=sqlite:///...)
and then failed on `git pull`; production actually runs docker-compose.yml on PostgreSQL
(hbrl-postgres/metadata_engine), while the workflow deployed vps.docker-compose.yml, which pinned
DATABASE_URL to SQLite. A successful run would have started the app on an empty SQLite file.

* Deploy docker-compose.yml (the production file) instead of vps.docker-compose.yml.
* The VPS .env owns DATABASE_URL: the deploy refuses to run unless it already points at
  hbrl-postgres/metadata_engine, does not write the stale GitHub secret, carries the value over and
  re-checks the assembled file before swapping it in (.env.new -> .env).
* git: fetch + checkout main + merge --ff-only instead of `git pull` (fails instead of mixing commits).
* No `compose down`: a failed build no longer takes the service down. The running image is tagged
  music-metadata-engine:previous first.
* Health-gated: wait for `healthy`, verify the container's DATABASE_URL still points at Postgres,
  otherwise roll back to :previous and fail the run.
* concurrency group so two deploys never overlap.
* vps.docker-compose.yml becomes a deprecated alias (`include: docker-compose.yml`); its resolved
  config was checked to be identical to docker-compose.yml.

The script was exercised against stubbed docker/git/curl in 7 scenarios (happy path, SQLite in .env,
missing .env, container on SQLite, never healthy, no fast-forward, first DATABASE_URL line SQLite).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…te file; fix docs

backup.yml called the `sqlite3` CLI, which is not installed on the VPS: it failed every day
(2026-09-14 .. 2026-09-21), the backups directory is empty and no artifact was ever uploaded.
It also targeted data/music_metadata.db, which production no longer uses (PostgreSQL since 2026-09-17).

* pg_dump -Fc of metadata_engine inside hbrl-postgres -> data/backups/mme-pg-<stamp>.dump on the VPS,
  written as .partial, verified (size + pg_restore --list), then renamed; mode 600; keeps the last 14
  dumps of this job only.
* No GitHub artifact upload: the repository is public and a dump holds user data.
* README / ARCHITECTURE / OPERATIONS now describe the real setup (PostgreSQL, docker-compose.yml, guarded
  deploy with rollback, pg_dump backup + restore steps) instead of SQLite + vps.docker-compose.yml.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@sourcery-ai

sourcery-ai Bot commented Sep 21, 2026 •

Copy link
Copy Markdown

Reviewer's Guide

The PR updates LLM integrations for currently available Groq reasoning models and restores three-provider consensus with longer, configurable vote windows. It also changes deployment and backup automation to protect the production PostgreSQL database through fail-closed environment checks, atomic configuration updates, health-gated image rollback, verified on-host pg_dump backups, and corresponding documentation.

Sequence diagram for the guarded PostgreSQL deployment and rollback

sequenceDiagram
    participant Actions as GitHub Actions
    participant VPS as VPS deploy script
    participant Git as Git repository
    participant Compose as Docker Compose
    participant Backend as metadata-backend

    Actions->>VPS: SSH deploy
    VPS->>VPS: Check .env DATABASE_URL
    alt Invalid or missing PostgreSQL URL
        VPS-->>Actions: Fail before changes
    else Valid PostgreSQL URL
        VPS->>VPS: Write and validate .env.new
        VPS->>VPS: mv .env.new .env
        VPS->>Git: git fetch origin main
        VPS->>Git: git checkout main
        VPS->>Git: git merge --ff-only origin/main
        VPS->>VPS: docker tag latest previous
        VPS->>Compose: docker compose up -d --build
        Compose->>Backend: Start new container
        VPS->>Backend: Poll health status
        VPS->>Backend: printenv DATABASE_URL
        alt Healthy and PostgreSQL URL verified
            VPS-->>Actions: Deploy OK
        else Verification failed
            VPS->>VPS: Restore previous image tag
            VPS->>Compose: docker compose up -d --no-build
            VPS-->>Actions: Fail after rollback
        end
    end
Loading

Sequence diagram for the verified PostgreSQL backup

sequenceDiagram
    participant Actions as GitHub Actions
    participant VPS as VPS backup script
    participant Postgres as hbrl-postgres
    participant Storage as VPS backup storage

    Actions->>VPS: SSH scheduled backup
    VPS->>Postgres: pg_dump -Fc metadata_engine
    Postgres-->>VPS: Binary dump stream
    VPS->>Storage: Write .partial dump
    VPS->>VPS: stat dump size
    VPS->>Postgres: pg_restore --list
    alt Size and TABLE DATA verified
        VPS->>Storage: Rename dump and chmod 600
        VPS->>Storage: Rotate mme-pg-* beyond 14
        VPS-->>Actions: Backup OK
    else Verification failed
        VPS->>Storage: Remove .partial dump
        VPS-->>Actions: Fail without registering backup
    end
Loading

Sequence diagram for the three-provider LLM consensus vote

sequenceDiagram
    participant Analyzer as Fresh track analyzer
    participant Ensemble as LLM ensemble
    participant Gemini as Gemini
    participant Mistral as Mistral
    participant Gemma as Gemma

    Analyzer->>Ensemble: _openrouter_vote(user_prompt, model_preference)
    Ensemble->>Gemini: _openrouter_classify
    Ensemble->>Mistral: _openrouter_classify
    Ensemble->>Gemma: _openrouter_classify
    Ensemble->>Ensemble: asyncio.wait(timeout=75s)
    alt Models respond within vote window
        Gemini-->>Ensemble: Classification result
        Mistral-->>Ensemble: Classification result
        Gemma-->>Ensemble: Classification result
    else Model exceeds vote window
        Ensemble->>Ensemble: Cancel pending task
    end
    Ensemble-->>Analyzer: Available vote results
Loading

File-Level Changes

Change Details Files
Centralize Groq model selection and adapt requests for the replacement reasoning models.
  • Add environment-overridable pro and flash model defaults.
  • Apply reasoning-effort settings and token headroom for gpt-oss models.
  • Update all Groq call sites, diagnostics, logging, and unit coverage.
backend/app/services/groq_models.py
backend/app/routes/generative.py
backend/app/routes/health.py
backend/app/services/fresh_track_analyzer.py
backend/app/services/groq_whisper.py
backend/app/services/llm_ensemble.py
backend/entrypoint.sh
backend/tests/test_groq_models.py
Restore reliable multi-model consensus by replacing the slow vote lineup and extending its execution budget.
  • Replace the default pro lineup with Gemini, Mistral, and Gemma models.
  • Make the vote timeout configurable with a 75-second default and grant analysis a configurable 90-second LLM window.
  • Add tests for timeout overrides, fallback behavior, and lineup composition.
backend/app/services/llm_ensemble.py
backend/app/services/fresh_track_analyzer.py
backend/app/routes/health.py
backend/tests/test_groq_models.py
Make VPS deployment PostgreSQL-safe, serialized, and rollback-capable.
  • Deploy the production compose file and fail closed unless the existing VPS DATABASE_URL targets hbrl-postgres/metadata_engine.
  • Preserve the database URL while atomically assembling the remaining environment, then use fetch/checkout/fast-forward-only synchronization.
  • Avoid taking the service down, retain the previous image, and health-check both container health and database configuration before accepting the swap; roll back on failure.
.github/workflows/deploy.yml
vps.docker-compose.yml
Replace the broken SQLite backup with verified, private PostgreSQL dumps and document the operational model.
  • Run pg_dump in custom format on the VPS, verify size and table data, restrict permissions, rotate only the latest 14 job dumps, and avoid public workflow artifacts.
  • Update architecture, deployment, database, backup, and restore documentation for PostgreSQL and guarded rollback deployment.
.github/workflows/backup.yml
README.md
docs/ARCHITECTURE.md
docs/OPERATIONS.md

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@vercel

vercel Bot commented Sep 21, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
metadata-engine Ready Ready Preview Sep 21, 2026 8:42pm UTC

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 5 issues

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="backend/app/services/fresh_track_analyzer.py" line_range="92-98" />
<code_context>
             # vote is cheap and quick (~4-40 s), so it always gets a guaranteed window instead of
             # being skipped - skipping silently produced template-only results ("0 LLMs").
-            remaining = max(remaining, 45.0)
+            # The window must cover the OpenRouter vote timeout (75s by default, see llm_ensemble); a
+            # 45s window cut the vote off and gave "1 LLMs"/"0 LLMs" results. Env: LLM_MIN_WINDOW_SEC.
+            try:
+                llm_min_window = float(os.getenv("LLM_MIN_WINDOW_SEC", "").strip() or 90.0)
+            except ValueError:
+                llm_min_window = 90.0
+            remaining = max(remaining, llm_min_window)

             # RELAXED LIMIT: Only skip if less than 3 seconds (was 5)
</code_context>
<issue_to_address>
**issue (bug_risk):** `remaining = max(remaining, llm_min_window)` ignores the caller's `time_budget` whenever the DSP layer leaves less than 90 seconds, so a request with a 4- or 6-second budget still grants the LLM an 89-second timeout and can run far beyond its advertised maximum.

**Triggers:** When the DSP pass consumes most of a caller-supplied analysis budget.

**Suggested fix:** Keep the guaranteed LLM window bounded by the original deadline, or make the guaranteed-window behavior an explicit separate budget rather than overriding `time_budget`.
</issue_to_address>

### Comment 2
<location path=".github/workflows/deploy.yml" line_range="135-162" />
<code_context>
+            mv .env.new .env
</code_context>
<issue_to_address>
**issue (bug_risk):** The workflow swaps `.env` before fetching and merging code, but failure handling only restores the previous Docker image. A failed fast-forward, failed build/start, or failed health check therefore leaves the newly assembled `.env` active instead of restoring the backed-up production configuration.

**Triggers:** When the Git operation or post-deploy verification fails after `.env` has been swapped.

**Suggested fix:** Perform the code update before swapping the environment, or install an error trap that restores `.env` together with the image during rollback.
</issue_to_address>

### Comment 3
<location path=".github/workflows/deploy.yml" line_range="90-92" />
<code_context>
-            cat <<'EOF' | sed -e 's/^[[:space:]]*//' > .env
+            # The sed strips that indentation back out before it reaches .env.new.
+            cat <<'EOF' | sed -e 's/^[[:space:]]*//' > .env.new
             GEMINI_API_KEY=${{ secrets.GEMINI_API_KEY }}
             GROQ_API_KEY=${{ secrets.GROQ_API_KEY }}
             MISTRAL_API_KEY=${{ secrets.MISTRAL_API_KEY }}
</code_context>
<issue_to_address>
**issue (broader_impact):** The generated `.env.new` contains only the hard-coded allowlist of GitHub secrets plus `DATABASE_URL`, so VPS-owned settings such as `GROQ_MODEL_PRO`, `GROQ_MODEL_FLASH`, `OPENROUTER_VOTE_TIMEOUT_SEC`, and `LLM_MIN_WINDOW_SEC` are deleted on every deploy instead of being preserved.

**Triggers:** When the VPS `.env` contains any supported local override that is not listed in the heredoc.

**Suggested fix:** Explicitly preserve or populate every supported VPS-local setting, or merge the generated secret values into the existing environment instead of replacing the file with an allowlist.
</issue_to_address>

### Comment 4
<location path=".github/workflows/deploy.yml" line_range="146" />
<code_context>
+            docker tag music-metadata-engine:latest music-metadata-engine:previous 2>/dev/null || true
</code_context>
<issue_to_address>
**issue (bug_risk):** If `music-metadata-engine:latest` does not exist, the ignored tag command creates no `:previous` image; a later health failure then runs `docker tag music-metadata-engine:previous music-metadata-engine:latest` under `set -e`, so the advertised automatic rollback fails instead of leaving a known rollback target.

**Triggers:** On the first deployment to a host without an existing `music-metadata-engine:latest` image.

**Suggested fix:** Require an existing rollback image before replacing the service, or handle the no-previous-image case explicitly and report that rollback is unavailable.

```suggestion
            docker image inspect music-metadata-engine:latest >/dev/null 2>&1 || { echo "FAIL: no existing image available for rollback; refusing to replace the service"; exit 1; }
            docker tag music-metadata-engine:latest music-metadata-engine:previous
```
</issue_to_address>

### Comment 5
<location path="backend/app/services/groq_whisper.py" line_range="230-240" />
<code_context>

         logger.info("Sending metadata prompt to Groq (length: %d)", len(prompt))

+        merge_model = groq_model("pro")
         response = client.chat.completions.create(
-            model="llama-3.3-70b-versatile",
+            model=merge_model,
             messages=[
                 {"role": "system", "content": "You are a professional music metadata API. Always respond with valid JSON only. Never include markdown or explanations outside the JSON."},
                 {"role": "user", "content": prompt},
             ],
             temperature=0.25,
-            max_tokens=2500,
+            max_tokens=groq_budget(merge_model, 2500),
             response_format={"type": "json_object"},
+            **groq_extra(merge_model),
         )

</code_context>
<issue_to_address>
**nitpick:** The class docstring still says metadata generation uses 'Llama 3.3 70B', but the changed implementation selects `groq_model("pro")`, whose default is `openai/gpt-oss-120b`; the documented model and actual model diverge.

**Suggested fix:** Update the class docstring to describe the configurable Groq model selected by `GROQ_MODEL_PRO`.
</issue_to_address>

Sourcery assessment

Needs a human reviewer. 4 findings to address first, and the deploy switches production from SQLite to the shared PostgreSQL database without showing a data migration, so it can immediately serve an empty or mismatched datastore and write new user data there. Reverting the code or image would not undo writes made to the wrong database, and the backup remains only on the VPS rather than providing an off-site recovery copy.

Blocking findings: backend/app/services/fresh_track_analyzer.py:98, .github/workflows/deploy.yml:162, .github/workflows/deploy.yml:92, .github/workflows/deploy.yml:146


Sourcery is free for open source - if you like our reviews please consider sharing them ✨

Comment on lines +92 to +98
# The window must cover the OpenRouter vote timeout (75s by default, see llm_ensemble); a
# 45s window cut the vote off and gave "1 LLMs"/"0 LLMs" results. Env: LLM_MIN_WINDOW_SEC.
try:
llm_min_window = float(os.getenv("LLM_MIN_WINDOW_SEC", "").strip() or 90.0)
except ValueError:
llm_min_window = 90.0
remaining = max(remaining, llm_min_window)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (bug_risk): remaining = max(remaining, llm_min_window) ignores the caller's time_budget whenever the DSP layer leaves less than 90 seconds, so a request with a 4- or 6-second budget still grants the LLM an 89-second timeout and can run far beyond its advertised maximum.

Triggers: When the DSP pass consumes most of a caller-supplied analysis budget.

Suggested fix: Keep the guaranteed LLM window bounded by the original deadline, or make the guaranteed-window behavior an explicit separate budget rather than overriding time_budget.

Comment on lines +135 to +162
mv .env.new .env

# Code: fast-forward only. Fails (instead of silently losing or mixing commits) if the
# VPS checkout ever diverges from origin/main again.
git fetch origin main
git checkout main
git merge --ff-only origin/main

# Keep the running image as a rollback target, then build + swap. No `compose down`:
# `up -d --build` builds first and only then replaces the container, so a failed build
# leaves the old container serving.
docker tag music-metadata-engine:latest music-metadata-engine:previous 2>/dev/null || true
docker compose -f "$COMPOSE_FILE" up -d --build

# Health-gated: container healthy AND still on production Postgres, else roll back.
healthy=0
status=unknown
for i in $(seq 1 30); do
status="$(docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' metadata-backend 2>/dev/null || echo missing)"
if [ "$status" = "healthy" ]; then healthy=1; break; fi
sleep 10
done
CONTAINER_DB="$( (docker exec metadata-backend printenv DATABASE_URL 2>/dev/null | sed -E 's#(://[^:]+:)[^@]+@#\1***@#') || true )"
echo "container health: $status | DATABASE_URL: ${CONTAINER_DB:-<none>}"
if [ "$healthy" != "1" ] || ! printf '%s' "$CONTAINER_DB" | grep -Eq '@hbrl-postgres:5432/metadata_engine'; then
echo "FAIL: post-deploy verification failed - rolling back to the previous image"
docker tag music-metadata-engine:previous music-metadata-engine:latest
docker compose -f "$COMPOSE_FILE" up -d --no-build

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (bug_risk): The workflow swaps .env before fetching and merging code, but failure handling only restores the previous Docker image. A failed fast-forward, failed build/start, or failed health check therefore leaves the newly assembled .env active instead of restoring the backed-up production configuration.

Triggers: When the Git operation or post-deploy verification fails after .env has been swapped.

Suggested fix: Perform the code update before swapping the environment, or install an error trap that restores .env together with the image during rollback.

Comment on lines 90 to 92
GEMINI_API_KEY=${{ secrets.GEMINI_API_KEY }}
GROQ_API_KEY=${{ secrets.GROQ_API_KEY }}
MISTRAL_API_KEY=${{ secrets.MISTRAL_API_KEY }}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (broader_impact): The generated .env.new contains only the hard-coded allowlist of GitHub secrets plus DATABASE_URL, so VPS-owned settings such as GROQ_MODEL_PRO, GROQ_MODEL_FLASH, OPENROUTER_VOTE_TIMEOUT_SEC, and LLM_MIN_WINDOW_SEC are deleted on every deploy instead of being preserved.

Triggers: When the VPS .env contains any supported local override that is not listed in the heredoc.

Suggested fix: Explicitly preserve or populate every supported VPS-local setting, or merge the generated secret values into the existing environment instead of replacing the file with an allowlist.

# Keep the running image as a rollback target, then build + swap. No `compose down`:
# `up -d --build` builds first and only then replaces the container, so a failed build
# leaves the old container serving.
docker tag music-metadata-engine:latest music-metadata-engine:previous 2>/dev/null || true

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (bug_risk): If music-metadata-engine:latest does not exist, the ignored tag command creates no :previous image; a later health failure then runs docker tag music-metadata-engine:previous music-metadata-engine:latest under set -e, so the advertised automatic rollback fails instead of leaving a known rollback target.

Triggers: On the first deployment to a host without an existing music-metadata-engine:latest image.

Suggested fix: Require an existing rollback image before replacing the service, or handle the no-previous-image case explicitly and report that rollback is unavailable.

Suggested change
docker tag music-metadata-engine:latest music-metadata-engine:previous 2>/dev/null || true
docker image inspect music-metadata-engine:latest >/dev/null 2>&1 || { echo "FAIL: no existing image available for rollback; refusing to replace the service"; exit 1; }
docker tag music-metadata-engine:latest music-metadata-engine:previous

Comment thread backend/app/services/groq_whisper.py Outdated
Comment on lines +230 to +240
merge_model = groq_model("pro")
response = client.chat.completions.create(
model="llama-3.3-70b-versatile",
model=merge_model,
messages=[
{"role": "system", "content": "You are a professional music metadata API. Always respond with valid JSON only. Never include markdown or explanations outside the JSON."},
{"role": "user", "content": prompt},
],
temperature=0.25,
max_tokens=2500,
max_tokens=groq_budget(merge_model, 2500),
response_format={"type": "json_object"},
**groq_extra(merge_model),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: The class docstring still says metadata generation uses 'Llama 3.3 70B', but the changed implementation selects groq_model("pro"), whose default is openai/gpt-oss-120b; the documented model and actual model diverge.

Suggested fix: Update the class docstring to describe the configurable Groq model selected by GROQ_MODEL_PRO.

Backend CI has been red on main since 2026-09-02: certificate_pdf.py runs os.makedirs(CERT_DIR)
at import (default /data/certificates) and security.py raises without SECRET_KEY, so every test
module failed at collection with "PermissionError: /data". Point CERT_DIR and a throw-away SQLite
DATABASE_URL at the runner temp dir and derive a per-run SECRET_KEY.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…un in CI) can import `app`

CI runs `pytest -q` from backend/, which - unlike `python -m pytest` - does not add the working
directory to sys.path, so every test module failed with "No module named 'app'" (hidden behind the
/data PermissionError until the previous commit). `pythonpath = .` (pytest >= 7) fixes it.
Reproduced locally with the cwd removed from sys.path: 12x "No module named 'app'" before, none after.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
With CI able to import the app, four tests that had been failing unnoticed since before 2026-09-21
showed up. None is a production regression:

* test_analysis_limit (2 tests) asserted the pre-1be3f48 behaviour "tight budget => LLM skipped /
  ~5s window". Since 1be3f48 the LLM always gets a guaranteed window (LLM_MIN_WINDOW_SEC), so the
  tests now cover the real contract: a fast LLM answer is used even with no DSP budget left, and a
  slow LLM is cut off at the window with the DSP fallback and target_met == False.
* test_analysis_timeout expected job.error == "TIMEOUT_1s"; that code exists nowhere in the app or the
  frontend - process_analysis reports the exception type name for an empty message ("TimeoutError").
* test_ai_proxy::test_config_keys required a real Gemini/Groq key in the environment; it now skips
  when none is configured (CI).

Verified locally (libraries missing on the dev machine stubbed): 12 passed, 1 skipped.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…token quota)

Groq's free tier gives each model its own daily token quota; openai/gpt-oss-120b (200k tokens/day)
was exhausted within hours by the library batch, after which the description stream and the Groq
metadata merge failed with HTTP 429 for every track - so results would have been inconsistent
(some with the Groq merge, some without).

* groq_models.groq_create(): one wrapper around chat.completions.create that walks
  primary -> GROQ_MODEL_FALLBACKS (default qwen/qwen3.8-27b, openai/gpt-oss-20b). Only 429 (limit)
  and 404 (retired model) move on; every other error is raised unchanged. Per-model reasoning
  settings and token headroom are applied inside the wrapper.
* All five Groq call sites (description stream, _groq_classify, metadata merge, lyrics, generative
  proxy) use it.
* Unit tests with a fake client: order/dedup/env, 429 -> next model, 404 -> next, 500 raised,
  all exhausted -> last error.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@HardbanRecordsLab
HardbanRecordsLab merged commit 120b88d into main Sep 22, 2026
6 checks passed
@HardbanRecordsLab
HardbanRecordsLab deleted the fix/groq-models-llm-window-2026-09-21 branch September 22, 2026 05:06

This branch was successfully deployed

1 active deployment
Preview — 6f5e7737 Deployed Sep 21, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant