feat: add analytics summary endpoint (#1050) - #1149
Conversation
|
@aaniya22 is attempting to deploy a commit to the rishabhmishra0510-5147's projects Team on Vercel. A member of the Team first needs to authorize it. |
📝 WalkthroughWalkthroughAdds a backend analytics summary endpoint with database aggregations, a frontend dashboard for visualizing the results, CLI telemetry configuration that prevents report submission when disabled, and telemetry regression tests. ChangesAnalytics and telemetry dashboard
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Dashboard
participant API
participant AnalyticsEngine
participant Database
Dashboard->>API: GET /api/v1/analytics/summary with bearer token
API->>AnalyticsEngine: await get_summary()
AnalyticsEngine->>Database: execute aggregation queries
Database-->>AnalyticsEngine: analytics summary data
AnalyticsEngine-->>API: consolidated summary
API-->>Dashboard: AnalyticsSummaryResponse
Dashboard->>Dashboard: render charts and tables
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
cli/envforage/cli.py (1)
445-451: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd regression tests for the telemetry gate.
Mock the HTTP client and verify that
ENVFORAGE_TELEMETRY=offnever callsPOST, including withquiet=True; also cover case/whitespace normalization and the enabled path.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@cli/envforage/cli.py` around lines 445 - 451, Add regression tests around the telemetry gate in the CLI flow using a mocked HTTP client: verify telemetry disabled by ENVFORAGE_TELEMETRY=off never calls POST for both quiet=False and quiet=True, cover case and whitespace normalization, and verify the enabled configuration still sends the request.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@backend/app/services/analytics.py`:
- Around line 27-31: Update the analytics aggregation around gpu_rows to group
by DiagnosticReport.gpu_name and count records in the database, retrieving only
grouped GPU names and counts. Then map each unique GPU name to its vendor with
_gpu_vendor and accumulate the returned counts into gpu_distribution without
loading every row.
---
Nitpick comments:
In `@cli/envforage/cli.py`:
- Around line 445-451: Add regression tests around the telemetry gate in the CLI
flow using a mocked HTTP client: verify telemetry disabled by
ENVFORAGE_TELEMETRY=off never calls POST for both quiet=False and quiet=True,
cover case and whitespace normalization, and verify the enabled configuration
still sends the request.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 9665b98d-603c-4e48-9967-bb883842ed58
📒 Files selected for processing (6)
backend/app/api/v1/analytics.pybackend/app/main.pybackend/app/schemas/analytics.pybackend/app/services/analytics.pycli/envforage/cli.pycli/envforage/config.py
…ishabh#1050) - Fix broken Authorization header (missing space) and Tailwind class typo in analytics dashboard frontend - Move GPU vendor classification into SQL (CASE expression) instead of loading all rows into memory, per CodeRabbit perf review - Fix ruff import ordering and mypy type annotations in analytics.py - Fix pip-audit CI config: --strict is incompatible with --skip-editable in pip-audit 2.10.1, causing false failures - Add regression tests for the telemetry gate (ENVFORAGE_TELEMETRY=off), covering case/whitespace variants and the enabled path, per CodeRabbit nitpick Signed-off-by: aaniya22 <aaniyaatomar@gmail.com>
github-actions
Bot
added
level:critical
and removed
level:intermediate
failing-checks
labels
🔍 PR Action RequiredHi @aaniya22, We detected some items on this Pull Request that require attention: ❌ Failing CI ChecksThe following check runs or commit statuses are failing (ignoring vercel): Please resolve the issues above to proceed. Last updated: Wed, 29 Jul 2026 00:07:15 GMT |
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (4)
frontend/src/app/dashboard/analytics/page.tsx (1)
4-10: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider excluding this auth-gated dashboard from indexing.
A canonical URL is declared for an internal, token-gated telemetry page. Adding
robots: { index: false, follow: false }keeps it out of search results.♻️ Proposed change
export const metadata: Metadata = { title: "Analytics Dashboard", description: "Aggregated CLI agent diagnostics and usage analytics.", + robots: { index: false, follow: false }, alternates: { canonical: "/dashboard/analytics", }, };🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@frontend/src/app/dashboard/analytics/page.tsx` around lines 4 - 10, Update the metadata export for the analytics dashboard to include robots directives that disable indexing and link following, while preserving the existing title, description, and canonical URL in metadata.frontend/src/services/api.ts (1)
76-83: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winSurface the HTTP status in the error.
The endpoint is auth-gated, so a missing/expired token yields
401, which is currently indistinguishable from a5xxin the UI ("Failed to load analytics: Failed to fetch analytics summary"). Including the status letsClient.tsxrender a sign-in prompt instead of a generic failure — relevant given the PR notes/auth/refreshcurrently returns404.♻️ Proposed refactor
getAnalyticsSummary: async (token?: string): Promise<AnalyticsSummary> => { const response = await fetchWithRetry(`${API_BASE_URL}/analytics/summary`, { cache: "no-store", headers: token ? { Authorization: `Bearer ${token}` } : undefined, }); - if (!response.ok) throw new Error("Failed to fetch analytics summary"); + if (!response.ok) + throw new Error( + `Failed to fetch analytics summary (${response.status})`, + ); return response.json(); },🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@frontend/src/services/api.ts` around lines 76 - 83, Update getAnalyticsSummary to include response.status in the thrown error when the fetch response is not OK, preserving the existing analytics-summary context so callers such as Client.tsx can distinguish 401 authentication failures from 5xx errors.frontend/src/app/dashboard/analytics/Client.tsx (2)
187-246: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueCharts render a blank 300px box when a distribution is empty.
HeatmapTable/FailuresTablehave empty states, but the three Recharts cards do not. On a fresh deployment with no telemetry yet, users get three empty boxes with no explanation. Consider a small wrapper that renders a "No data yet" message when the series is empty.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@frontend/src/app/dashboard/analytics/Client.tsx` around lines 187 - 246, Update the GPU, OS, and Python chart cards in the analytics dashboard to detect empty series before rendering Recharts components and show a concise “No data yet” empty state instead. Apply the same behavior consistently using the existing gpuData, osData, and pythonData values, while preserving the current charts for non-empty data.
260-266: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valuePage-local
AuthProviderduplicates auth state.Mounting the provider here scopes token state to this route, so it re-initializes on every navigation into the page and won't share state with any future global provider. Fine as the documented follow-up, but worth tracking so this wrapper is removed once the provider is hoisted to the root layout — otherwise you'll end up with nested providers.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@frontend/src/app/dashboard/analytics/Client.tsx` around lines 260 - 266, Track the page-local AuthProvider wrapper in Client for removal when authentication is hoisted to the root layout; then render AnalyticsDashboard directly to avoid nested providers and preserve shared auth state.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/security-scan.yml:
- Line 35: Update the pip-audit command in the security scan workflow to restore
the --strict option alongside --desc and --skip-editable, ensuring uncollectable
non-editable dependencies fail the scan while editable packages remain excluded.
In `@cli/tests/test_telemetry_gate.py`:
- Around line 72-78: Update test_never_posts_when_disabled_quiet_false to
inspect capsys after _send_report and assert the captured stdout contains the
telemetry-disabled notice. Keep the existing mock_httpx.assert_not_called()
assertion to preserve verification that no POST occurs.
- Around line 85-90: Isolate test_posts_when_unset from repository or CI .env
files by running _send_report with an empty temporary working directory or by
patching the CliSettings settings source. Preserve the deleted
ENVFORAGE_TELEMETRY process variable and ensure the test always exercises the
default "on" path before asserting mock_httpx was called.
In `@frontend/src/app/dashboard/analytics/Client.tsx`:
- Around line 41-48: Update the cudaVersions sorting in the analytics data
preparation near the lookup and maxCount calculations to compare version strings
by numeric segments rather than default lexicographic order, ensuring versions
such as 12.9 precede 12.10 while preserving the existing unique-version and
heatmap ordering behavior.
- Around line 153-173: Update the analytics fetch useEffect to clear the
previous error and reset loading before each request, so refetches can display
progress and recover from earlier failures. In the catch handler, narrow the
unknown rejection value before accessing its message, providing a safe fallback
for non-Error values. Preserve the cancelled guard and existing success,
failure, and cleanup behavior.
---
Nitpick comments:
In `@frontend/src/app/dashboard/analytics/Client.tsx`:
- Around line 187-246: Update the GPU, OS, and Python chart cards in the
analytics dashboard to detect empty series before rendering Recharts components
and show a concise “No data yet” empty state instead. Apply the same behavior
consistently using the existing gpuData, osData, and pythonData values, while
preserving the current charts for non-empty data.
- Around line 260-266: Track the page-local AuthProvider wrapper in Client for
removal when authentication is hoisted to the root layout; then render
AnalyticsDashboard directly to avoid nested providers and preserve shared auth
state.
In `@frontend/src/app/dashboard/analytics/page.tsx`:
- Around line 4-10: Update the metadata export for the analytics dashboard to
include robots directives that disable indexing and link following, while
preserving the existing title, description, and canonical URL in metadata.
In `@frontend/src/services/api.ts`:
- Around line 76-83: Update getAnalyticsSummary to include response.status in
the thrown error when the fetch response is not OK, preserving the existing
analytics-summary context so callers such as Client.tsx can distinguish 401
authentication failures from 5xx errors.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 6bbd75b4-954d-483d-a974-02024809c4b0
⛔ Files ignored due to path filters (1)
frontend/package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (8)
.github/workflows/security-scan.ymlbackend/app/services/analytics.pycli/tests/test_telemetry_gate.pyfrontend/package.jsonfrontend/src/app/dashboard/analytics/Client.tsxfrontend/src/app/dashboard/analytics/page.tsxfrontend/src/services/api.tsfrontend/src/types/index.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- backend/app/services/analytics.py
| run: | | ||
| cd backend | ||
| pip-audit --strict --desc | ||
| pip-audit --desc --skip-editable |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Repository workflow snippet:"
if [ -f .github/workflows/security-scan.yml ]; then
cat -n .github/workflows/security-pip-audit.yml || true
echo "---"
cat -n .github/workflows/security-scan.yml | sed -n '1,120p'
else
echo ".github/workflows/security-scan.yml not found"
fi
echo "--- pip-audit option docs in repo/requirements if present ---"
rg -n "pip-audit|strict|skip-editable|AuditError|collection" .github requirements* pyproject.toml setup.cfg setup.py 2>/dev/null || true
echo "--- package metadata for pip-audit if installed ---"
python3 - <<'PY'
import subprocess, sys
try:
out = subprocess.run([sys.executable, "-m", "pip", "show", "pip-audit"], text=True, capture_output=True, check=True)
print(out.stdout)
except subprocess.CalledProcessError as e:
print("pip-audit package not installed")
print(e.stdout.decode(errors="replace"))
print(e.stderr.decode(errors="replace"))
# Read docs/help snippets from installed command, but do not execute repository code
try:
proc = subprocess.run([sys.executable, "-m", "pip_audit", "-h"], text=True, capture_output=True, timeout=20)
print("pip-audit help returncode:", proc.returncode)
for needle in ["Strict", "strict", "skip-editable", "editable", "AuditError"]:
lines = [line for line in proc.stdout.splitlines() if needle.lower() in line.lower()]
if lines:
print(f"--- hits for {needle!r} ---")
for line in lines[:10]:
print(line)
except FileNotFoundError:
print("pip-audit command not installed")
except Exception as exc:
print("Error reading help:", exc)
PYRepository: rishabh0510rishabh/EnvForage
Length of output: 3635
🌐 Web query:
pip-audit documentation --strict --skip-editable dependency collection failed AuditError
💡 Result:
The error "dependency collection failed" combined with the --strict flag in pip-audit occurs because --strict forces pip-audit to treat any skipped dependency or collection issue as a fatal error [1][2][3]. When you use --skip-editable, pip-audit is instructed to skip packages marked as editable [2][4]. However, when combined with --strict, the tool interprets these skipped packages as a failure of the "all or nothing" audit requirement [1]. The --strict flag is designed specifically to ensure that if any dependency cannot be audited—for any reason, including being an editable package—the entire process fails rather than proceeding with a partial audit [1][2]. If you wish to audit your project while ignoring editable dependencies without triggering a fatal error, you should remove the --strict flag [1]. If you encounter this error and believe it is not due to editable packages (for example, if a dependency is missing from PyPI or is unpinned), pip-audit will also fail under --strict mode because it cannot resolve or verify that dependency [5][6]. To resolve these issues, ensure all dependencies are resolvable and available in your configured index [5]. If you are using --require-hashes alongside --skip-editable, ensure your environment configuration is compatible, as pip-audit has historically faced issues where editable dependencies still trigger hash validation requirements [7].
Citations:
- 1: Despite --skip-editable, --strict still fails due to editable packages pypa/pip-audit#664
- 2: https://github.com/pypa/pip-audit/blob/fe414d39/pip_audit/_cli.py
- 3: https://pypi.org/project/pip-audit/2.5.0/
- 4: https://github.com/pypa/pip-audit/blob/fe414d39/pip_audit/_dependency_source/requirement.py
- 5: does not work when one of many dependencíes is not present on PyPI pypa/pip-audit#157
- 6: Cannot include git repo as a dependency pypa/pip-audit#601
- 7: Bug: Hash required for editable dependency even with skip-editable pypa/pip-audit#1024
🌐 Web query:
pip-audit GitHub README or docs strict skip-editable collection failed exit code
💡 Result:
In pip-audit, the --strict flag causes the audit to fail with a non-zero exit code if any dependency collection fails [1][2]. A common point of confusion is how this interacts with the --skip-editable flag [3]. Even when --skip-editable is used, pip-audit may still fail in --strict mode if it encounters an editable package [3]. The tool's maintainers have clarified that this behavior is intentional: --skip-editable instructs the tool to bypass the auditing step for those packages, but it does not remove them from the dependency collection process [3]. In --strict mode, the tool is designed to fail if any dependency is skipped or fails collection for any reason [3][4]. Consequently, if your environment contains editable packages and you use --strict, the tool will trigger a failure because it cannot fully "collect" those dependencies according to its strict policy, regardless of the --skip-editable flag [3]. There is no current flag to suppress this specific failure while maintaining --strict for other dependency issues [3]. Summary of relevant flags: --strict (-S): Fails the entire audit if dependency collection fails on any dependency [1][2]. --skip-editable: Marks editable packages to be skipped during the actual auditing/scanning phase, but they are still processed during the initial dependency collection phase [3][5]. If you are encountering this issue, you may need to remove --strict from your command or accept that editable packages are incompatible with strict audit requirements in the current version of pip-audit [3].
Citations:
- 1: https://github.com/pypa/pip-audit/blob/main/README.md
- 2: https://github.com/pypa/pip-audit
- 3: Despite --skip-editable, --strict still fails due to editable packages pypa/pip-audit#664
- 4: https://github.com/pypa/pip-audit/blob/fe414d39/pip_audit/_cli.py
- 5: https://github.com/pypa/pip-audit/blob/fe414d39/pip_audit/_dependency_source/requirement.py
🌐 Web query:
pip-audit source AuditError strict skip_editable dependency_source.py
💡 Result:
In the pip-audit project, AuditError is a base exception class, and the combination of --strict and --skip-editable controls how the tool handles non-auditable dependencies [1][2][3]. The --strict flag (or -S) is designed to force pip-audit to fail the entire audit if dependency collection fails for any reason [4][5]. Without --strict, pip-audit will typically skip non-auditable dependencies (such as those not found on PyPI or those marked as editable) and proceed with a warning, but with --strict enabled, these skipped dependencies are promoted to fatal errors [6][2]. The --skip-editable flag is used to explicitly ignore packages installed in editable mode [4][3]. When --skip-editable is used alongside --strict, it creates a specific interaction: - If a dependency is marked as editable and --skip-editable is present, pip-audit intends to skip that dependency [2]. - However, if the logic triggers a state where the dependency collection is considered "skipped," the presence of --strict may cause the audit to fail anyway, as it treats any skipped dependency as a fatal error [6][2]. The file dependency_source.py (and related logic in _cli.py) handles the collection of dependencies [1][3]. When auditing, if a dependency cannot be resolved or falls into an excluded category, a skip reason is attached to the dependency [3]. In the CLI entry point, the tool iterates through collected dependencies and checks if --strict is enabled; if it is, the tool calls a fatal error handler whenever a dependency has a skip reason [3]. This mechanism ensures that users who require strict, "all or nothing" auditing—particularly in CI/CD environments—can ensure no dependencies were unintentionally bypassed [6][2].
Citations:
- 1: https://tessl.io/registry/tessl/pypi-pip-audit/2.9.0/files/docs/dependency-sources.md
- 2: Despite --skip-editable, --strict still fails due to editable packages pypa/pip-audit#664
- 3: https://github.com/pypa/pip-audit/blob/fe414d39/pip_audit/_cli.py
- 4: https://github.com/pypa/pip-audit/blob/main/README.md
- 5: https://pypi.org/project/pip-audit/2.10.0/
- 6: Bug: Can't disable dependency that's not on PYPI pypa/pip-audit#874
🌐 Web query:
pip-audit GitHub README --strict dependency collection fails skipped dependencies exit 1
💡 Result:
In pip-audit, the --strict flag is used to force a failure (exit code 1) if the tool fails to collect or audit any dependency [1][2][3]. When --strict is enabled, any dependency that would normally be skipped (e.g., because it cannot be found on PyPI or is marked as an editable installation) is treated as a fatal error, causing the entire process to exit with a non-zero status [3][4]. The intended behavior of --strict is to provide an "all or nothing" guarantee for audits, ensuring that users are alerted if any part of their dependency tree could not be fully resolved or audited [4]. If you are encountering this error and wish to avoid it while still auditing your other packages, you can remove the --strict flag [3][4]. If you need to skip specific dependencies without failing, you should check for other available options, such as --skip-editable, which can be used to handle specific cases like editable packages without triggering a fatal error (unless --strict is also present) [4].
Citations:
- 1: https://github.com/pypa/pip-audit?tab=readme-ov-file
- 2: https://github.com/pypa/pip-audit/blob/v2.10.0/README.md
- 3: Bug: Can't disable dependency that's not on PYPI pypa/pip-audit#874
- 4: Despite --skip-editable, --strict still fails due to editable packages pypa/pip-audit#664
🌐 Web query:
pip-audit strict flag fails dependency collection failed non-zero README
💡 Result:
In pip-audit, the --strict flag is designed to ensure that dependency collection and resolution are performed without any errors or skipped packages [1][2]. When --strict is enabled, pip-audit will fail the entire audit (exiting with a non-zero exit code) if it encounters any dependency that cannot be resolved, is marked as editable (unless specifically skipped), or is otherwise not found on PyPI [1][3][4]. If you are experiencing "dependency collection failed" errors while using --strict, it is typically because one or more dependencies in your environment or requirements file fall into one of these categories: 1. Editable packages: By default, pip-audit will fail on editable packages when in strict mode [3]. If you want to continue using strict mode but avoid this, you can use the --skip-editable flag [3][5]. 2. Packages not on PyPI: If your project relies on private packages, local paths, or direct VCS (e.g., git+) URLs that cannot be resolved against PyPI, strict mode will treat these as failures [4]. 3. Environment resolution errors: Because pip-audit relies on pip for dependency resolution, any issue that prevents pip from resolving your environment's dependencies (e.g., missing dependencies or incompatible versions) will trigger a failure in strict mode [3][6]. To resolve these errors, you can either: - Remove the --strict flag if you do not require "all-or-nothing" validation [4]. - Use --skip-editable if the failure is caused by editable installations [3]. - Ensure all dependencies are resolvable via PyPI or your configured --index-url [5][4]. If you encounter non-zero exit codes that you believe should not be treated as failures (such as when vulnerabilities are found but no fix is available), note that pip-audit is designed to be noisy in these cases to alert maintainers [7]. You cannot currently configure it to ignore unfixed vulnerabilities directly through CLI flags; however, users often pipe the JSON output to tools like jq to filter results and conditionally suppress exit codes [8][9].
Citations:
- 1: https://github.com/pypa/pip-audit/blob/fe414d39/pip_audit/_cli.py
- 2: https://github.com/pypa/pip-audit/blob/main/CHANGELOG.md
- 3: Despite --skip-editable, --strict still fails due to editable packages pypa/pip-audit#664
- 4: Bug: Can't disable dependency that's not on PYPI pypa/pip-audit#874
- 5: https://github.com/pypa/pip-audit/
- 6: Bug: doesn't work without pip installed pypa/pip-audit#842
- 7: CLI: add a flag to exit with non-zero code only for fixable vulnerabilities pypa/pip-audit#670
- 8: https://pypi.org/project/pip-audit/2.10.0/
- 9: https://github.com/pypa/pip-audit/blob/main/README.md
🌐 Web query:
"pip-audit --strict" "dependency collection failed"
💡 Result:
The error "dependency collection failed" when using pip-audit --strict occurs because the --strict flag forces pip-audit to treat any skipped dependency—such as those it cannot audit—as a fatal error that terminates the process with a non-zero exit code [1][2][3]. In standard mode (without --strict), pip-audit will warn you about dependencies it cannot audit but will continue the process [1][2]. When --strict is enabled, these warnings are promoted to fatal errors [2]. Common causes for this failure include: 1. Packages Not on PyPI: pip-audit relies on the PyPI vulnerability database [4]. If a dependency is a local package, a private package, or hosted exclusively on a non-PyPI index, pip-audit cannot audit it and will skip it [1][5][4]. 2. Editable Installs: Packages installed in "editable" mode (pip install -e .) often cannot be audited because they are local source trees rather than standard distributions [2][6][3]. 3. Version Mismatches: If a specific version of a package is only available on a private index (e.g., 2.0.0+cpu), pip-audit may fail to find that exact version on PyPI [4]. To resolve or mitigate this error, consider the following strategies: * Remove --strict: If your environment contains expected non-auditable dependencies (like local/private packages), removing the --strict flag is the most direct solution [1][5][3]. You will still receive warnings, but the audit will complete [2]. * Use --skip-editable: If your only issue is with editable packages, you can use the --skip-editable flag [7]. Note that --strict will still cause the process to fail if other non-auditable dependencies (like private packages) exist, so this is often used in combination with removing --strict if multiple types of non-auditable dependencies are present [2][3]. * Audit a Virtual Environment: Ensure you are running pip-audit within an active virtual environment that has all dependencies installed [8]. This is generally more reliable than running it against a requirements file, as it avoids complex dependency resolution issues [8]. * Verify Dependencies: Ensure all dependencies are published to a location accessible by pip-audit if you require strict enforcement, or accept that certain internal/local packages will always trigger this error in strict mode [1][6]. Top results: [1], [2], [5], [3], [7]
Citations:
- 1: Bug: Can't disable dependency that's not on PYPI pypa/pip-audit#874
- 2: Despite --skip-editable, --strict still fails due to editable packages pypa/pip-audit#664
- 3: django-getpaid/python-getpaid-payu@00e7278
- 4: Failure to find dependency that was installed from extra index url pypa/pip-audit#599
- 5: vardhjain/Ethiclens@cbe7b53
- 6: ci: add hardened pip-audit dependency-vulnerability gate ToxMCP/admetlab-mcp#6
- 7: https://pypi.org/project/pip-audit/
- 8: pypi_provider asks for bogus requirement pkg_resources==0.0.0 pypa/pip-audit#210
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect a stable pip-audit implementation/source directly from the pypi package tarball
# so this does not depend on the repository environment.
python3 - <<'PY'
import sys, urllib.request, tarfile, io
url = sys.argv[1] if len(sys.argv) > 1 else "https://files.python.io/packages/packages/2.10.0/pip-audit/pip_audit-2.10.0.tar.gz"
print("Fetching:", url)
data = urllib.request.urlopen(url, timeout=20).read()
with tarfile.open(fileobj=io.BytesIO(data), mode="r:gz") as tf:
names = [n for n in tf.getnames() if n.endswith((".py", ".md")) and ("README" in n or "_cli.py" in n or "dependency_sources" in n)]
for name in names[:20]:
print("FILE", name)
print("\nRelevant strings:")
text = data.decode("utf-8", errors="replace")
for term in ["--strict", "strict", "skip-editable", "skip_editable", "AuditError", "dependency collection failed", "dependency collection failed"]:
print(f"\n--- {term!r} ---")
hits = [line for line in text.splitlines() if term in line]
for line in hits[:40]:
print(line)
PYRepository: rishabh0510rishabh/EnvForage
Length of output: 2545
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
import urllib.request, tarfile, io, sys, ssl
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
url = "https://pypi.org/pypi/pip-audit/2.9.0/json"
data = urllib.request.urlopen(url, context=ctx, timeout=20).read().decode()
print("json", data[:500])
PY
python3 - <<'PY'
import urllib.request, tarfile, io, ssl
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
urls = [
"https://files.pythonhosted.org/packages/2.9.0/pip-audit/pip_audit-2.9.0.tar.gz",
"https://files.pythonhosted.org/packages/2.8.0/pip-audit/pip_audit-2.8.0.tar.gz",
]
import ast, re
for url in urls:
print("FETCH", url)
data = urllib.request.urlopen(url, context=ctx, timeout=30).read()
with tarfile.open(fileobj=io.BytesIO(data), mode="r:gz") as tf:
names = [n for n in tf.getnames() if n.endswith((".py", ".md"))]
for path in names:
if path.endswith((".py", ".md")):
try:
text = tf.extractfile(path).read().decode("utf-8", errors="replace")
except Exception:
continue
for term in ["--strict", "strict", "skip-editable", "skip_editable", "AuditError"]:
if term in text:
lines = text.splitlines()
for i, line in enumerate(lines, 1):
if term in line:
start=max(1,i-4); end=min(len(lines),i+4)
print("\nFILE", path, "line", i, term)
for n,l in enumerate(lines[start-1:end], start):
print(f"{n:4}: {l}")
print("---")
PYRepository: rishabh0510rishabh/EnvForage
Length of output: 1544
🌐 Web query:
github.com pypa pip_audit README --strict skip-editable dependency collection failed pip audit strict
💡 Result:
In pip-audit, the --strict option and --skip-editable option interact such that --strict treats any skipped dependency (including those skipped via --skip-editable) as a failure [1][2]. When you run pip-audit --strict --skip-editable, the following occurs: 1. The --skip-editable flag identifies packages marked as editable and categorizes them as "skipped dependencies" rather than auditing them [1][3][4]. 2. The --strict flag is designed to fail the entire audit process if dependency collection fails or if any dependency is skipped for any reason [5][1][6]. 3. Consequently, the combination of these two flags results in the tool exiting with a non-zero (failure) status because an editable package was "skipped" [1][2]. This behavior is intentional by the maintainers, as strict mode is meant to ensure that every dependency in the environment is successfully audited; skipping a dependency, even explicitly, violates the "strict" requirement [1]. If you need to avoid this failure while still using --skip-editable, you must remove the --strict flag [2]. Note that even without --strict, pip-audit will still return a non-zero exit code if it detects actual vulnerabilities in the non-skipped dependencies [2].
Citations:
- 1: Despite --skip-editable, --strict still fails due to editable packages pypa/pip-audit#664
- 2: golikovichev/secure-log2test@f0b13da
- 3: https://github.com/pypa/pip-audit/blob/fe414d39/pip_audit/_dependency_source/requirement.py
- 4: https://github.com/pypa/pip-audit/blob/fe414d39/test/dependency_source/test_pip.py
- 5: https://github.com/pypa/pip-audit/blob/main/README.md
- 6: https://github.com/pypa/pip-audit/blob/fe414d39/pip_audit/_cli.py
Keep --strict on the dependency audit unless collection failures are handled separately.
Without --strict, skipped or uncollectable non-editable dependencies can leave the scan incomplete while still passing CI. The current command only excludes editable packages; restore --strict so other collection failures are treated as required failures.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/security-scan.yml at line 35, Update the pip-audit command
in the security scan workflow to restore the --strict option alongside --desc
and --skip-editable, ensuring uncollectable non-editable dependencies fail the
scan while editable packages remain excluded.
| async def test_never_posts_when_disabled_quiet_false(self, report, mock_httpx, monkeypatch, capsys): | ||
| """quiet=False should print a notice but still not POST.""" | ||
| monkeypatch.setenv("ENVFORAGE_TELEMETRY", "off") | ||
|
|
||
| await _send_report(report, "http://localhost:8000", quiet=False) | ||
|
|
||
| mock_httpx.assert_not_called() |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Assert the promised disabled notice.
The test description says quiet=False prints a notice, but capsys is unused. Assert the captured output contains the telemetry-disabled message so that UX regression is detected.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@cli/tests/test_telemetry_gate.py` around lines 72 - 78, Update
test_never_posts_when_disabled_quiet_false to inspect capsys after _send_report
and assert the captured stdout contains the telemetry-disabled notice. Keep the
existing mock_httpx.assert_not_called() assertion to preserve verification that
no POST occurs.
| async def test_posts_when_unset(self, report, mock_httpx, monkeypatch): | ||
| monkeypatch.delenv("ENVFORAGE_TELEMETRY", raising=False) | ||
|
|
||
| await _send_report(report, "http://localhost:8000", quiet=True) | ||
|
|
||
| mock_httpx.assert_called_once() |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Isolate the “unset” default from .env.
CliSettings loads .env; deleting only the process variable means a local or CI .env containing ENVFORAGE_TELEMETRY=off makes this test exercise the disabled path instead of the default "on" path. Run this case from an empty temporary directory or patch the settings source.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@cli/tests/test_telemetry_gate.py` around lines 85 - 90, Isolate
test_posts_when_unset from repository or CI .env files by running _send_report
with an empty temporary working directory or by patching the CliSettings
settings source. Preserve the deleted ENVFORAGE_TELEMETRY process variable and
ensure the test always exercises the default "on" path before asserting
mock_httpx was called.
| const cudaVersions = Array.from( | ||
| new Set(data.map((d) => d.cuda_version)), | ||
| ).sort(); | ||
| const gpuNames = Array.from(new Set(data.map((d) => d.gpu_name))).sort(); | ||
| const lookup = new Map( | ||
| data.map((d) => [`${d.cuda_version}|${d.gpu_name}`, d.count]), | ||
| ); | ||
| const maxCount = Math.max(...data.map((d) => d.count)); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Version axis sorts lexicographically.
.sort() on version strings orders "12.10" before "12.9", so the heatmap columns/rows will be misordered once double-digit minors appear. A numeric-segment comparator fixes it.
♻️ Proposed fix
+ const compareVersions = (a: string, b: string) => {
+ const pa = a.split(".").map(Number);
+ const pb = b.split(".").map(Number);
+ for (let i = 0; i < Math.max(pa.length, pb.length); i++) {
+ const diff = (pa[i] ?? 0) - (pb[i] ?? 0);
+ if (!Number.isNaN(diff) && diff !== 0) return diff;
+ }
+ return a.localeCompare(b);
+ };
const cudaVersions = Array.from(
new Set(data.map((d) => d.cuda_version)),
- ).sort();
+ ).sort(compareVersions);📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const cudaVersions = Array.from( | |
| new Set(data.map((d) => d.cuda_version)), | |
| ).sort(); | |
| const gpuNames = Array.from(new Set(data.map((d) => d.gpu_name))).sort(); | |
| const lookup = new Map( | |
| data.map((d) => [`${d.cuda_version}|${d.gpu_name}`, d.count]), | |
| ); | |
| const maxCount = Math.max(...data.map((d) => d.count)); | |
| const compareVersions = (a: string, b: string) => { | |
| const pa = a.split(".").map(Number); | |
| const pb = b.split(".").map(Number); | |
| for (let i = 0; i < Math.max(pa.length, pb.length); i++) { | |
| const diff = (pa[i] ?? 0) - (pb[i] ?? 0); | |
| if (!Number.isNaN(diff) && diff !== 0) return diff; | |
| } | |
| return a.localeCompare(b); | |
| }; | |
| const cudaVersions = Array.from( | |
| new Set(data.map((d) => d.cuda_version)), | |
| ).sort(compareVersions); | |
| const gpuNames = Array.from(new Set(data.map((d) => d.gpu_name))).sort(); | |
| const lookup = new Map( | |
| data.map((d) => [`${d.cuda_version}|${d.gpu_name}`, d.count]), | |
| ); | |
| const maxCount = Math.max(...data.map((d) => d.count)); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@frontend/src/app/dashboard/analytics/Client.tsx` around lines 41 - 48, Update
the cudaVersions sorting in the analytics data preparation near the lookup and
maxCount calculations to compare version strings by numeric segments rather than
default lexicographic order, ensuring versions such as 12.9 precede 12.10 while
preserving the existing unique-version and heatmap ordering behavior.
| useEffect(() => { | ||
| if (authLoading) return; | ||
|
|
||
| let cancelled = false; | ||
|
|
||
| api | ||
| .getAnalyticsSummary(accessToken ?? undefined) | ||
| .then((result) => { | ||
| if (!cancelled) setData(result); | ||
| }) | ||
| .catch((err: Error) => { | ||
| if (!cancelled) setError(err.message); | ||
| }) | ||
| .finally(() => { | ||
| if (!cancelled) setLoading(false); | ||
| }); | ||
|
|
||
| return () => { | ||
| cancelled = true; | ||
| }; | ||
| }, [accessToken, authLoading]); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Stale error survives a successful refetch.
The effect re-runs when accessToken changes (e.g. token arrives after auth resolves, or a refresh), but error is never cleared. If the first attempt fails, the second attempt sets data while error stays non-null, and line 179's error || !data keeps rendering the failure screen forever. loading is likewise never reset, so refetches show no spinner.
Also, err in .catch isn't guaranteed to be an Error; narrow it before reading .message.
🐛 Proposed fix
useEffect(() => {
if (authLoading) return;
let cancelled = false;
+ setLoading(true);
+ setError(null);
api
.getAnalyticsSummary(accessToken ?? undefined)
.then((result) => {
if (!cancelled) setData(result);
})
- .catch((err: Error) => {
- if (!cancelled) setError(err.message);
- })
+ .catch((err: unknown) => {
+ if (!cancelled)
+ setError(err instanceof Error ? err.message : String(err));
+ })
.finally(() => {
if (!cancelled) setLoading(false);
});📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| useEffect(() => { | |
| if (authLoading) return; | |
| let cancelled = false; | |
| api | |
| .getAnalyticsSummary(accessToken ?? undefined) | |
| .then((result) => { | |
| if (!cancelled) setData(result); | |
| }) | |
| .catch((err: Error) => { | |
| if (!cancelled) setError(err.message); | |
| }) | |
| .finally(() => { | |
| if (!cancelled) setLoading(false); | |
| }); | |
| return () => { | |
| cancelled = true; | |
| }; | |
| }, [accessToken, authLoading]); | |
| useEffect(() => { | |
| if (authLoading) return; | |
| let cancelled = false; | |
| setLoading(true); | |
| setError(null); | |
| api | |
| .getAnalyticsSummary(accessToken ?? undefined) | |
| .then((result) => { | |
| if (!cancelled) setData(result); | |
| }) | |
| .catch((err: unknown) => { | |
| if (!cancelled) | |
| setError(err instanceof Error ? err.message : String(err)); | |
| }) | |
| .finally(() => { | |
| if (!cancelled) setLoading(false); | |
| }); | |
| return () => { | |
| cancelled = true; | |
| }; | |
| }, [accessToken, authLoading]); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@frontend/src/app/dashboard/analytics/Client.tsx` around lines 153 - 173,
Update the analytics fetch useEffect to clear the previous error and reset
loading before each request, so refetches can display progress and recover from
earlier failures. In the catch handler, narrow the unknown rejection value
before accessing its message, providing a safe fallback for non-Error values.
Preserve the cancelled guard and existing success, failure, and cleanup
behavior.
1 participant
Description
Adds the telemetry Analytics Dashboard page under
/dashboard/analytics, visualizing aggregated CLI agent diagnostics: GPU distribution, OS distribution, Python version histogram, CUDA version × GPU heatmap, and top compatibility failures. Wires the page to the existing/api/v1/analytics/summarybackend endpoint.Related Issues
Resolves #1050
Changes Made
app/dashboard/analytics/page.tsxandClient.tsxwith 5 visualizations (Recharts: GPU pie chart, OS donut chart, Python version bar chart, CUDA×GPU heatmap table, compatibility failures table)rechartsdependencyservices/api.tsgetAnalyticsSummary()to accept and forward an optional bearer token, since/analytics/summaryrequires authentication (Depends(get_current_user)on the backend)<AuthProvider>so it can safely calluseAuth()— note:AuthProvideris not currently mounted at the root layout anywhere in the app, so this page instantiates its own instance. This should be revisited once app-wide auth is wired up properly (separate, larger task).Verification
pytest tests/successfullySafetyFilterManually verified:
rechartsinstalls cleanly with no vulnerabilitiesAnalyticsSummary/CudaHeatmapEntrytypes andgetAnalyticsSummary()API method already existed and matched the component's expected shape/api/v1/analytics/summaryis registered and returns401 INVALID_TOKENas expected when unauthenticated (confirmed via directcurl)/dashboard/analyticsin the browser: page renders cleanly, shows the expected error state (Failed to load analytics: Failed to fetch analytics summary) when no auth session exists, since there's no full login flow in the app yetnpm run buildpasses with no TypeScript errorsKnown follow-up (not in scope for this PR): the app has no working end-to-end auth flow yet —
AuthProviderisn't mounted globally, and/auth/refreshreturns404on the backend. Once auth is wired up app-wide, this dashboard should work without further changes.Documentation
docs/FEATURES.md(if adding a feature/profile)CHANGELOG.mdSummary by CodeRabbit
ENVFORAGE_TELEMETRY=off(no report is sent); a console notice is shown unless quiet mode is enabled.