Skip to content

Add default-on CLI telemetry and lifecycle events - #97

Merged
woud420 merged 11 commits into
masterfrom
codex/eng-200-create-telemetry
Jul 20, 2026
Merged

Add default-on CLI telemetry and lifecycle events#97
woud420 merged 11 commits into
masterfrom
codex/eng-200-create-telemetry

Conversation

@woud420

@woud420 woud420 commented Jul 19, 2026

Copy link
Copy Markdown
Owner

Primary changes

  • Enable pseudonymous telemetry by default for eligible installed CLI runs with configured delivery. Missing state remains read-only until the first eligible event lazily creates a stable UUIDv4 outside the replaceable application payload.
  • Preserve persistent telemetry status|enable|disable|reset-id controls plus DO_NOT_TRACK, KICKSTART_TELEMETRY_DISABLED, CI, pytest, evaluation, and source-checkout development suppressions.
  • Keep provider-neutral telemetry and PostHog request DTOs under src/model/dto/; isolate PostHog behind a replaceable sink.
  • Emit exactly three closed event families: scaffold_create_completed, cli_install_completed, and cli_upgrade_completed.
  • Return a typed provider-neutral upgrade result so upgrade outcomes can be classified without parsing output or serializing raw exceptions.
  • Embed the public PostHog phc_ token into trusted wheel, source-distribution, and binary builds from POSTHOG_PUBLIC_CUSTOMER_API_TOKEN.
  • Move the binary installation runbook to docs/operations/install-binaries.md, preserve ENG-10's substantive maintainer-entrypoint guidance without promising an unstable internal payload path, and document first-run opt-out commands and event-counting limits.

Reviewer walkthrough

  1. Review the public policy, exact event allowlists, and prohibited data in docs/contracts/telemetry.md, README.md, and docs/operations/install-binaries.md.
  2. Review default-on resolution, durable identity, symlink rejection, and hard suppressions in src/telemetry/state.py, src/telemetry/policy.py, and src/cli/telemetry.py.
  3. Review provider-neutral event construction in src/model/dto/telemetry.py, src/model/dto/upgrade.py, and src/telemetry/events.py.
  4. Review install/upgrade CLI boundaries and typed updater results in src/cli/main.py and src/utils/updater.py.
  5. Review the isolated provider mapping in src/model/dto/posthog.py, src/telemetry/posthog.py, and src/telemetry/reporter.py.
  6. Review trusted artifact staging and verification in scripts/embed_posthog_token.py, scripts/verify_embedded_telemetry_artifact.py, and the GitHub workflows.

Correctness and invariants

  • Token configuration enables delivery but never overrides a persisted or process-level opt-out.
  • A missing state file is default-enabled without an ID; telemetry status does not create state or send an event.
  • The first eligible event with a configured sink atomically creates the ID. Existing IDs survive CLI upgrades.
  • Missing delivery configuration uses a no-op sink and creates no telemetry state.
  • Malformed, unreadable, busy, symlinked, or unwritable state fails closed.
  • Names, paths, repository metadata, URLs, raw arguments, environment values, generated content, checksums, and exception text are excluded from event properties.
  • Delivery is best-effort, bounded to one request with a two-second timeout, and cannot alter command output, exit status, or generated files.
  • Install --check, uninstall, help, version, completion, and telemetry-control commands emit no usage event.
  • cli_install_completed measures only kickstart install; cli_upgrade_completed measures only kickstart upgrade. They are not universal installation/upgrade totals.
  • A PATH-only install change reports success; an unresolved requested PATH update reports non-fatal partial_success/path_update; a true binary/PATH no-op reports no_change.
  • The checked-in build module is tokenless. phx_ personal keys and other read-capable credentials are rejected.
  • Rebasing onto merged PR Document the canonical local kickstart entrypoint #98 preserves ENG-10's substantive Local Development Entrypoint guidance while this PR relocates the complete runbook under docs/operations/. The physical payload path remains an implementation detail; the pre-existing updater nesting defect is tracked separately in ENG-204.

Testing and QA

  • After rebasing onto dd0c284, CI=true make check passes Ruff, mypy, type hygiene, import hygiene, template audit, and all 784 tests.
  • cd website && bun run check passes TypeScript validation and all 18 website tests.
  • git diff --check origin/master...HEAD passes, and git range-diff reports all ten telemetry commits patch-equivalent to their pre-rebase versions.
  • All 22 current-commit GitHub checks pass after audit round 2, including the Python 3.12-3.14 Linux/macOS matrix, integration generation, website, coverage, and Codecov gates.
  • make coverage: 95.8% overall. Telemetry DTOs, events, policy, and reporter are at 100%; state is 93.9%; updater is 98.7%.
  • Regression coverage includes default-on lazy identity, persistent and hard opt-outs, concurrent disable precedence, no-op delivery, malformed/symlinked state, exact property allowlists, install/upgrade success and failure classes, PATH-only/no-op/unresolved outcomes, cancellation, and transport containment.
  • Actions run 29702698190 previously built and verified production-secret wheel/sdist plus Linux x64, Linux arm64, and macOS arm64 binaries.
  • The signed-in PostHog Live Events screenshot confirms receipt of scaffold_create_completed for synthetic distinct ID cc3e0145-c8a5-41de-ab97-f7a3ce105967.
  • The initial GitHub run exposed inherited CI=true in two default-status tests; commit 04c639a isolates that test environment, and CI=true make check passes locally after the rebase.
  • Audit round 2 corrected an inherited exact-layout claim without changing updater behavior; CI=true make check still passes all 784 tests, and the focused consistency re-audit reports zero remaining findings.
  • Two independent telemetry audit passes report zero remaining findings after 5235f75 and 24d1076 fixed symlinked-state handling and PATH outcome classification.

Risk / Rollout

  • Official release artifacts require the repository secret and fail loudly when it is absent or malformed.
  • Direct Git source installs remain tokenless unless POSTHOG_PUBLIC_CUSTOMER_API_TOKEN is exported at runtime.
  • Existing packaged-artifact evidence predates the default-on/install/upgrade follow-up. Production-shaped receipt checks for the two new event families remain part of ENG-201 release verification.
  • Release, dashboard, deletion-runbook, observation-window, and deletion-contact work remains in ENG-201.
  • The first-party relay is intentionally deferred to ENG-199 until direct-capture evidence justifies it.
  • ENG-10 remains complete through merged PR Document the canonical local kickstart entrypoint #98; this PR preserves and relocates its substantive runbook guidance but does not claim to resolve the ticket.
  • The pre-existing self-updater nesting defect found during the rebase audit is deferred to ENG-204; this PR deliberately does not change updater behavior.

Resolves ENG-197
Resolves ENG-198
Refs ENG-199
Resolves ENG-200
Refs ENG-201
Resolves ENG-202
Resolves ENG-203

@codecov

codecov Bot commented Jul 19, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.70968% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.80%. Comparing base (dd0c284) to head (4de5c85).

Files with missing lines Patch % Lines
src/telemetry/state.py 93.86% 10 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master      #97      +/-   ##
==========================================
+ Coverage   94.41%   95.80%   +1.38%     
==========================================
  Files          46       59      +13     
  Lines        3167     3905     +738     
==========================================
+ Hits         2990     3741     +751     
+ Misses        177      164      -13     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@woud420 woud420 changed the title Add default-off CLI telemetry with embedded PostHog delivery Add default-on CLI telemetry and lifecycle events Jul 19, 2026
woud420 added 10 commits July 19, 2026 19:07
Persist default-off pseudonymous telemetry state outside generated projects, expose explicit user controls, and document the privacy contract.

Checks: make check (656 tests)
Refs ENG-197
Refs ENG-198
Map provider-neutral telemetry envelopes through typed PostHog DTOs, read the public project token only from the process environment, and contain all delivery failures.

Checks: make check (676 tests)
Refs ENG-202
Attempt one allowlisted terminal event for create outcomes, normalize aliases and unknown values without names or paths, suppress telemetry in tests and evals, and smoke-test the packaged controls.

Checks: make check (700 tests); run_evals.py --tier smoke; binary packaging smoke
Refs ENG-200
Suppress synthetic Cloudflare smoke events and skip descriptor chmod where the platform does not provide it.

Findings: 1, 2
Checks: make check (701 tests); bash -n scripts/ci/smoke-cloudflare-worker.sh
Refs ENG-197
Refs ENG-200
Make configured telemetry default-on with lazy, stable identity creation while preserving persistent and process-level opt-outs. Add closed install and upgrade outcome events, typed upgrade results, fail-closed symlink handling, and move the binary runbook under docs/operations.\n\nRefs ENG-197\nRefs ENG-198\nRefs ENG-200\nRefs ENG-201\nRefs ENG-202\nRefs ENG-203\n\nTests: make check (781 passed)\nCoverage: make coverage (95.8%)
The default-enabled status tests deliberately remove pytest suppression, so also neutralize GitHub Actions' inherited CI=true value.\n\nRefs ENG-198\nRefs ENG-201\n\nTests: CI=true make check (781 passed)
Classify PATH-only changes as success, unresolved requested PATH integration as partial success, and path-phase failures as partial whenever a usable binary is already present. Document and test each boundary.\n\nRefs ENG-203\n\nTests: CI=true make check (784 passed)\nCoverage: CI=true make coverage (95.8%)
@woud420
woud420 force-pushed the codex/eng-200-create-telemetry branch from 533d91e to 24d1076 Compare July 19, 2026 23:08
Correct the inherited ENG-10 wording without changing updater behavior. The existing nesting defect is tracked separately as ENG-204.

Tests: CI=true make check (784 passed)
@woud420
woud420 marked this pull request as ready for review July 20, 2026 00:12
@woud420
woud420 merged commit 705e8e9 into master Jul 20, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant