research: how to understand Cortex adoption (installs, versions, retention) without being intrusive

Feature Description

We cannot currently answer basic questions about Cortex adoption: how many people have installed
it, which versions are in use, how many installs are still running a week later. The only signal
today is anecdotal — someone mentions in chat that they installed it.

This issue is research, not implementation. The goal is to understand the available options
for obtaining adoption information, what each one costs in user trust, and what the honest
trade-offs are — and to produce a recommendation for discussion. No approach should be
implemented off the back of this issue. Whatever comes out of it gets its own issue, scoped
and agreed separately.

The reason for the care: Cortex sits in the request path and decrypts the user's agent traffic.
That earns a level of trust that is very easy to spend and very hard to get back. Users are
rightly sensitive about anything leaving their machine, and a tool positioned as "see what your
coding agent actually sends" undermines its own pitch if it quietly sends things itself. The
default assumption for this research is that intrusive collection is off the table, and the
burden is on any proposal to show it does not cross that line.

It is a legitimate outcome of this issue to conclude that we should collect nothing from
installed clients, and get what we need from other sources — or to accept that some questions
simply stay unanswered.

Proposed Solution

Investigate and write up the options. Not a plan to build any of them.

1. What do we actually need to know, and why?

List the specific questions worth answering (install count? version spread? retention?
which agent integrations are used? platform mix?)
For each, state the decision it would inform. Questions that would not change any decision
should be dropped rather than carried along
Separate "would genuinely change what we build" from "nice to know" — the second category
does not justify any user-visible cost
Note what precision is actually required: an order of magnitude is often enough, and cheap
approximations may beat exact counts that need instrumentation

2. Survey the approaches, including the ones that collect nothing

Zero-collection and off-machine sources first, since these carry no user cost:

Download and release-asset counts (GitHub release API, install-script fetches, container
pulls, package-manager stats) — what they can and cannot tell us, and how badly CI and
mirrors distort them
Repo-side signals: stars, forks, clones, issue and feedback volume, questions in chat
Asking people directly: surveys, a check-in with known users, opt-in registration for
early adopters. Low-tech, honest, and possibly sufficient at our current scale
Update-check traffic, if an update check is something users would want anyway for its own
sake — and whether using it for counting is a bait-and-switch

Then approaches that do involve the client, each assessed honestly rather than advocated for:

Fully opt-in reporting (explicit affirmative action, off by default, no dark patterns)
Local-only accounting the user can voluntarily share — Cortex keeps its own numbers, nothing
leaves the machine unless someone chooses to send them
Aggregate or privacy-preserving techniques, and whether their complexity is remotely
justified at our scale
Anonymous reporting — documented here specifically so its costs are on the record. Note
that "anonymous" is a weaker claim than it sounds: IP addresses, timing and rare
configuration combinations can re-identify small populations

3. Assess each against the trust bar

What leaves the machine, in concrete terms — not a category label
Whether it can be meaningfully described to a user in one sentence they would accept
Re-identification risk, especially at small user counts where uniqueness is high
Whether it is verifiable by a suspicious user, or requires taking our word for it
Reaction if it were discovered rather than announced — a useful test of whether we are
comfortable with it
What it would do to Cortex's positioning as a transparency tool

4. Look at what comparable tools do — and how it went

How other local developer tools and proxies handle this (both the ones that collect and the
ones that deliberately do not)
Cases where telemetry damaged a developer tool's standing, and what specifically triggered
it — usually defaults, disclosure and discoverability rather than the payload itself
Any legal or policy constraints that apply to us

5. Deliverable

A written comparison of the options with the trade-offs stated plainly, including the
do-nothing option as a real candidate
A recommendation, which may be "collect nothing from clients"
If any client-side approach is recommended: what it sends, the default, the disclosure, the
off switch, and how a user verifies the claim — as a proposal for discussion, not a
decision already taken
Open questions needing a product or maintainer call, flagged rather than resolved quietly
Follow-up issues filed for whatever is agreed. Nothing ships from this issue.

Additional Context

Part of the Cortex v0.9.0 release, Pillar 2 (Observability) of rossoctl/rossoctl#2514 — here
pointed at understanding adoption of the tool itself rather than at the agent's traffic.

cortex#898 — report statistics to a central collector. Overlapping concerns: its payload is
traffic and cost rather than adoption, but it raises the same trust questions and should
probably not be designed independently of whatever this research concludes
cortex#944 / cortex#945 — macOS and Linux install + service lifecycle (where install-time
signals would originate, if any)
cortex#959 — first-run path, install to first useful reading (the drop-off we cannot currently
see)
cortex#1236 / cortex#1237 — agentop configure surfaces (relevant to the integration-mix
question)
cortex#1002 — Cortex being down can break the harness (any client-side network call would have
to avoid this failure mode entirely)

Splitting this into the two independent tracks it actually contains. They have different trust profiles and should be decided separately.

Track 1 — Server side: GitHub download counts

Status: researched, concluded, implemented. #1345 + rossoctl/rossoctl#2628.

This was the first approach in §2, and the one that collects nothing from clients — so the research question was narrow: can off-machine data answer the §1 questions, and how badly does CI distort it? It can, partially, and the distortion turned out to be the whole story. Findings below; the implementation follows from them.

Clone traffic is not adoption. 16,967 clones / 1,021 uniques over two weeks is almost entirely automation — ~1,000 workflow runs plus the Go module proxy, code-search indexers and scanners. Someone installing Cortex clones zero times; the installer uses curl and a release tarball. Also uniques dedupes per-day only, so it never meant 1,021 people. Reading this as adoption would have overstated it by three orders of magnitude.

checksums.txt is the install count. The installer fetches two tarballs per install and verifies both against one checksums file, so platform counts double-count while checksums counts once. The 1:2 ratio holds across every release with traffic (v0.8.1 66:133, v0.8.0 10:22, v0.7.0 18:37).

CI must be subtracted, not filtered. release-smoke-linux.yaml installs "exactly as a real user would (curl | sh, downloading from GitHub Releases)" — its own words — so nothing behavioural separates it from a Linux user. It is countable: one job, no matrix, one install's worth per successful run. Only success counts; 18 of 64 runs in the first sampled window were skipped and downloaded nothing.

Platform cannot split CI from people. Reading linux_amd64 as "CI" would erase every real Linux user — the failure mode worth avoiding, since it is confidently wrong rather than merely noisy.

Current numbers, v0.8.1: 66 installs − 50 CI runs ≈ 16.

Against §1 — what this answers and what it does not:

Question	Answered?
Install count	Yes, ±maintainers
Version spread	Yes, per release
Platform mix	Yes
Retention	No
Still running a week later	No
Which integrations are used	No

Hard limits, stated on the page itself: it includes maintainers and contributors (at ~16, a few of us could be most of it); it measures acquisition only; and there is no history before the first snapshot, since GitHub reports lifetime totals and a daily figure only exists as a difference between two.

Track 2 — Client side: confirming an install worked and is operational

Status: open research. This is where the original rules still apply in full — nothing ships from this track without its own scoped issue.

Track 1 structurally cannot answer this. A download says a tarball left a server — not that it extracted, that the service started, that the proxy bound its port, or that anyone pointed an agent at it. The install could fail at every step after the download and the number would look identical.

That gap matters more than the headcount: #959 is about the install-to-first-useful-reading drop-off, and it is exactly what we cannot see. A funnel of 66 downloads → ? installs → ? operational → ? still running is three unknowns deep.

What needs deciding, carried forward from §2–§4:

Does an update check belong here at all? §2 already flags the bait-and-switch risk: if users would want update notification for its own sake, the request is justified independently and the count is a side effect. If we would only add it to count things, that is the dark pattern this issue warns about. Decide the feature on its own merit first.
Local-only accounting (§2) — Cortex keeps its own numbers, nothing leaves unless someone chooses to send them. Possibly the honest shape: it answers "is my install healthy?" for the user, and a voluntary share answers ours.
Just asking (§2) — at ~16 installs, a survey or a check-in with known users may beat any instrumentation. Worth taking seriously rather than listing for completeness.
The 🐛 Cortex being down can break harness #1002 constraint is load-bearing. Cortex being down can already break the harness. Any client-side network call has to be incapable of making that worse — which rules out anything on a startup or request path, and probably anything blocking at all.
feature: Report statistics to a central collector #898 overlap — reporting to a central collector raises the same trust questions with a different payload. Per that issue, these should not be designed independently.
What does "operational" even mean without reading the user's traffic? Service-started is probably knowable; proxy-saw-a-request starts to look like usage data.

Suggested direction

Track 1 being cheap does not make Track 2 cheap, and its existence should not create pressure to close this with something. The two tracks answer different questions.

If a recommendation is useful now: local-only accounting plus asking people directly looks like the only combination that gets at retention without spending trust. An update check is defensible if it is a feature users want on its own, and indefensible as a counting mechanism wearing a feature's clothes.

Open question for a maintainer call, flagged not resolved: is "we cannot measure retention" an acceptable permanent answer? It is a legitimate outcome per the original framing, and at our current scale the alternative may cost more than the information is worth.

Assisted-By: Claude (Anthropic AI) noreply@anthropic.com

Track 2 — a proposal to discuss, not a decision

Per this issue's rules: nothing here ships without its own scoped issue. This is the §5 deliverable for the client side, narrowed by one finding.

The finding that changes the question

agentop doctor already answers "is this install working?" — locally, with zero network calls.

cmd/agentop/cmd_doctor.go iterates buildSetupSteps and checks each one as it was set up: binariesStep, pathStep, configStep, serviceStep, optionally claudeCodeStep, then cleanupStep — plus CA expiry (checkCA), unread state, and Python. It prints version and goos/arch in its header and exits 0 when there is nothing to fix, 1 otherwise.

So the capability is not missing. What is missing is that the answer never leaves the machine, by design.

That reframes Track 2 from "build a health check" to "decide whether an existing local answer should be shareable, and on whose initiative." A much smaller question, and one that can be answered without touching the request path — which matters, because #1002 (Cortex being down can break the harness) rules out anything automatic or blocking.

Proposal: `agentop doctor --share`

User-initiated, payload shown before anything is sent, no default and no background call.

$ agentop doctor --share

  rosso cortex · doctor            v0.8.1 · darwin/arm64

  ✓ binaries      ✓ config
  ✓ path          ✓ service
  ✓ claude code   ✓ CA valid (expires 2027-01-14)

  Nothing to fix.

  Would send:
    {"version":"v0.8.1","platform":"darwin/arm64",
     "checks":{"binaries":"ok","path":"ok","config":"ok",
               "service":"ok","claude_code":"ok","ca":"ok"},
     "installed_days_ago":4}

  Send this to the Cortex maintainers? [y/N]

What it is not: not on by default, not a flag that can be set once and forgotten, not reachable from startup or the request path, and not a background timer. Running the command is the consent.

Why this shape rather than telemetry: the payload is rendered before the prompt, from the same check results the user just read. There is no gap between what is displayed and what is sent, so the claim is verifiable by the user rather than by trusting our docs.

What it would and would not tell us


Install completed and is healthy	yes, for users who run it
Which step fails when one does	yes — this is the part Track 1 cannot touch
Version and platform spread	yes
Retention	partially: `installed_days_ago` on a healthy install is evidence of survival
Representative install count	no — self-selected, so a count from this is worse than Track 1's

The second row is the real prize and the reason this is worth discussing at all. #959 is about the install-to-first-reading drop-off; a ✗ service or ✗ claude code from a real user's machine names the step that lost them. Download counts cannot do that at any resolution.

Open questions for a maintainer call

Is a share prompt still a trust cost? Cortex decrypts the user's agent traffic. Even an explicit, inspectable, user-initiated send establishes that the tool has a channel to us. Someone may reasonably object to the channel existing in the binary at all, independent of its defaults.
Where would it send, and what does that endpoint then hold? This proposal deliberately stops at the client. An endpoint is infrastructure we do not have, and it has its own retention and access questions — likely overlapping feature: Report statistics to a central collector #898.
Does installed_days_ago cross a line? It is the only field that is not a current-state fact. It is also the only one that speaks to retention, which is the thing Track 1 structurally cannot measure.
Is the self-selection fatal? Only engaged users run doctor --share. For diagnosing which step fails that bias is tolerable; for anything resembling a count it is not, and the output should never be presented as one.

The cheaper option, which I would do first

At ~16-21 installs, asking the known users directly gets better data than this would, costs nothing, and carries no trust question. It also answers why someone stopped, which no payload of check results will.

My suggestion: ask first, and treat --share as worth building only if the answers show a drop-off we cannot explain by asking. It is a small change (--share formats results doctor already computes) and it will still be small later.

Status

Track 1 is closed (#1345, #1351, #1367 for a follow-up bug). Track 2 stays open research. Nothing above is agreed, and "collect nothing from clients" remains a legitimate outcome per this issue's own framing.

Assisted-By: Claude (Anthropic AI) noreply@anthropic.com

Navigation Menu

Repository navigation

research: how to understand Cortex adoption (installs, versions, retention) without being intrusive #1321

Feature Description

Proposed Solution

Additional Context

Track 1 — Server side: GitHub download counts

Track 2 — Client side: confirming an install worked and is operational

Suggested direction

Track 2 — a proposal to discuss, not a decision

The finding that changes the question

Proposal: `agentop doctor --share`

What it would and would not tell us

Open questions for a maintainer call

The cheaper option, which I would do first

Status

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

research: how to understand Cortex adoption (installs, versions, retention) without being intrusive #1321

Description

Feature Description

Proposed Solution

Additional Context

Activity

mrsabath commented on Oct 9, 2026

Track 1 — Server side: GitHub download counts

Track 2 — Client side: confirming an install worked and is operational

Suggested direction

mrsabath commented on Oct 13, 2026

Track 2 — a proposal to discuss, not a decision

The finding that changes the question

Proposal: agentop doctor --share

What it would and would not tell us

Open questions for a maintainer call

The cheaper option, which I would do first

Status

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Proposal: `agentop doctor --share`