feat(#108): seeding pipeline - CLI provider, domain-rationale, infrastructure domain, concurrency

[Hareet]

Builds on #109. Enhances the memory distillation pipeline and adds the taxonomy needed to seed agent-memory/.

Pipeline

• Claude CLI provider (LLM_PROVIDER=claude-cli) — filter/distill via claude -p on the operator's subscription, no API key (requested in feat(#108): memory distillation pipeline — scraper, filter, distiller, CLI, and CI workflow #109 review).
• Resumable, concurrent runner — --concurrency (worker pool), --last N (removes the ~100-PR cap), --resume, --force.
• Global-failure batch-stop — rate-limit (429/usage/session) and auth (401) errors stop the run with a resume hint (exit 2) instead of flagging every PR.

Taxonomy / output

• domainFit + ## Domain Rationale per draft; related_workflows (CHTWorkflow += ui-extensions/nouveau-search/observability).
• New infrastructure CHTDomain, scoped to operational lifecycle (excludes data-layer internals).
• Single-sourced taxonomy: CHTDomain/CHTService/CHTWorkflow derived from the CHT_* const arrays (incl. VALID_DOMAINS); a test asserts schema.json enums match the consts.
• All 8 canonical body sections; TEMPLATE.md aligned.

Tooling

• validate-schema --pending-only/--include-pending; wired as a CI gate (corpus on PRs, drafts in the pipeline run).
• Containerized seeder + runbook; .nvmrc pins Node 22.
  ⚠️ Shared schema — see the merge-order comment below.

Testing: 528 unit tests; full backlog seeded (338 drafts, validate-schema 0 failures). Reviewed (code/security/architecture) — see follow-up checklist.

Evidence: docs/domain-taxonomy-findings.md.

Review follow-ups (tracked, non-blocking):

• Dedupe the vendored docker/ base once Dockerize cht-agent + git/credential sandbox #114 merges.
• issueUrl /issues/ fallback for PRs closing no issue
• Unify provider selection — factory vs inline OpenRouter/Anthropic in filter/distiller.

3. Merge order

#109 → main (the base pipeline)
#87 coordination — both #87 and this PR edit agent-memory/schema.json (and the 4 contacts services fixes overlap). Cleanest: #87 merges, then rebase this PR onto main so its validator sees our infrastructure/domainFit/workflow enums. If this PR merges first, #87 must rebase and keep our enum additions.
This PR → main.

[sugat009]

Review: seeding pipeline keystone

Thanks for this, it is a strong, well-structured PR and the parts that matter most are solid. The taxonomy single-sourcing is genuine (the taxonomy-schema-sync test is a real bidirectional drift lock against the live schema.json enums), the VALID_DOMAINS = CHT_DOMAINS fix means infrastructure is accepted everywhere, the Claude CLI provider has a clean injection posture (shell:false, prompt written to stdin, never argv), and it extends the merged #128 pipeline additively with no destructive changes. The docker stack also gets a lot right: no Docker socket mount (with a fail-closed runtime assertion), openssh-client purged, credentials and .git/config mounted read-only, and push blocked at several independent layers.

I went through this across a few dimensions and then re-verified the findings independently, which led me to correct a couple I had initially over-rated (noted at the end for transparency). Posting as comments to open the discussion rather than as a hard block. The one I would genuinely like to settle before merge is the first; the rest are open for debate and I am happy to be wrong on any of them.

Worth resolving before merge

1. Credential exfiltration from the interactive containers (security). In the interactive cht-agent container the ANTHROPIC_API_KEY is in the env and the Claude OAuth token is a readable mount, and the agent can run curl with no egress restriction, so a prompt-injected agent could exfiltrate either in one request. Two things I would rather discuss than assert:

• The automated run-pipeline path looks genuinely safe (filter and distill run with --disallowedTools Bash,... --max-turns 1, so Bash/curl are out of the model's context). The exposure I see is the interactive cht-agent container and the docker exec -it cht-seeder claude login session the runbook prescribes. Does that match your model, or is there a control I am missing?
• One subtlety so we do not chase a false fix: because the provider always passes --dangerously-skip-permissions, the settings.json allow list is ignored and only the deny list is enforced. So removing Bash(curl:*) from the allow list would not close this. The levers that would are network egress isolation (internal: true or an egress allowlist) or adding curl to the deny list, plus keeping secrets out of the agent-readable env. Open to other approaches.

Questions / things that look off (open)

2. Coverage-gate scope. This PR adds the whole .nycrc.json exclude list (there is none on main), and it currently excludes src/llm/providers/** (the new claude-cli.ts, which has a 354-line spec) and src/utils/domain-inference.ts. Net effect: the PR's headline new code and domain-inference.ts do not count toward the 95/85 gate. Was the exclusion meant to be this broad, or a holdover from when providers hit a live API? Since claude-cli.ts is now unit-testable offline, narrowing it would let the new code count.

3. A real usage-limit looks like it can be swallowed. If a Claude usage limit comes back as exit-0 plain-text stdout (no "type":"result" envelope), parseResponse returns it as a success (is_error:false), so the rate-limit classifier never sees it; downstream invokeForJSON then throws the generic "did not contain valid JSON object", which is a non-fatal message, so the batch keeps going against an exhausted subscription. Is that path reachable in practice with claude -p, and if so should a no-result-envelope (or usage-limit text) be treated as batch-fatal? Also probably worth confirming the distiller will not persist the limit text as a draft body.

4. --force --resume --pr <already-done> exits 0. The resume filter drops already-processed PRs before --force is consulted, so re-distilling a specific PR you have already run silently no-ops ("Nothing to process."). Intended, or should --force exempt explicitly named PRs from the resume skip?

5. domain-inference.ts enrichment prompt lists 7 domains. inferUsingLLM (181-230) enumerates 7 and says "pick one of the 7", but VALID_DOMAINS is the 9-value CHT_DOMAINS and the appended example uses infrastructure, so interoperability is never offered and the prompt contradicts its own validator. The distiller path derives the list from CHT_DOMAINS and is fine, so this reads like one spot the single-sourcing did not reach. Deliberate, or worth deriving this prompt from the const too?

6. Seeder volume + token hygiene (smaller). Two operational ones: the seeder's /home/agent/.claude named volume is seeded from the image only on first creation, so later security fixes to settings.json/CLAUDE.md will not reach an existing volume on rebuild; and init-agent-git.sh warns (does not fail) on GH_TOKEN with no actual scope check, so an accidental write-scope PAT would start normally. Both feel worth tightening; curious whether there is a reason they are warn-only.

Checked, looks low-impact (sharing so you know they were considered)

A few I initially flagged higher and walked back after re-checking, for transparency rather than as asks:

• The batch-stop regexes (/limit reached/, /quota/, /unauthorized/) are unanchored, but the realistic OS errors I worried about do not actually produce those strings (Node EMFILE is "too many open files", ENOSPC is "no space left on device"), and a real gh 401/403 is global so stopping is the right call, just with a Claude-flavored message. Low impact; the main residue is that the message can mislead an operator on a gh-token failure (it suggests claude /login).
• settings.json being agent-writable is real but hollow under skip-permissions (only the deny list is live, curl is not denied anyway, and push is blocked at other layers).
• The SIGINT/SIGTERM handler registration is fine in practice (the chains are memoized, so ~4 handlers, and a second signal terminates via Node's default action); only a narrow orphan-on-second-signal edge remains.
• extractResultEnvelope's multi-object parse is latent only, since the provider only ever passes --output-format json (a single envelope).

Overall this is close to mergeable. Mainly the credential-exfil scope and the coverage-gate scope feel worth settling first, and I am glad to dig into any of the above together.

[Hareet]

Thanks @sugat009

1. Credential exfiltration from the interactive containers (security)

I'll grab this in #133 , as that may have conflicts and was meant to over-write some of the container portion. Sped ahead to seed the data, and tried to keep them separated.

• Add curl/wget/nc//dev/tcp to the deny list in docker/agent/settings.json

2. coverage excludes providers/**

I'll fix this here. holdover and forgot to fix as was done with run-pipeline.ts

Agree on 3,4,5,6 and will grab them here.

[Hareet]

Removed cht-agent docker files and will merge #133 first.

Addressed other feedback 2-6.

[sugat009]

Re-reviewed against the actual fix code, the current .nycrc, and the seeder container. 2-6 check out, several with more hardening than I asked for: the usage/auth notice now returns an error envelope via isBatchFatalError (and the patterns do match the real CLI notices); filterResumable exempts a named --pr under --force; the inference roster is derived from CHT_DOMAINS with a DOMAIN_DESCRIPTIONS: Record<CHTDomain, string> that fails to compile on a missing domain (nicer drift-lock than I suggested); and the volume-staleness caveat is documented. Agreed on landing #133 first, and removing the vendored sandbox to layer on FROM cht-agent:local also moves the credential-exfil concern onto #114/#133 where the egress follow-up lives. Nice turnaround.

One sliver of the coverage finding is still open: src/utils/domain-inference.ts. The provider half is fixed (providers/** dropped, claude-cli.ts now gated, 81%→96%), but this file is still in the new exclude list. Since main's .nycrc has no excludes, it is gated on main today, and it has a spec, so the PR effectively turns its gate off in the same change that adds the new roster/inference logic. The domain bug is fixed and compile-locked, so this isn't a correctness risk. Is the exclusion just to dodge the live-LLM call? If so, could we drop the file-level exclude and either mock the call (the provider specs now do this) or scope an istanbul ignore to that one function, so the rest of the file stays gated?

Approving on the strength of everything else; would just like that one exclusion folded in before or shortly after merge. Carryover nit for whenever, not this PR: the rate-limit.ts patterns are still unanchored and the gh-token failure still points at claude /login.

[Hareet]

Is the exclusion just to dodge the live-LLM call?

Yeah it was, but i like your suggested approach.