[codex] Document Python and process provider plan

docs/README.md

@@ -40,9 +40,9 @@ Working now:
 
 Still in progress:
 
-- Provider-specific redaction fixtures.
+- Provider-specific redaction fixtures for future collectors.
 - Richer systemd and package metadata.
-- Python and native-process providers.
+- Python and native-process provider implementations.
 - Better table sorting, filtering, graph navigation, and logs UI.
 - OpenAPI docs and stable versioned API routes.
 - Safe write mode with backups and rollback. This is not enabled today.
@@ -84,6 +84,8 @@ daemon, provider, auth, deployment, or write-mode behavior.
   data flow, runtime-map model, and provider rules.
 - [architecture/PAGE_LOGIC.md](architecture/PAGE_LOGIC.md): intended UI routes,
   page behavior, and cross-page navigation.
+- [planning/PYTHON_AND_PROCESS_PROVIDERS.md](planning/PYTHON_AND_PROCESS_PROVIDERS.md):
+  read-only provider plan for future Python application and native-process collection.
 - [testing/TESTING_PLAN.md](testing/TESTING_PLAN.md): local checks, security
   tests, browser smoke tests, live-Docker tests, and manual release smoke.
 - [testing/SANDBOX_FIXTURE.md](testing/SANDBOX_FIXTURE.md): isolated labeled

docs/architecture/ARCHITECTURE.md

@@ -73,8 +73,6 @@ The map is read-only and currently contains:
   package-update/advisory metadata, but no runtime registry or advisory lookup is enabled
   today. UI update labels are derived from local model/demo data unless a future opt-in
   advisory provider is implemented.
-- Python projects discovered from common project files as a later application-provider
-  peer to npm.
 - scheduled jobs from `/etc/crontab`, `/etc/cron.d/*`, and the current user's `crontab -l` when readable.
 - PM2 apps from `pm2 jlist` when PM2 is installed.
 - tmux sessions from `tmux list-sessions` when tmux is installed and reachable.
@@ -89,6 +87,11 @@ Provider commands are fixed read-only invocations, not user-supplied shell comma
 Filesystem discovery must stay bounded by configured roots, skip dependency/build
 directories, and avoid reading secrets such as `.env` values.
 
+Planned Python application and native-process providers should plug into this same map as
+later read-only peers. The implementation plan in
+[`docs/planning/PYTHON_AND_PROCESS_PROVIDERS.md`](../planning/PYTHON_AND_PROCESS_PROVIDERS.md)
+defines safe sources, discovery bounds, omitted data, diagnostics, and follow-up slices.
+
 Kubernetes and other orchestrators should plug into this same model as additional providers, not replace the local Docker/host model. Kubernetes support should be opt-in because it needs kubeconfig or in-cluster credentials, namespace scoping, and RBAC permissions. A safe first Kubernetes provider should read namespaces, pods, services, deployments, ingress objects, persistent volume claims, and selected labels/owner references, then map them to `orchestrator_workload` nodes and edges.
 
 ## Runtime Layers
@@ -127,7 +130,10 @@ Forge (npm) -> forge.service -> tmux session -> GPT worker
 Relationship discovery should prefer explicit evidence first, such as systemd dependency
 fields, Compose labels, process working directories, package manifests, lockfiles, known
 reverse-proxy routes, and Docker network membership. Heuristics are allowed only when
-metadata records why the edge exists.
+metadata records why the edge exists. Future Python/native-process edges must include
+evidence metadata such as `proc_cwd`, `project_manifest`, or `systemd_main_pid`.
+Process-to-listener edges stay out of the first Python/native-process implementation unless
+a later security review approves a safe source that does not read `/proc/<pid>/fd` targets.
 
 ## Docker Access
 

docs/planning/IMPLEMENTATION_PLAN.md

@@ -51,14 +51,14 @@ still the runtime model, collectors, contracts, and security evidence.
 - Frontend component split, Compose scan/graph/edit-plan endpoints, CLI
   scan/validate/export, Compose/runtime correlation, override merge semantics, and
   contract compatibility tests are done.
-- Still to build before full alpha: provider-specific redaction fixtures for new
-  collectors, richer package/update/advisory behavior, Python/native-process provider
-  peers, table sorting, advanced filters, clickable graph nodes, log level filter, live
-  tail, broader browser smoke coverage, and log pagination UI.
-- Immediately after the next implementation commit lands, execute the follow-up queue in
-  [`docs/release/RELEASE_CHECKLIST.md`](../release/RELEASE_CHECKLIST.md): provider redaction fixtures,
-  package advisory/network opt-in docs, live-Docker/reverse-proxy release-host evidence,
-  and Python/native-process provider planning.
+- Still to build before full alpha: provider fixtures for future collectors, richer
+  package/update/advisory behavior, Python/native-process provider implementations, table
+  sorting, advanced filters, clickable graph nodes, log level filter, live tail, broader
+  browser smoke coverage, and log pagination UI.
+- The Python/native-process provider plan lives in
+  [`docs/planning/PYTHON_AND_PROCESS_PROVIDERS.md`](PYTHON_AND_PROCESS_PROVIDERS.md).
+  Remaining release-host evidence is tracked in
+  [`docs/release/RELEASE_CHECKLIST.md`](../release/RELEASE_CHECKLIST.md).
 - Testing plan: see [`docs/testing/TESTING_PLAN.md`](../testing/TESTING_PLAN.md).
 
 ---

docs/planning/PYTHON_AND_PROCESS_PROVIDERS.md

@@ -0,0 +1,220 @@
+# Python And Native Process Provider Plan
+
+This plan defines how DockerMap should add Python application and native-process
+providers before any collector code is written. The goal is to make these providers peers
+in the read-only runtime map without scanning the whole host, exposing secrets, or adding
+controls that can change running services.
+
+## Goals
+
+- Represent Python applications and ordinary host processes in `GET /daemon/runtime/map`.
+- Reuse the existing runtime-map concepts for services, packages, processes, edges, and
+  diagnostics.
+- Prefer explicit evidence over guesses, and explain heuristic edges with metadata.
+- Keep every source fixed, local, bounded, and read-only.
+- Keep tests runnable from fixtures without a real production host.
+
+## Non-Goals
+
+- No process start, stop, restart, signal, kill, reload, or package-install actions.
+- No user-supplied shell commands.
+- No process environment dumps.
+- No log, stdout, stderr, terminal scrollback, process memory, open-file, or file-descriptor
+  scraping.
+- No `.env`, `.pypirc`, `pip.conf`, `pip.ini`, Poetry auth config, private-index config, or
+  credential-file reads.
+- No package registry, advisory, DNS-provider, or other external-network lookup by default.
+- No recursive home-directory, virtualenv, cache, or host-wide filesystem crawl.
+
+## Contract Fit
+
+The current contracts already contain most of the vocabulary needed for a first pass:
+
+- Python application nodes can use `provider: "process"` and `type:
+  "python_application"` for the first collector, because the current contracts do not
+  include a separate `python` provider.
+- Native process nodes can use `provider: "process"` and `type: "process"` or `type:
+  "worker"` when evidence is strong enough.
+- Package metadata can use existing `package` and `package_dependency` node types with
+  `manager: "pip"` once Rust and TypeScript contract shapes are intentionally aligned.
+- Provider-neutral service fields should stay the long-term target, but the first Rust
+  implementation can keep metadata string-only until richer runtime-node fields are added
+  to Rust and fixture drift checks cover both languages.
+
+Do not add a new `python` provider enum in the first collector unless fixtures prove that
+Python project discovery cannot be represented clearly with `provider: "process"`,
+`type: "python_application"`, `package`, and `package_dependency` vocabulary. If a later
+provider enum is needed, make it a separate contract issue before collector work lands.
+
+## Data To Expose
+
+Python application nodes may expose:
+
+- Stable node id derived from safe project-relative evidence, not raw secret-bearing paths.
+- Project or package name when read from safe manifest fields.
+- Project-relative path under `DOCKERMAP_PROJECT_ROOT`.
+- Manifest type: `pyproject.toml`, `requirements.txt`, `Pipfile`, `poetry.lock`,
+  `uv.lock`, `setup.cfg`, or conservatively parsed `setup.py` metadata.
+- Declared version and Python version markers when available from safe manifest fields.
+- Entrypoint file basename when known from safe manifest metadata.
+- Lockfile presence and package-manager hint such as `pip`, `poetry`, `uv`, or `pipenv`.
+- Capped dependency names and versions when parsed from safe local files.
+- Framework hints derived from dependency names or safe manifest fields, with evidence
+  metadata.
+
+Native process nodes may expose:
+
+- Stable node id derived from pid plus safe process identity for the current snapshot.
+- Pid and parent pid.
+- Executable basename or command name.
+- Sanitized command summary. Raw argv and raw `ps args` must never leave the provider
+  boundary in API responses, fixtures, logs, diagnostics, screenshots, docs, or issue
+  comments.
+- Process status and start-time or uptime fields when read from fixed local sources.
+- User or uid if readable without privilege escalation.
+- Cwd only when the resolved path is inside a documented project root; otherwise expose a
+  redacted or omitted location.
+- Runtime hints such as `python`, `node`, `binary`, or `shell` when derived from command
+  basename and not from sensitive args.
+
+Edges may expose:
+
+- Process `runs_on` host.
+- Python application `runs_on` process when the process cwd is under the project root.
+- Process `managed_by` or `related_to` systemd, PM2, tmux, or scheduled-job nodes only when
+  there is explicit evidence such as main pid, cwd, or a known manager output.
+- Python application `contains` package dependency nodes when dependency files are parsed.
+
+Every heuristic edge must include metadata such as `evidence=proc_cwd`,
+`evidence=project_manifest`, or `evidence=systemd_main_pid`. Defer process-to-listener
+edges unless a future safe source provides pid/listener evidence without reading
+`/proc/<pid>/fd` targets.
+
+## Data To Avoid
+
+Providers must not expose:
+
+- Environment variable values or raw `/proc/<pid>/environ`.
+- Raw argv or raw `ps args`, even when they do not appear secret-like.
+- Full cwd paths outside `DOCKERMAP_PROJECT_ROOT`.
+- Secret-bearing package options such as `--index-url`, `--extra-index-url`, trusted-host
+  credentials, auth headers, or private repository URLs.
+- Process memory, maps, open files, fd targets, stdin/stdout/stderr, terminal scrollback, or
+  logs.
+- Dependency cache contents, virtualenv package trees, home-directory config, or arbitrary
+  filesystem contents.
+
+## Fixed Read-Only Sources
+
+Preferred native-process sources on Linux:
+
+- `/proc/<pid>/stat`
+- `/proc/<pid>/status`
+- `/proc/<pid>/cmdline`
+- `/proc/<pid>/cwd` symlink target
+
+If `/proc` is unavailable, a future fallback may use only a fixed command with no user input:
+
+```bash
+ps -eo pid,ppid,user,stat,lstart,comm,args
+```
+
+Preferred Python project files under the configured project root:
+
+- `pyproject.toml`
+- `requirements.txt`
+- `requirements/*.txt`
+- `Pipfile`
+- `poetry.lock`
+- `uv.lock`
+- `setup.cfg`
+- `setup.py`, metadata only, parsed conservatively without executing it
+
+Do not execute Python, import project modules, run package managers, or contact registries.
+
+## Discovery Bounds
+
+Python project discovery must follow the existing npm provider pattern:
+
+- Start under `DOCKERMAP_PROJECT_ROOT` or the daemon working directory.
+- Reuse the general directory traversal cap and stop when it is hit.
+- Enforce these minimum hard caps before implementation:
+  - `MAX_PYTHON_PROJECTS = 64`
+  - `MAX_PYTHON_DEPENDENCIES_PER_PROJECT = 64`
+  - `MAX_PYTHON_MANIFEST_BYTES = 262144`
+  - `MAX_PROVIDER_DIAGNOSTICS = 128`
+- Skip `.git`, `.hg`, `.svn`, `.venv`, `venv`, `env`, `__pycache__`, `site-packages`,
+  `.mypy_cache`, `.pytest_cache`, `.tox`, `.ruff_cache`, `dist`, `build`, `target`,
+  `node_modules`, `coverage`, and cache directories.
+
+Native process discovery should be Linux-first, use `/proc`, cap the number of process
+entries inspected, and never recurse into cwd trees. Unreadable or short-lived processes are
+normal and should produce capped diagnostics instead of failing the runtime-map endpoint.
+Enforce these minimum hard caps before implementation:
+
+- `MAX_NATIVE_PROCESSES = 256`
+- `MAX_PROCESS_CMDLINE_BYTES = 8192`
+- `MAX_PROCESS_STATUS_BYTES = 65536`
+- `MAX_PROCESS_CWD_BYTES = 4096`
+- `MAX_PROVIDER_DIAGNOSTICS = 128`
+
+## Diagnostics
+
+Use diagnostics instead of endpoint failures for expected provider limits:
+
+- Unsupported platform.
+- `/proc` unavailable or partially unreadable.
+- Process disappeared during inspection.
+- Process count capped.
+- Process cmdline, status, or cwd value capped.
+- Python project count capped.
+- Manifest too large.
+- Manifest parse failed.
+- Path skipped because it is outside the project root.
+- Secret-like command, path, manifest, or dependency value redacted or omitted.
+- Optional fixed command unavailable.
+
+Diagnostic text must not include raw secrets, raw argv, private URLs, or full paths outside
+the allowed root.
+
+## Fixture And Test Plan
+
+Implementation follow-ups should add fixture-first coverage before enabling live collection:
+
+- Fake `/proc` trees for normal processes, short-lived processes, unreadable fields, cwd
+  outside the project root, and secret-bearing cmdline values.
+- Optional fake `ps` output if a fallback parser is implemented.
+- Python manifest fixtures for `pyproject.toml`, `requirements.txt`, `Pipfile`, `poetry.lock`,
+  `uv.lock`, and oversized/invalid manifests.
+- Redaction sentinels using `DOCKERMAP_TEST_FAKE_*` in process args, manifest URLs, dependency
+  options, project paths, diagnostics, and edge metadata.
+- Cap fixtures for too many processes, too many projects, too many dependencies, and too-large
+  manifests.
+- Contract fixtures showing a Python application node, a native process node, and a
+  project-cwd edge.
+- A negative fixture proving the process provider does not read `/proc/<pid>/fd` entries or
+  fd targets while building process nodes.
+
+Default tests must not require Docker, systemd, tmux, Python services, real `/proc` contents,
+or external network access.
+
+## Follow-Up Issues
+
+1. Add bounded Python project manifest parser fixtures and redaction tests.
+2. Add Linux `/proc` native-process parser fixtures with caps and soft diagnostics.
+3. Add shared runtime-map fixture examples for Python application and native-process nodes;
+   expand contracts only if fixtures prove the existing vocabulary is insufficient.
+4. Implement the bounded Python project collector under `DOCKERMAP_PROJECT_ROOT`.
+5. Implement the native-process collector behind hard process caps.
+6. Add cross-provider edge derivation for process-to-host, process-to-Python-project, and
+   manager-to-process relationships with evidence metadata.
+7. Run a `security-readonly` review before enabling either provider by default.
+8. Defer process-to-listener edge derivation to a separate security-reviewed issue unless a
+   safe source provides pid/listener evidence without reading `/proc/<pid>/fd` targets.
+9. Add API proxy and contract tests proving `/api/runtime/map` returns the new provider
+   nodes with the same auth, CORS, query-limit, and daemon-error redaction guarantees as
+   existing read-only routes.
+10. Update deployment, security, testing, and release docs after implementation to record the
+   exact sources read, caps used, skipped data, and validation commands.
+11. Teach the web app to consume provider nodes from `/api/runtime/map` in a separate
+   frontend issue.

docs/planning/ROADMAP.md

@@ -32,9 +32,8 @@ Working:
 
 Not finished:
 
-- Provider-specific redaction fixtures for expanded host collectors.
 - Richer metadata for systemd and npm/package providers.
-- Python and native-process providers.
+- Python and native-process provider implementation.
 - List sorting, advanced filters, clickable graph nodes, and richer logs UI.
 - OpenAPI docs, route versioning, and dashboard/widget endpoints.
 - Safe write mode.
@@ -98,8 +97,8 @@ These items improve the product without changing host state.
   evidence where safe.
 - Enrich npm provider output with scripts, framework hints, dependency nodes,
   and bounded package metadata.
-- Plan Python and native-process providers as peers under the same read-only
-  runtime map.
+- Implement Python and native-process providers from the documented read-only plan in
+  [`PYTHON_AND_PROCESS_PROVIDERS.md`](PYTHON_AND_PROCESS_PROVIDERS.md).
 - Add parser-level fixtures for systemd, cron, PM2, tmux, Tailscale, Headscale,
   reverse proxy, DNS, and listener output.
 

docs/release/RELEASE_CHECKLIST.md

@@ -58,8 +58,10 @@ tasks before starting new GUI work:
 - [ ] Capture reverse-proxy smoke evidence on the release host, including bearer-token
   injection, SSE streaming, public review URL access, and direct daemon-port
   inaccessibility.
-- [ ] Plan Python and native-process providers as the next backend provider peers after
+- [x] Plan Python and native-process providers as the next backend provider peers after
   the current Rust runtime model and contracts settle.
+  The planning doc is `docs/planning/PYTHON_AND_PROCESS_PROVIDERS.md`. Implementation,
+  fixture, contract, and frontend follow-ups remain separate future work.
 
 ## Second Round Before Wider Beta
 

docs/security/THREAT_MODEL.md

@@ -73,6 +73,14 @@ Protections:
 - Provider commands stay fixed and read-only, such as list/status/introspection calls.
 - Provider discovery stays bounded to explicit request parameters, known config locations, or
   capped fixture-like scans instead of recursive host crawling.
+- Python/native-process collection must not read `/proc/<pid>/environ`, process memory,
+  open files, `/proc/<pid>/fd` targets, terminal scrollback, logs, or cwd contents beyond
+  bounded manifest checks under the configured project root.
+- Native-process command summaries must be tokenized, redacted, or omitted before they reach
+  nodes, edges, diagnostics, fixtures, screenshots, logs, or issue comments. Raw argv and
+  raw `ps args` must never leave the provider boundary.
+- Python project discovery must not read `.env`, `.pypirc`, `pip.conf`, `pip.ini`, Poetry auth
+  config, private-index credentials, or virtualenv package trees.
 - Package registry/advisory traffic, DNS-provider API calls, Cloudflare API calls, and
   generic external-API lookups are disabled or not implemented in DockerMap runtime today.
 - Tailscale and Headscale discovery use fixed local CLI commands when those tools are