diff --git a/docs/README.md b/docs/README.md index 793c9c7..157da02 100644 --- a/docs/README.md +++ b/docs/README.md @@ -40,9 +40,9 @@ Working now: Still in progress: -- Provider-specific redaction fixtures. +- Provider-specific redaction fixtures for future collectors. - Richer systemd and package metadata. -- Python and native-process providers. +- Python and native-process provider implementations. - Better table sorting, filtering, graph navigation, and logs UI. - OpenAPI docs and stable versioned API routes. - Safe write mode with backups and rollback. This is not enabled today. @@ -84,6 +84,8 @@ daemon, provider, auth, deployment, or write-mode behavior. data flow, runtime-map model, and provider rules. - [architecture/PAGE_LOGIC.md](architecture/PAGE_LOGIC.md): intended UI routes, page behavior, and cross-page navigation. +- [planning/PYTHON_AND_PROCESS_PROVIDERS.md](planning/PYTHON_AND_PROCESS_PROVIDERS.md): + read-only provider plan for future Python application and native-process collection. - [testing/TESTING_PLAN.md](testing/TESTING_PLAN.md): local checks, security tests, browser smoke tests, live-Docker tests, and manual release smoke. - [testing/SANDBOX_FIXTURE.md](testing/SANDBOX_FIXTURE.md): isolated labeled diff --git a/docs/architecture/ARCHITECTURE.md b/docs/architecture/ARCHITECTURE.md index 01e9961..04d4a25 100644 --- a/docs/architecture/ARCHITECTURE.md +++ b/docs/architecture/ARCHITECTURE.md @@ -73,8 +73,6 @@ The map is read-only and currently contains: package-update/advisory metadata, but no runtime registry or advisory lookup is enabled today. UI update labels are derived from local model/demo data unless a future opt-in advisory provider is implemented. -- Python projects discovered from common project files as a later application-provider - peer to npm. - scheduled jobs from `/etc/crontab`, `/etc/cron.d/*`, and the current user's `crontab -l` when readable. - PM2 apps from `pm2 jlist` when PM2 is installed. - tmux sessions from `tmux list-sessions` when tmux is installed and reachable. @@ -89,6 +87,11 @@ Provider commands are fixed read-only invocations, not user-supplied shell comma Filesystem discovery must stay bounded by configured roots, skip dependency/build directories, and avoid reading secrets such as `.env` values. +Planned Python application and native-process providers should plug into this same map as +later read-only peers. The implementation plan in +[`docs/planning/PYTHON_AND_PROCESS_PROVIDERS.md`](../planning/PYTHON_AND_PROCESS_PROVIDERS.md) +defines safe sources, discovery bounds, omitted data, diagnostics, and follow-up slices. + Kubernetes and other orchestrators should plug into this same model as additional providers, not replace the local Docker/host model. Kubernetes support should be opt-in because it needs kubeconfig or in-cluster credentials, namespace scoping, and RBAC permissions. A safe first Kubernetes provider should read namespaces, pods, services, deployments, ingress objects, persistent volume claims, and selected labels/owner references, then map them to `orchestrator_workload` nodes and edges. ## Runtime Layers @@ -127,7 +130,10 @@ Forge (npm) -> forge.service -> tmux session -> GPT worker Relationship discovery should prefer explicit evidence first, such as systemd dependency fields, Compose labels, process working directories, package manifests, lockfiles, known reverse-proxy routes, and Docker network membership. Heuristics are allowed only when -metadata records why the edge exists. +metadata records why the edge exists. Future Python/native-process edges must include +evidence metadata such as `proc_cwd`, `project_manifest`, or `systemd_main_pid`. +Process-to-listener edges stay out of the first Python/native-process implementation unless +a later security review approves a safe source that does not read `/proc//fd` targets. ## Docker Access diff --git a/docs/planning/IMPLEMENTATION_PLAN.md b/docs/planning/IMPLEMENTATION_PLAN.md index a411e5e..978e026 100644 --- a/docs/planning/IMPLEMENTATION_PLAN.md +++ b/docs/planning/IMPLEMENTATION_PLAN.md @@ -51,14 +51,14 @@ still the runtime model, collectors, contracts, and security evidence. - Frontend component split, Compose scan/graph/edit-plan endpoints, CLI scan/validate/export, Compose/runtime correlation, override merge semantics, and contract compatibility tests are done. -- Still to build before full alpha: provider-specific redaction fixtures for new - collectors, richer package/update/advisory behavior, Python/native-process provider - peers, table sorting, advanced filters, clickable graph nodes, log level filter, live - tail, broader browser smoke coverage, and log pagination UI. -- Immediately after the next implementation commit lands, execute the follow-up queue in - [`docs/release/RELEASE_CHECKLIST.md`](../release/RELEASE_CHECKLIST.md): provider redaction fixtures, - package advisory/network opt-in docs, live-Docker/reverse-proxy release-host evidence, - and Python/native-process provider planning. +- Still to build before full alpha: provider fixtures for future collectors, richer + package/update/advisory behavior, Python/native-process provider implementations, table + sorting, advanced filters, clickable graph nodes, log level filter, live tail, broader + browser smoke coverage, and log pagination UI. +- The Python/native-process provider plan lives in + [`docs/planning/PYTHON_AND_PROCESS_PROVIDERS.md`](PYTHON_AND_PROCESS_PROVIDERS.md). + Remaining release-host evidence is tracked in + [`docs/release/RELEASE_CHECKLIST.md`](../release/RELEASE_CHECKLIST.md). - Testing plan: see [`docs/testing/TESTING_PLAN.md`](../testing/TESTING_PLAN.md). --- diff --git a/docs/planning/PYTHON_AND_PROCESS_PROVIDERS.md b/docs/planning/PYTHON_AND_PROCESS_PROVIDERS.md new file mode 100644 index 0000000..02a8dc4 --- /dev/null +++ b/docs/planning/PYTHON_AND_PROCESS_PROVIDERS.md @@ -0,0 +1,220 @@ +# Python And Native Process Provider Plan + +This plan defines how DockerMap should add Python application and native-process +providers before any collector code is written. The goal is to make these providers peers +in the read-only runtime map without scanning the whole host, exposing secrets, or adding +controls that can change running services. + +## Goals + +- Represent Python applications and ordinary host processes in `GET /daemon/runtime/map`. +- Reuse the existing runtime-map concepts for services, packages, processes, edges, and + diagnostics. +- Prefer explicit evidence over guesses, and explain heuristic edges with metadata. +- Keep every source fixed, local, bounded, and read-only. +- Keep tests runnable from fixtures without a real production host. + +## Non-Goals + +- No process start, stop, restart, signal, kill, reload, or package-install actions. +- No user-supplied shell commands. +- No process environment dumps. +- No log, stdout, stderr, terminal scrollback, process memory, open-file, or file-descriptor + scraping. +- No `.env`, `.pypirc`, `pip.conf`, `pip.ini`, Poetry auth config, private-index config, or + credential-file reads. +- No package registry, advisory, DNS-provider, or other external-network lookup by default. +- No recursive home-directory, virtualenv, cache, or host-wide filesystem crawl. + +## Contract Fit + +The current contracts already contain most of the vocabulary needed for a first pass: + +- Python application nodes can use `provider: "process"` and `type: + "python_application"` for the first collector, because the current contracts do not + include a separate `python` provider. +- Native process nodes can use `provider: "process"` and `type: "process"` or `type: + "worker"` when evidence is strong enough. +- Package metadata can use existing `package` and `package_dependency` node types with + `manager: "pip"` once Rust and TypeScript contract shapes are intentionally aligned. +- Provider-neutral service fields should stay the long-term target, but the first Rust + implementation can keep metadata string-only until richer runtime-node fields are added + to Rust and fixture drift checks cover both languages. + +Do not add a new `python` provider enum in the first collector unless fixtures prove that +Python project discovery cannot be represented clearly with `provider: "process"`, +`type: "python_application"`, `package`, and `package_dependency` vocabulary. If a later +provider enum is needed, make it a separate contract issue before collector work lands. + +## Data To Expose + +Python application nodes may expose: + +- Stable node id derived from safe project-relative evidence, not raw secret-bearing paths. +- Project or package name when read from safe manifest fields. +- Project-relative path under `DOCKERMAP_PROJECT_ROOT`. +- Manifest type: `pyproject.toml`, `requirements.txt`, `Pipfile`, `poetry.lock`, + `uv.lock`, `setup.cfg`, or conservatively parsed `setup.py` metadata. +- Declared version and Python version markers when available from safe manifest fields. +- Entrypoint file basename when known from safe manifest metadata. +- Lockfile presence and package-manager hint such as `pip`, `poetry`, `uv`, or `pipenv`. +- Capped dependency names and versions when parsed from safe local files. +- Framework hints derived from dependency names or safe manifest fields, with evidence + metadata. + +Native process nodes may expose: + +- Stable node id derived from pid plus safe process identity for the current snapshot. +- Pid and parent pid. +- Executable basename or command name. +- Sanitized command summary. Raw argv and raw `ps args` must never leave the provider + boundary in API responses, fixtures, logs, diagnostics, screenshots, docs, or issue + comments. +- Process status and start-time or uptime fields when read from fixed local sources. +- User or uid if readable without privilege escalation. +- Cwd only when the resolved path is inside a documented project root; otherwise expose a + redacted or omitted location. +- Runtime hints such as `python`, `node`, `binary`, or `shell` when derived from command + basename and not from sensitive args. + +Edges may expose: + +- Process `runs_on` host. +- Python application `runs_on` process when the process cwd is under the project root. +- Process `managed_by` or `related_to` systemd, PM2, tmux, or scheduled-job nodes only when + there is explicit evidence such as main pid, cwd, or a known manager output. +- Python application `contains` package dependency nodes when dependency files are parsed. + +Every heuristic edge must include metadata such as `evidence=proc_cwd`, +`evidence=project_manifest`, or `evidence=systemd_main_pid`. Defer process-to-listener +edges unless a future safe source provides pid/listener evidence without reading +`/proc//fd` targets. + +## Data To Avoid + +Providers must not expose: + +- Environment variable values or raw `/proc//environ`. +- Raw argv or raw `ps args`, even when they do not appear secret-like. +- Full cwd paths outside `DOCKERMAP_PROJECT_ROOT`. +- Secret-bearing package options such as `--index-url`, `--extra-index-url`, trusted-host + credentials, auth headers, or private repository URLs. +- Process memory, maps, open files, fd targets, stdin/stdout/stderr, terminal scrollback, or + logs. +- Dependency cache contents, virtualenv package trees, home-directory config, or arbitrary + filesystem contents. + +## Fixed Read-Only Sources + +Preferred native-process sources on Linux: + +- `/proc//stat` +- `/proc//status` +- `/proc//cmdline` +- `/proc//cwd` symlink target + +If `/proc` is unavailable, a future fallback may use only a fixed command with no user input: + +```bash +ps -eo pid,ppid,user,stat,lstart,comm,args +``` + +Preferred Python project files under the configured project root: + +- `pyproject.toml` +- `requirements.txt` +- `requirements/*.txt` +- `Pipfile` +- `poetry.lock` +- `uv.lock` +- `setup.cfg` +- `setup.py`, metadata only, parsed conservatively without executing it + +Do not execute Python, import project modules, run package managers, or contact registries. + +## Discovery Bounds + +Python project discovery must follow the existing npm provider pattern: + +- Start under `DOCKERMAP_PROJECT_ROOT` or the daemon working directory. +- Reuse the general directory traversal cap and stop when it is hit. +- Enforce these minimum hard caps before implementation: + - `MAX_PYTHON_PROJECTS = 64` + - `MAX_PYTHON_DEPENDENCIES_PER_PROJECT = 64` + - `MAX_PYTHON_MANIFEST_BYTES = 262144` + - `MAX_PROVIDER_DIAGNOSTICS = 128` +- Skip `.git`, `.hg`, `.svn`, `.venv`, `venv`, `env`, `__pycache__`, `site-packages`, + `.mypy_cache`, `.pytest_cache`, `.tox`, `.ruff_cache`, `dist`, `build`, `target`, + `node_modules`, `coverage`, and cache directories. + +Native process discovery should be Linux-first, use `/proc`, cap the number of process +entries inspected, and never recurse into cwd trees. Unreadable or short-lived processes are +normal and should produce capped diagnostics instead of failing the runtime-map endpoint. +Enforce these minimum hard caps before implementation: + +- `MAX_NATIVE_PROCESSES = 256` +- `MAX_PROCESS_CMDLINE_BYTES = 8192` +- `MAX_PROCESS_STATUS_BYTES = 65536` +- `MAX_PROCESS_CWD_BYTES = 4096` +- `MAX_PROVIDER_DIAGNOSTICS = 128` + +## Diagnostics + +Use diagnostics instead of endpoint failures for expected provider limits: + +- Unsupported platform. +- `/proc` unavailable or partially unreadable. +- Process disappeared during inspection. +- Process count capped. +- Process cmdline, status, or cwd value capped. +- Python project count capped. +- Manifest too large. +- Manifest parse failed. +- Path skipped because it is outside the project root. +- Secret-like command, path, manifest, or dependency value redacted or omitted. +- Optional fixed command unavailable. + +Diagnostic text must not include raw secrets, raw argv, private URLs, or full paths outside +the allowed root. + +## Fixture And Test Plan + +Implementation follow-ups should add fixture-first coverage before enabling live collection: + +- Fake `/proc` trees for normal processes, short-lived processes, unreadable fields, cwd + outside the project root, and secret-bearing cmdline values. +- Optional fake `ps` output if a fallback parser is implemented. +- Python manifest fixtures for `pyproject.toml`, `requirements.txt`, `Pipfile`, `poetry.lock`, + `uv.lock`, and oversized/invalid manifests. +- Redaction sentinels using `DOCKERMAP_TEST_FAKE_*` in process args, manifest URLs, dependency + options, project paths, diagnostics, and edge metadata. +- Cap fixtures for too many processes, too many projects, too many dependencies, and too-large + manifests. +- Contract fixtures showing a Python application node, a native process node, and a + project-cwd edge. +- A negative fixture proving the process provider does not read `/proc//fd` entries or + fd targets while building process nodes. + +Default tests must not require Docker, systemd, tmux, Python services, real `/proc` contents, +or external network access. + +## Follow-Up Issues + +1. Add bounded Python project manifest parser fixtures and redaction tests. +2. Add Linux `/proc` native-process parser fixtures with caps and soft diagnostics. +3. Add shared runtime-map fixture examples for Python application and native-process nodes; + expand contracts only if fixtures prove the existing vocabulary is insufficient. +4. Implement the bounded Python project collector under `DOCKERMAP_PROJECT_ROOT`. +5. Implement the native-process collector behind hard process caps. +6. Add cross-provider edge derivation for process-to-host, process-to-Python-project, and + manager-to-process relationships with evidence metadata. +7. Run a `security-readonly` review before enabling either provider by default. +8. Defer process-to-listener edge derivation to a separate security-reviewed issue unless a + safe source provides pid/listener evidence without reading `/proc//fd` targets. +9. Add API proxy and contract tests proving `/api/runtime/map` returns the new provider + nodes with the same auth, CORS, query-limit, and daemon-error redaction guarantees as + existing read-only routes. +10. Update deployment, security, testing, and release docs after implementation to record the + exact sources read, caps used, skipped data, and validation commands. +11. Teach the web app to consume provider nodes from `/api/runtime/map` in a separate + frontend issue. diff --git a/docs/planning/ROADMAP.md b/docs/planning/ROADMAP.md index ddc0d12..2d6883a 100644 --- a/docs/planning/ROADMAP.md +++ b/docs/planning/ROADMAP.md @@ -32,9 +32,8 @@ Working: Not finished: -- Provider-specific redaction fixtures for expanded host collectors. - Richer metadata for systemd and npm/package providers. -- Python and native-process providers. +- Python and native-process provider implementation. - List sorting, advanced filters, clickable graph nodes, and richer logs UI. - OpenAPI docs, route versioning, and dashboard/widget endpoints. - Safe write mode. @@ -98,8 +97,8 @@ These items improve the product without changing host state. evidence where safe. - Enrich npm provider output with scripts, framework hints, dependency nodes, and bounded package metadata. -- Plan Python and native-process providers as peers under the same read-only - runtime map. +- Implement Python and native-process providers from the documented read-only plan in + [`PYTHON_AND_PROCESS_PROVIDERS.md`](PYTHON_AND_PROCESS_PROVIDERS.md). - Add parser-level fixtures for systemd, cron, PM2, tmux, Tailscale, Headscale, reverse proxy, DNS, and listener output. diff --git a/docs/release/RELEASE_CHECKLIST.md b/docs/release/RELEASE_CHECKLIST.md index 885a3f1..cad59a9 100644 --- a/docs/release/RELEASE_CHECKLIST.md +++ b/docs/release/RELEASE_CHECKLIST.md @@ -58,8 +58,10 @@ tasks before starting new GUI work: - [ ] Capture reverse-proxy smoke evidence on the release host, including bearer-token injection, SSE streaming, public review URL access, and direct daemon-port inaccessibility. -- [ ] Plan Python and native-process providers as the next backend provider peers after +- [x] Plan Python and native-process providers as the next backend provider peers after the current Rust runtime model and contracts settle. + The planning doc is `docs/planning/PYTHON_AND_PROCESS_PROVIDERS.md`. Implementation, + fixture, contract, and frontend follow-ups remain separate future work. ## Second Round Before Wider Beta diff --git a/docs/security/THREAT_MODEL.md b/docs/security/THREAT_MODEL.md index eb00db2..7a0b2a8 100644 --- a/docs/security/THREAT_MODEL.md +++ b/docs/security/THREAT_MODEL.md @@ -73,6 +73,14 @@ Protections: - Provider commands stay fixed and read-only, such as list/status/introspection calls. - Provider discovery stays bounded to explicit request parameters, known config locations, or capped fixture-like scans instead of recursive host crawling. +- Python/native-process collection must not read `/proc//environ`, process memory, + open files, `/proc//fd` targets, terminal scrollback, logs, or cwd contents beyond + bounded manifest checks under the configured project root. +- Native-process command summaries must be tokenized, redacted, or omitted before they reach + nodes, edges, diagnostics, fixtures, screenshots, logs, or issue comments. Raw argv and + raw `ps args` must never leave the provider boundary. +- Python project discovery must not read `.env`, `.pypirc`, `pip.conf`, `pip.ini`, Poetry auth + config, private-index credentials, or virtualenv package trees. - Package registry/advisory traffic, DNS-provider API calls, Cloudflare API calls, and generic external-API lookups are disabled or not implemented in DockerMap runtime today. - Tailscale and Headscale discovery use fixed local CLI commands when those tools are diff --git a/docs/testing/TESTING_PLAN.md b/docs/testing/TESTING_PLAN.md index ff0bd77..90b88be 100644 --- a/docs/testing/TESTING_PLAN.md +++ b/docs/testing/TESTING_PLAN.md @@ -78,11 +78,12 @@ The API tests cover: These tests run against the real Express entry point with mock fallback or a stub daemon. They do not require Docker, systemd, tmux, reverse-proxy software, DNS services, or a GUI. -After the next implementation commit lands, execute the post-commit follow-up queue in -[`docs/release/RELEASE_CHECKLIST.md`](../release/RELEASE_CHECKLIST.md). That queue is the source of truth for -provider redaction fixtures, package advisory/network opt-in documentation, live-Docker -release-host evidence, reverse-proxy smoke evidence, and Python/native-process provider -planning. +The Python/native-process provider plan now lives in +[`docs/planning/PYTHON_AND_PROCESS_PROVIDERS.md`](../planning/PYTHON_AND_PROCESS_PROVIDERS.md). +Keep the post-commit follow-up queue in +[`docs/release/RELEASE_CHECKLIST.md`](../release/RELEASE_CHECKLIST.md) current for remaining +release-host evidence and for implementation, fixture, API, and docs evidence when new +providers land. ## Alpha Security Gates For Expanded Providers @@ -119,6 +120,15 @@ markers, provider diagnostics, and provider edge metadata. These fixtures delibe raw values. The native-process check is an output-boundary regression test until a real bounded native-process collector is implemented. +Before Python or native-process collectors are enabled, add fixture-first tests for fake +`/proc` trees, optional fixed `ps` output, Python manifests, skipped directories, cap +diagnostics, and redaction sentinels. These tests should prove that env values, raw secret +arguments, private package-index URLs, cwd paths outside the configured project root, and +manifest credentials are omitted or redacted. They should also cover process disappearance, +unreadable proc fields, too many processes, too many Python projects, too many dependencies, +and oversized manifests without requiring Docker, systemd, tmux, real Python services, or +external network access. + Run the provider redaction fixture checks with: ```bash