security audit and OWASP ZAP CI

[lost-rob0t]

What changed

• remove the implicit first-run star:intel administrator credential
• leave the human-user store empty unless an initial password is explicitly configured
• preserve the one-time bootstrap-secret API-key path for fresh deployments
• enforce credential grant attenuation so delegated issuers cannot mint admin or broader-scope credentials
• bind non-admin delegated credentials to the caller's own principal ID and principal type, preventing cross-principal impersonation
• prevent delegated rotate/revoke/disable operations from crossing identity or targeting credentials more authoritative than the caller
• make administrator authority depend on explicit admin scope, not the descriptive principal-type string
• explicitly map human-user management and self-password-change routes through the centralized authorization boundary
• add global HTTP response hardening headers
• add an OWASP ZAP Automation Framework scan against a disposable Compose stack
• add Common Lisp reader/eval and process-execution RCE canaries that fail if a /tmp marker is created inside the disposable server
• add CouchDB/Clouseau injection probes for boolean/field scope escape and encoded design/view path traversal
• add malformed JSON, invalid UTF-8, duplicate-key observations, oversized-body, deep-nesting, bulk-amplification, and bounded concurrent parser tests
• hard-refuse the negative-test script unless it is running inside a starintel-zap-* disposable Compose project with STAR_SECURITY_DISPOSABLE=1
• upload ZAP reports, negative-test JSONL results, CouchDB/server logs, and stack diagnostics as workflow artifacts
• document the security audit, hostile-input suite, residual findings, and next hardening priorities

Security rationale

The audit found four privilege-boundary failures:

1. A fresh auth database created an administrator account with a known built-in password. The public login route could exchange that password for a normal administrator API key before the required-password-change flag had any enforcement effect.
2. A principal with only credentials:create could request arbitrary scopes and principal type, including an administrator credential with admin. Lifecycle capabilities could also rotate a more privileged credential and receive its replacement secret.
3. API-key owner becomes the authenticated principal ID, but delegated creation accepted any owner. A credential issuer could therefore mint a same-scope key as another principal and bypass identity/ownership checks. Lifecycle operations had the same cross-identity problem.
4. administrator-principal-p treated the descriptive principal type administrator as authority even without admin scope, while the central policy engine used scope-based authority. The bulk-job status path used that helper for its ownership bypass.

This branch removes those privilege paths and now actively tests hostile input against the deployed CI image. Lisp injection probes are non-destructive canaries: successful execution only writes a marker under /tmp, which is still treated as proof of RCE. CouchDB tests use two datasets with the same search fixture and fail if a restricted principal can retrieve the denied document through Lucene/Clouseau syntax manipulation. DoS probes are deliberately bounded and run only against the disposable stack.

The remaining audit findings are deliberately documented rather than hidden: direct HTTP bearer transport has no in-process TLS invariant, must_change_password is still advisory, development auth bypass needs a loopback invariant, the application body limit is downstream of HTTP request materialization, and container/CI least privilege can be tightened further.

Validation

GitHub CI is the validation environment for this branch. Existing Nix/unit/integration/container jobs plus the ZAP and hostile-input workflow must pass before this leaves draft.

[lost-rob0t]

RAGE/TDD failure evidence for the current repair transaction:

• immutable failing head: 055c8bfdb5f291c764decd676c2163911e851021
• failing command: bash ./scripts/zap-ci.sh passive
• deterministic behavior: OWASP ZAP Security failed while establishing the scoped CouchDB/Clouseau fixtures; POST /new/document/note returned HTTP 422 before the hostile search probes could run
• uploaded server diagnostics identified the exact server response as unsupported_schema_version: Document schema version is not supported
• cause: scripts/security-negative-test.sh hard-coded document version:"0.9.0", so the security harness had drifted from the server's canonical document spec
• proof: the failure occurs in fixture creation, before the assertions intended to detect Clouseau/CouchDB scope escape; therefore weakening those assertions would be a false green

Repair commit c7d01c49538e5ab980cf3a9c3af0266e984d888d now discovers doc_spec_version from the server metadata endpoint and uses that value for all canonical document fixtures, while retaining the hostile-input assertions. It also emits the response body on fixture creation failure instead of hiding the reason behind curl --fail.

The branch then conflicted with #104/master, so current master 8f6e50701081f0335c7552082dba5ff392b859e5 was merged explicitly. Current exact head is 1c6f1efacc531d67a7d86a9dd29ee5d414bad38f; it is 0 commits behind master and preserves #104's client/admin systems/tests plus #105's security hardening. Fresh exact-head CI is in progress; do not merge or remove draft status until every required gate, especially ZAP, is green.

[lost-rob0t]

Second-stage TDD evidence: the stale-schema harness defect is fixed, and the hostile suite now exposes a real server boundary defect.

At exact branch head 6cba035145d7a4b5b8c35d77a64596e4f3494f3b (PR merge ref 83f7875de3be3314e9be79b4459a01593e1b7db4), OWASP ZAP Security run 32693397947, job 97331044353, reached the malformed-JSON phase after stack health, delegated credential checks, scoped CouchDB/Clouseau fixtures, search injection, and view-path probes all succeeded.

The deterministic regression is now:

malformed JSON probe trailing-comma expected 400, got HTTP 200

The request used the valid canonical document schema version 0.8.0; the server accepted and normalized it as {"_id":"security-json-comma","dataset":"security-a","dtype":"note","version":"0.8.0"}. This proves malformed JSON is reaching successful document handling rather than being rejected at the HTTP syntax boundary.

Per the StarIntel human-gated ADADR policy, no parser implementation is being added to this PR yet. The research transaction is now lost-rob0t/starintel-auto-research#142 / STAR-RESEARCH-041 Strict JSON HTTP Boundary, with research approval PENDING, architecture NOT STARTED, and implementation BLOCKED until explicit operator research approval, subsequent design/adversarial review, and separate design approval.

Keep this PR draft/red. Do not weaken the expected 400 regression to make ZAP green.

[lost-rob0t]

RAGE freshness reconciliation (2026-08-25): the trailing-comma RED evidence on this PR is no longer sufficient to establish a defect on current master.

The immutable failure remains valid for head 6cba035145d7a4b5b8c35d77a64596e4f3494f3b / ZAP run 32693397947, but current master is now 52948721d8981e9aa5ccad4efd77185e52af845b after #119. This branch has diverged substantially from that master (merge base 8f6e50701081f0335c7552082dba5ff392b859e5; current-master side is 74 commits beyond the #105 head in the compare).

More importantly, current master changed the HTTP parser boundary: source/frontends/http-boundary-core.lisp now decodes UTF-8, validates the raw JSON text with yason:parse, and only then converts it to the existing JSOWN representation. That is directly relevant to the old failure mode where permissive JSOWN parsing accepted a trailing comma. Current master also has a malformed-json-is-a-400-client-error unit regression, although it currently uses a truncated { fixture rather than the exact trailing-comma payload.

Therefore do not implement a parser fix from the old RED badge and do not merge/rebase this security branch wholesale. The next TDD action is to re-prove the exact trailing-comma regression against current master 52948721... (preferably the smallest parse-json-octets fixture plus the deployed HTTP/ZAP probe). If current master returns malformed_json/HTTP 400, record #119 as observed superseding implementation evidence and reconcile STAR-RESEARCH-041 without fabricating approval. If it still returns 200, preserve the RED evidence and continue through the existing human-gated ADADR transaction.