feat(observability): correlate agent traces with OpenShell enforcement and OCSF #5210

Problem Statement

Connect an agent's tool/model call to the OpenShell decision and matching OCSF security records, without guessing from timestamps.

HTTP 403 could mean policy denial, middleware refusal or upstream rejection. HTTP 200 could still end in incomplete delivery. The existing OTLP relay transports traces but does not establish these operation-level connections.

Proposed Design

sequenceDiagram
    box Inside sandbox
        participant H as Agent tool + tracing SDK
    end
    box OpenShell supervisor
        participant P as HTTP/L7 proxy
        participant O as OTLP trace receiver
    end
    participant A as External API
    participant C as Telemetry collector
    participant V as Trace viewer / log store

    H->>P: HTTP request + W3C traceparent
    Note over P: Start OpenShell enforcement span<br/>Assign OpenShell request ID

    alt Allowed
        P->>A: Approved request<br/>Inject credential if configured
        A-->>P: API response
    else Blocked before egress
        Note over P: Generate denial locally<br/>No request reaches the API
    end

    P-->>H: HTTP response + status<br/>X-OpenShell-Request-ID
    H->>H: Optional adapter records a receipt mark

    H-->>O: Agent spans + marks via OTLP
    O-->>C: Forward attributed agent traces
    P-->>C: Export OpenShell spans + OCSF logs
    C-->>V: Traces and security records
    Note over V: Span parentage/links connect execution<br/>Request and trace IDs join security records

• Use W3C traceparent to connect the caller to an OpenShell enforcement span. The prototype parents sampled context, links unsampled context, and ignores malformed/duplicate context and incoming tracestate.
• Mint one OpenShell request ID shared by the span, middleware RequestContext.request_id, OCSF metadata.correlation_uid and response X-OpenShell-Request-ID. Prevent header substitution on inspected paths.
• Treat workload context and receipts as correlation handles, not identity or authorization. Reconcile parent/link provenance and OCSF representation with feat(ocsf): add trace_id/span_id correlation fields to OCSF event builders #2640 and feat(ocsf): record the middleware request ID on OCSF events for middleware-evaluated requests #4557.
• Keep policy, middleware, credential, target-reached and delivery outcomes separate, bounded and free of secrets/payloads. Collector failures must not change enforcement.

Prototype implementation

These are private prototype capabilities, not a claim that all have shipped.

Uses existing OCSF, OTel, network, middleware, gateway policy and supervisor relay components. Rust source-compatibility changes need review before upstreaming.

Validation and limits

• Ran 14 real Hermes/Relay workflows containing 807 spans, with Relay 0.9.3 and local Qwen3.5 9B inside OpenShell. Tested one cumulative prototype build based on 0.1.6-dev.123.
• Used protected synthetic services plus authenticated GitHub access. Verified denials, redaction, response withholding, approval/reload, incomplete streams and collector failures against native span parents, receipts, raw OCSF records and target observations. Parent IDs were not rewritten.
• Scenarios were scripted, but model/tool execution and enforcement were real. Private callback-context/OTLP helpers and automated operator approvals were used. Native MCP discovery and production/cross-driver behavior were not demonstrated. Final cleanup received automated tests, not another live replay; known test limitations remain recorded.

No full OCSF records return to the sandbox. The demo records marks from actual HTTP responses and receipts; security records are joined outside it. A richer feedback API needs separate sandbox-scoped authentication. L4/process/filesystem correlation, policy composition and additional telemetry/artifact formats are separate work.

Acceptance criteria

• Navigate from an agent call to matching OpenShell enforcement and OCSF records.
• Test context provenance, receipt anti-spoofing and secret-free bounded outcomes.
• Preserve terminal evidence through cancellation, reload, collector failure and diagnostic exhaustion.
• Document a real harness example, generic integration points and limitations.

Alternatives Considered

Timestamp-only joins are ambiguous. Caller-owned IDs are not trusted attribution. HTTP success cannot explain enforcement. Sending all security logs into the sandbox is unnecessary and risks exposing operator-only data.

Agent Investigation

Reviewed runtime code, saved native traces, receipts, OCSF records and target counters. This builds on the supervisor-direct relay, not a new gateway/session hop.

Related: #2640, #4557, #2641, #4276, #4544, #4233, #1055.

Checklist

• I've reviewed existing issues and component documentation.
• This is a design proposal, not a "please build this" request.