Conversation
Add a new `jaiph serve` command that exposes workflows over HTTP instead of only stdio JSON-RPC, so jaiph can run as a deployed, network-callable service. Built hand-rolled on node:http with no new runtime dependencies. The server mirrors `jaiph mcp` startup (graph load, diagnostics, --env, Docker image prep, credential preflight) and serves run-resource endpoints under /v1, a browser-usable /docs Swagger UI (CDN assets pinned with SRI), and a per-request OpenAPI 3.1 document generated by a pure buildOpenApi from the same deriveTools exposure rules as MCP. Workflow execution reuses the MCP call layer, now moved to src/cli/exec/call.ts (McpCallResult renamed to WorkflowCallResult, caller-supplied runId), and the generation hot-reload machinery is extracted to src/cli/shared/generation.ts shared by both commands. Adds bearer auth (JAIPH_SERVE_TOKEN, required for /v1 and for non-loopback binds), a concurrency cap (JAIPH_SERVE_MAX_CONCURRENT), body/content-type limits, graceful drain-then-cancel shutdown, docs, and unit/integration/e2e coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add run-inspection endpoints to `jaiph serve` so clients can watch a run
in flight and retrieve what it produced, closing the API half of the
"inspect what it is doing" feedback.
GET /v1/runs/{id}/events serves the run's hash-chained, credential-redacted
run_summary.jsonl verbatim: NDJSON by default, or SSE (Accept:
text/event-stream) that replays existing journal lines, follows the file
via ~250ms polling for appended lines, sends :ka keep-alives every 15s,
and emits `event: end` once the registry marks the run terminal (works for
already-terminal runs too). GET /v1/runs/{id}/artifacts lists files under
the run dir's artifacts/, and GET /v1/runs/{id}/artifacts/{path} downloads
one as application/octet-stream. Downloads are traversal-proof: the request
is resolved and realpath-contained against the artifacts dir, rejecting
`..`, absolute paths, and symlinks escaping it with 404. Raw %06d-*.out/.err
capture files are never exposed by any route. Docker-mode runs read the
host-side run dir via the existing discoverDockerRunDir/remapContainerPath
layer. Docs and integration/e2e coverage (SSE mid-run, redaction, artifact
round-trip, traversal battery) included.
A completed run now exports one OpenTelemetry span tree (workflow → steps → prompts) to any OTLP/HTTP collector — a local otel-collector, Grafana Tempo, Honeycomb, Datadog — enabled by the standard OTEL_EXPORTER_OTLP_* environment variables. Export happens host-side after the run reaches terminal state by reading the run's already credential-redacted run_summary.jsonl, so nothing new crosses the sandbox boundary and no OTEL_* env is forwarded into the container. The new src/cli/telemetry/otlp.ts adds zero runtime dependencies (a node:https/node:http request is the whole transport). The pure runSummaryToOtlp() maps journal lines to a JSON ExportTraceServiceRequest with deterministic trace/span ids, and a single shared exportRunTelemetry() hook fires at every host terminal state (jaiph run completion and the shared workflow-call layer covering MCP and jaiph serve). Telemetry is never load-bearing: an unreachable or erroring collector produces exactly one stderr warning and leaves the run's exit code, output, and journal untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A run that terminates unsuccessfully (nonzero exit or a signal) is now pushed to a Sentry error tracker as a single error event, enabled iff SENTRY_DSN is set. Operators running workflows on a schedule or as a service get alerting and grouping without scraping run directories. The new dependency-free src/cli/telemetry/sentry.ts hand-rolls a Sentry envelope POST over node:https, sharing the timeout-guarded POST helper extracted into src/cli/telemetry/http.ts (otlp.ts now reuses it too). Reporting fires from the single shared post-run hook that all modes funnel through, so it is one choke point across jaiph run and the MCP/HTTP call layer. Event content — message, tags, extra, fingerprint, release, environment — is sourced only from the already credential- redacted run_summary.jsonl; re-occurrences group per workflow + failing step. Like trace export it is host-side, end-of-run, and never load- bearing: unreachable Sentry, a non-2xx response, or a timeout produces exactly one stderr warning and leaves the run's exit code and output untouched. Successful runs send nothing. SENTRY_DSN/SENTRY_ENVIRONMENT/SENTRY_RELEASE are documented in the env-vars telemetry section and docs/observability.md. No JAIPH_* vars added; dependencies remain empty. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The ghcr.io/jaiphlang/jaiph-runtime image already bundles jaiph and the claude/cursor/codex backends, but it was only usable as a sandbox rootfs orchestrated by a host jaiph process. Running it directly failed because jaiph defaults to Docker sandboxing on Linux and there is no daemon inside the container. Bake ENV JAIPH_UNSAFE=true into runtime/Dockerfile so the container acts as its own sandbox in host mode; the inner `jaiph run --raw` invocation never launches Docker, so host-orchestrated sandbox behavior is unchanged (pinned by test). When jaiph would launch Docker but the CLI is missing and a container indicator (/.dockerenv or /run/.containerenv) is present, the error now tells the user to set JAIPH_UNSAFE=true. Add docs/deploy.md (one-shot docker run, CI note, Kubernetes) with a real docs/deploy/k8s.yaml manifest and an explicit no-jaiph-sandbox security posture, linked from README, sandboxing, and setup docs. Add a gated standalone-image e2e smoke test and a kubectl dry-run manifest gate in CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Record the deployment review as executable tasks and use Fable for their implementation. Co-authored-by: Cursor <cursoragent@cursor.com>
composeResult in src/cli/exec/call.ts built a failed call's text from
the run's live captured output, so jaiph serve (result_text) and jaiph
mcp (tool errors) could return a secret verbatim even though the event
journal, OTLP, and Sentry all redacted it. The credential rule is now a
single shared helper, redactCredentials, extracted from
RuntimeEventEmitter into src/runtime/kernel/redact.ts and imported by
both the journal writer and composeResult. composeResult redacts the
fully assembled failure text once, so failed-step detail, raw
stderr/stdout, and collected log messages all pass the same boundary,
and the Docker --env passthrough is merged back in for redaction.
Successful return values stay verbatim as intentional API output.
Covered by src/cli/exec/call.test.ts and integration tests that a
failing workflow echoing a credential never leaks it through ?wait=true,
GET /v1/runs/{id}, or an MCP tool result.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
jaiph mcp handled tool calls concurrently, but a hot reload deleted the previous generation's out dir (emitted scripts + serialized graph) the instant it swapped, so a call started just before the reload lost the scripts its remaining steps spawn from. Signal shutdown also removed the shared temp root without draining or cancelling active calls. Extract the LiveGeneration refcount model that jaiph serve already used into createGenerationTracker in src/cli/shared/generation.ts and share it between both commands. Each call acquires a lease on the generation live at call start; a superseded generation's out dir is deleted only when its last lease releases, so a call spanning one or more reloads always runs from the generation it captured and no directory is left behind. Shutdown is now drain-then-cancel: stdin close or the first SIGINT/SIGTERM stops accepting input and awaits in-flight calls before cleanup, exiting 0. A second signal invokes McpServer.cancelAll(), terminating each run's child process tree (SIGINT then SIGKILL) and force-removing its Docker container, so no child process or container is orphaned; killed runs settle with normal isError responses so the drain still completes. Adds generation.test.ts, server.test.ts cancelAll coverage, and e2e test 149_mcp_generation_lifecycle.sh; updates CLI and MCP docs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wait visibly between usage checks before selecting queue work, and keep Fable as the implementation model. Co-authored-by: Cursor <cursoragent@cursor.com>
The serve concurrency cap limited active children but not process
memory, so a long-lived server could be exhausted three ways: the
in-memory run map grew forever, each completed run's result_text stayed
resident, and a single run's raw stdout/stderr/log capture accumulated
unbounded.
Add per-run UTF-8 byte caps (OutputCaps) threaded through the call
layer: attachOutputCollector stops accumulating stdout, stderr, and log
output at the cap and composeResult caps result_text as a backstop, with
overflow dropped and replaced by a deterministic truncation marker
(capBytes slices on a byte boundary to keep valid UTF-8). Defaults stay
effectively unbounded so jaiph mcp and direct callers are unchanged;
only jaiph serve passes finite caps via JAIPH_SERVE_MAX_OUTPUT_BYTES
(default 1 MiB).
Add completed-run retention: evictCompleted runs after every finalize,
dropping terminal records past JAIPH_SERVE_RETAIN_RUNS (default 500,
oldest first) and older than JAIPH_SERVE_RETAIN_AGE_SEC (default 86400,
0 disables). Active runs are never evicted, and eviction removes only
the in-memory record — durable journals and artifacts on disk remain the
operator's to prune.
Paginate GET /v1/runs via ?limit (default 100, clamped to 1000) and
?offset, returning {runs, total, limit, offset} so the response is never
unbounded and newest-first paging stays stable. Update the OpenAPI
document, docs, and add unit and e2e coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pause engineer queue processing once current-session usage exceeds 90 percent. Co-authored-by: Cursor <cursoragent@cursor.com>
Make the Kubernetes example operationally safe and keep HTTP request, event, and artifact handling bounded by bytes transferred. Co-authored-by: Cursor <cursoragent@cursor.com>
Route run, serve, and MCP through one shared execution-policy parser so sandbox posture, workspace, repeatable --env, --inplace, --unsafe, and --yes behave identically in every mode. Reject mutually exclusive postures and command-specific unsupported flags as usage errors before spawning, keeping only display (--raw) and transport (--host/--port) options command-specific. Define a single precedence order across CLI flags, JAIPH_* env vars, and workflow runtime metadata, resolve and print the effective posture once at server startup, and apply it to every call while preserving the standalone-container exception. Adopt one lifecycle-hook contract for direct, HTTP, and MCP invocations, backed by a table-driven integration suite and hook tests, with matching docs and env-var reference updates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Standalone `jaiph run --raw` previously bypassed the shared OTLP/Sentry export hook, and the two exporters were awaited sequentially so two unreachable backends could hold a completed run — and a server's execution-concurrency slot — for up to two full timeouts. `runWorkflowRaw` now fires the shared export hook after the child exits, gated by `shouldExportRawTelemetry` so a user-invoked raw run exports exactly like a normal run while the inner raw process of a Docker run skips it (the host stays the single exporter, so a container run exports exactly once). The OTLP and Sentry exporters now run concurrently under one flush budget configurable via `JAIPH_TELEMETRY_FLUSH_MS`. On long-lived `jaiph serve` / `jaiph mcp` processes the call layer detaches delivery via `deliverRunTelemetryDetached`, marking the run terminal and releasing its concurrency slot before best-effort export; detached failures are tracked in bounded cumulative metrics with warnings capped. `OTEL_*` / `SENTRY_*` stay operator-side — dropped by the prompt-env allowlist and excluded from Docker forwarding, now pinned by tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Company deployments needed both protocols without running two processes; POST /mcp now shares jaiph serve's tools, run registry, concurrency, auth, and cancel contract with the existing REST API. Co-authored-by: Cursor <cursoragent@cursor.com>
HTTP run discovery was an in-memory map, so restarting jaiph serve made
completed run ids unreachable, lost in-flight state, and let client
retries spawn duplicate expensive workflows.
Persist each run's public record as run.json beside its journal (atomic
temp-file + rename at finalize) and reconstruct terminal runs on startup
via loadPersistedRuns, so list/get/events/artifacts keep working across a
restart. A run with a journal but no run.json was running when the
process died; it is reconciled from its WORKFLOW_START line into a new
explicit terminal status "interrupted" and never reported as permanently
running. POST /v1/workflows/{name}/runs now honors an Idempotency-Key
scoped to the workflow plus authenticated principal: the key is reserved
synchronously before any await, a repeat with identical args returns the
original run, and a reused key with different args is
409 E_IDEMPOTENCY_CONFLICT and spawns nothing. The idempotency index is
rebuilt from reconstructed records at startup and evicted with the run,
so it survives restarts and cannot outgrow the registry.
jaiph serve is documented as single-replica by design (per-process
registry, concurrency cap, and idempotency index); k8s.yaml pins
replicas: 1 with a Recreate strategy. Covered by run-store and handler
unit tests plus an integration test that survives a real process restart.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The static JAIPH_SERVE_TOKEN gate is fail-closed but anonymous: no user identity, revocation, per-action authorization, or attribution, which is unsafe once multiple users can invoke arbitrary engineering workflows. Keep the shared bearer token as an explicit single-operator mode and add a standard OIDC/JWT mode configured by issuer, audience, and JWKS discovery, verified with a maintained JWT library rather than custom crypto. Authorize invoke, inspect/artifact, and cancel as separate capabilities so a principal cannot inspect or cancel runs outside its policy. Attach the authenticated principal and a request/correlation ID to run metadata, logs, OTLP resources, and Sentry tags while keeping bearer tokens and secret-bearing claims out of journals. Make /docs and /openapi.json exposure configurable and keep health probes free of credentials. Integration and unit tests cover valid, expired, wrong-audience, wrong-issuer, unknown-key, and insufficient-scope tokens plus audit attribution. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.