diff --git a/README.md b/README.md index cdd852d..337307e 100644 --- a/README.md +++ b/README.md @@ -11,6 +11,8 @@ checkpointed. Executions can pause, survive process restart, resume, replay, and The LLM may reason while the graph controls execution. +![StateOps Control Room showing a resolved incident](docs/assets/stateops-control-room.png) + ## Why this is a graph rather than a pipeline The workflow cannot be reduced to a fixed sequence of LLM calls. Investigation expands dynamically @@ -34,6 +36,8 @@ application design instead of incidental framework behavior. - `get_state`, `get_state_history`, replay, and controlled `update_state` forks; - an atomic Redis `SET NX` boundary for idempotent simulated remediations; - v3 state streaming, structured logs, and opt-in metadata-only OpenTelemetry; +- provider-portable structured-output schemas with application-enforced collection bounds; +- terminal reasoning failures modeled as serializable `WorkflowError` values and `FAILED` state; - a production LLM adapter that knows only the provider-neutral Governed LLM Gateway client. ## Engineering evidence map @@ -48,6 +52,7 @@ application design instead of incidental framework behavior. | Typed update-and-route decisions with `Command` | [`VerificationGraph`](src/stateops/graphs/verification/graph.py) | | History, replay, controlled fork, and nested state reads | [`GraphIncidentRuntime`](src/stateops/graphs/runtime.py) | | Durable checkpoints and restricted deserialization | [`Redis checkpointer`](src/stateops/adapters/persistence/checkpointer.py) | +| Metadata-safe reasoning failure state | [`graphs/failures.py`](src/stateops/graphs/failures.py) | | Atomic effect deduplication | [`Redis remediation ledger`](src/stateops/adapters/remediation/redis_executor.py) | | Real restart/resume proof | [`Redis integration test`](tests/integration/test_redis_runtime.py) | @@ -112,6 +117,8 @@ STATEOPS_LLM_WORKLOAD=stateops.incident.reasoning The external Gateway/Policy Model Router must authorize that dotted workload. StateOps does not assume that it is authorized and does not fall back to a direct provider when it is denied. +Terminal Gateway failures are reduced to stable error metadata and a `FAILED` phase; raw client or +provider exceptions are never added to checkpoint state. ## Quick start @@ -122,7 +129,13 @@ uv sync --frozen --all-groups --extra observability docker compose up --build ``` -Create an incident: +Open [http://127.0.0.1:8000/ui](http://127.0.0.1:8000/ui) to use the **StateOps Control +Room**. The frontend is served by the FastAPI process, uses no CDN or JavaScript framework, and +follows the LangGraph state stream. It can start or load a thread, visualize investigation fan-out, +approve or reject the selected action, and show the verified recovery. **Capture mode** hides the +input controls and expands the dashboard for screenshots. + +To drive the same demonstration through the API, create an incident: ```bash curl --request POST http://127.0.0.1:8000/incidents \ @@ -187,6 +200,7 @@ dedicated node and the atomic Redis claim makes re-execution safe. ## History, replay, fork, and streaming ```text +GET /ui GET /incidents/{incident_id}/history POST /incidents/{incident_id}/replay/{checkpoint_id} POST /incidents/{incident_id}/fork diff --git a/README.pt-br.md b/README.pt-br.md index 3dee870..67337ca 100644 --- a/README.pt-br.md +++ b/README.pt-br.md @@ -12,6 +12,8 @@ processo, ser retomadas, reproduzidas e bifurcadas. O LLM pode raciocinar enquanto o grafo controla a execução. +![StateOps Control Room mostrando um incidente resolvido](docs/assets/stateops-control-room.png) + ## Por que isto é um grafo, e não uma pipeline O workflow não pode ser reduzido a uma sequência fixa de chamadas a LLMs. A investigação se expande @@ -35,6 +37,10 @@ do design da aplicação, em vez de serem comportamentos incidentais do framewor - `get_state`, `get_state_history`, replay e forks controlados com `update_state`; - uma barreira Redis atômica com `SET NX` para remediações simuladas idempotentes; - streaming de estado v3, logs estruturados e OpenTelemetry opcional somente com metadados; +- schemas de saída estruturada portáveis entre providers, com limites de coleção aplicados pela + aplicação; +- falhas terminais de raciocínio modeladas como valores `WorkflowError` serializáveis e estado + `FAILED`; - um adapter de LLM para produção que conhece apenas o cliente provider-neutral do Governed LLM Gateway. @@ -50,6 +56,7 @@ do design da aplicação, em vez de serem comportamentos incidentais do framewor | Decisões tipadas de atualização e rota com `Command` | [`VerificationGraph`](src/stateops/graphs/verification/graph.py) | | Histórico, replay, fork controlado e leitura de estado aninhado | [`GraphIncidentRuntime`](src/stateops/graphs/runtime.py) | | Checkpoints duráveis e desserialização restrita | [`Checkpointer Redis`](src/stateops/adapters/persistence/checkpointer.py) | +| Estado de falha de raciocínio seguro para metadados | [`graphs/failures.py`](src/stateops/graphs/failures.py) | | Deduplicação atômica de efeitos | [`Ledger de remediação Redis`](src/stateops/adapters/remediation/redis_executor.py) | | Prova real de reinicialização e retomada | [`Teste de integração Redis`](tests/integration/test_redis_runtime.py) | @@ -115,6 +122,8 @@ STATEOPS_LLM_WORKLOAD=stateops.incident.reasoning O Gateway/Policy Model Router externo deve autorizar esse workload em formato pontuado. O StateOps não presume que ele esteja autorizado e não recorre diretamente a um provider quando a solicitação é negada. +Falhas terminais do Gateway são reduzidas a metadados estáveis de erro e à fase `FAILED`; exceções +brutas do cliente ou do provider nunca são adicionadas ao estado persistido em checkpoint. ## Início rápido @@ -125,7 +134,13 @@ uv sync --frozen --all-groups --extra observability docker compose up --build ``` -Crie um incidente: +Abra [http://127.0.0.1:8000/ui](http://127.0.0.1:8000/ui) para usar o **StateOps Control +Room**. O frontend é servido pelo próprio processo FastAPI, não usa CDN nem framework JavaScript e +acompanha o stream de estados do LangGraph. Ele permite iniciar ou carregar uma thread, observar o +fan-out das investigações, aprovar ou rejeitar a ação selecionada e conferir a recuperação. O botão +**Modo captura** remove os controles de entrada e expande o painel para screenshots. + +Para operar a mesma demonstração pela API, crie um incidente: ```bash curl --request POST http://127.0.0.1:8000/incidents \ @@ -192,6 +207,7 @@ em um nó dedicado, e a reivindicação atômica no Redis torna a reexecução s ## Histórico, replay, fork e streaming ```text +GET /ui GET /incidents/{incident_id}/history POST /incidents/{incident_id}/replay/{checkpoint_id} POST /incidents/{incident_id}/fork diff --git a/docs/DEMO.md b/docs/DEMO.md index 2799d69..44e2784 100644 --- a/docs/DEMO.md +++ b/docs/DEMO.md @@ -2,16 +2,17 @@ ## Five-minute path -1. Run `docker compose up --build`. -2. Create the checkout incident from the README. -3. Observe three dynamically created investigation branches and `waiting_approval`. -4. Approve `rollback-primary` and observe `resolved`. -5. Read `/incidents/INC-2026-00817/history` and choose the checkpoint whose next node is +1. Run `docker compose up --build` and open `http://127.0.0.1:8000/ui`. +2. Start the prefilled checkout incident in the StateOps Control Room. +3. Observe the dynamically created investigation branches and `waiting_approval`. +4. Capture the interrupted state, approve the selected action, and observe `resolved`. +5. Enable **Capture mode** for a clean screenshot of the resolved graph. +6. Read `/incidents/INC-2026-00817/history` and choose the checkpoint whose next node is `verification`. -6. Fork it with `selected_action_id=restart-secondary`. -7. Approve the restart. Synthetic verification fails, the graph replans, and a new approval interrupt +7. Fork it with `selected_action_id=restart-secondary`. +8. Approve the restart. Synthetic verification fails, the graph replans, and a new approval interrupt recommends rollback. -8. Approve rollback. The original idempotency key is detected and the prior simulated effect evidence +9. Approve rollback. The original idempotency key is detected and the prior simulated effect evidence is reused instead of executing it again. ## Process crash and resume diff --git a/docs/assets/stateops-control-room.png b/docs/assets/stateops-control-room.png new file mode 100644 index 0000000..53da3c4 Binary files /dev/null and b/docs/assets/stateops-control-room.png differ diff --git a/src/stateops/adapters/llm/governed_gateway.py b/src/stateops/adapters/llm/governed_gateway.py index 73e47fa..9cefe7b 100644 --- a/src/stateops/adapters/llm/governed_gateway.py +++ b/src/stateops/adapters/llm/governed_gateway.py @@ -6,6 +6,11 @@ from typing import cast from governed_llm_gateway_client import GatewayClient +from governed_llm_gateway_client.errors import ( + GatewayClientError, + GatewayHTTPError, + GatewayTransportError, +) from governed_llm_gateway_contracts import ( DataClassification, Message, @@ -14,7 +19,9 @@ StructuredOutputSchema, WorkloadRequirements, ) +from governed_llm_gateway_contracts import ExecutionStatus as GatewayExecutionStatus +from stateops.application.ports.reasoner import IncidentReasoningError from stateops.domain.enums import ActionKind, Severity from stateops.domain.models import ( Evidence, @@ -26,9 +33,19 @@ ) -class GatewayReasoningError(RuntimeError): +class GatewayReasoningError(IncidentReasoningError): """Raised when a governed response does not satisfy the StateOps contract.""" + def __init__( + self, + message: str, + *, + code: str = "invalid_gateway_response", + retryable: bool = False, + ) -> None: + """Retain stable Gateway metadata without preserving transport objects.""" + super().__init__(message, code=code, retryable=retryable) + _HYPOTHESES_SCHEMA: dict[str, object] = { "type": "object", @@ -36,7 +53,6 @@ class GatewayReasoningError(RuntimeError): "hypotheses": { "type": "array", "minItems": 1, - "maxItems": 8, "items": { "type": "object", "properties": { @@ -83,13 +99,23 @@ class GatewayReasoningError(RuntimeError): "actions": { "type": "array", "minItems": 1, - "maxItems": 5, "items": { "type": "object", "properties": { "id": {"type": "string"}, "kind": {"type": "string"}, - "parameters": {"type": "object", "additionalProperties": {"type": "string"}}, + "parameters": { + "type": "array", + "items": { + "type": "object", + "properties": { + "name": {"type": "string"}, + "value": {"type": "string"}, + }, + "required": ["name", "value"], + "additionalProperties": False, + }, + }, "risk": {"type": "string"}, }, "required": ["id", "kind", "parameters", "risk"], @@ -131,6 +157,18 @@ def _items(value: object, *, field: str) -> list[Mapping[str, object]]: return [_mapping(item, field=field) for item in value] +def _bounded_items( + value: object, *, field: str, minimum: int, maximum: int +) -> list[Mapping[str, object]]: + """Validate collection limits outside provider-specific JSON Schema dialects.""" + values = _items(value, field=field) + if not minimum <= len(values) <= maximum: + raise GatewayReasoningError( + f"gateway field {field!r} must contain between {minimum} and {maximum} items" + ) + return values + + def _text(value: object, *, field: str) -> str: if not isinstance(value, str) or not value or len(value) > 2000: raise GatewayReasoningError(f"gateway field {field!r} must be bounded text") @@ -179,19 +217,46 @@ async def _generate( role=MessageRole.USER, content=f"{instruction}\nINPUT_JSON={json.dumps(payload, separators=(',', ':'))}", ) - response = await self._client.generate( - workload=self._workload, - messages=(message,), - risk_level=RiskLevel.HIGH, - data_classification=DataClassification.INTERNAL, - requirements=WorkloadRequirements(structured_output=True), - structured_output=StructuredOutputSchema(name=schema_name, schema=schema), - context_tokens_estimated=1000, - max_output_tokens=2000, - provider_timeout_seconds=self._timeout_seconds, - ) + try: + response = await self._client.generate( + workload=self._workload, + messages=(message,), + risk_level=RiskLevel.HIGH, + data_classification=DataClassification.PUBLIC, + requirements=WorkloadRequirements(structured_output=True), + structured_output=StructuredOutputSchema(name=schema_name, schema=schema), + context_tokens_estimated=1000, + max_output_tokens=2000, + provider_timeout_seconds=self._timeout_seconds, + ) + except GatewayHTTPError as exc: + raise GatewayReasoningError( + "gateway rejected the reasoning request", + code=exc.code, + ) from exc + except GatewayTransportError as exc: + raise GatewayReasoningError( + "gateway transport failed", + code="gateway_transport_error", + retryable=True, + ) from exc + except GatewayClientError as exc: + raise GatewayReasoningError( + "gateway client failed", + code="gateway_client_error", + ) from exc + if response.status is not GatewayExecutionStatus.SUCCEEDED: + error = response.error + raise GatewayReasoningError( + "gateway reasoning execution failed", + code=error.code if error is not None else "gateway_execution_failed", + retryable=error.retryable if error is not None else False, + ) if response.content is None: - raise GatewayReasoningError("gateway returned no structured content") + raise GatewayReasoningError( + "gateway returned no structured content", + code="gateway_content_missing", + ) try: decoded = cast(object, json.loads(response.content)) except json.JSONDecodeError as exc: @@ -208,6 +273,9 @@ async def generate_hypotheses( schema_name="stateops_hypotheses", schema=_HYPOTHESES_SCHEMA, ) + hypothesis_items = _bounded_items( + payload.get("hypotheses"), field="hypotheses", minimum=1, maximum=8 + ) hypotheses = tuple( Hypothesis( id=_text(item.get("id"), field="id"), @@ -215,10 +283,8 @@ async def generate_hypotheses( rationale=_text(item.get("rationale"), field="rationale"), confidence=_confidence(item.get("confidence")), ) - for item in _items(payload.get("hypotheses"), field="hypotheses") + for item in hypothesis_items ) - if not hypotheses: - raise GatewayReasoningError("gateway returned no hypotheses") return hypotheses async def investigate( @@ -287,17 +353,22 @@ async def propose_remediations( schema=_ACTIONS_SCHEMA, ) actions: list[RemediationAction] = [] - for item in _items(payload.get("actions"), field="actions"): - parameters = _mapping(item.get("parameters"), field="parameters") + action_items = _bounded_items(payload.get("actions"), field="actions", minimum=1, maximum=5) + for item in action_items: + parameter_items = _bounded_items( + item.get("parameters"), field="parameters", minimum=0, maximum=16 + ) normalized = tuple( sorted( ( - _text(key, field="parameter_name"), - _text(value, field="parameter_value"), + _text(parameter.get("name"), field="parameter_name"), + _text(parameter.get("value"), field="parameter_value"), ) - for key, value in parameters.items() + for parameter in parameter_items ) ) + if len({name for name, _ in normalized}) != len(normalized): + raise GatewayReasoningError("gateway returned duplicate parameter names") try: kind = ActionKind(_text(item.get("kind"), field="kind")) risk = Severity(_text(item.get("risk"), field="risk")) @@ -311,6 +382,4 @@ async def propose_remediations( risk=risk, ) ) - if not actions: - raise GatewayReasoningError("gateway returned no remediation actions") return tuple(actions) diff --git a/src/stateops/application/ports/reasoner.py b/src/stateops/application/ports/reasoner.py index 9d38c66..71e4971 100644 --- a/src/stateops/application/ports/reasoner.py +++ b/src/stateops/application/ports/reasoner.py @@ -12,6 +12,16 @@ ) +class IncidentReasoningError(RuntimeError): + """Metadata-safe failure raised by an incident-reasoning implementation.""" + + def __init__(self, message: str, *, code: str, retryable: bool = False) -> None: + """Retain only stable failure metadata that is safe to checkpoint.""" + super().__init__(message) + self.code = code + self.retryable = retryable + + class IncidentReasoner(Protocol): """Reason about incidents without exposing model or provider selection.""" diff --git a/src/stateops/entrypoints/api/app.py b/src/stateops/entrypoints/api/app.py index 90d30b3..6931460 100644 --- a/src/stateops/entrypoints/api/app.py +++ b/src/stateops/entrypoints/api/app.py @@ -4,11 +4,13 @@ from collections.abc import AsyncIterator, Callable from contextlib import AbstractAsyncContextManager, asynccontextmanager from dataclasses import asdict +from pathlib import Path from typing import Literal, cast from fastapi import FastAPI, HTTPException from fastapi.encoders import jsonable_encoder -from fastapi.responses import StreamingResponse +from fastapi.responses import FileResponse, StreamingResponse +from fastapi.staticfiles import StaticFiles from pydantic import Field from pydantic_settings import BaseSettings, SettingsConfigDict @@ -26,6 +28,18 @@ from stateops.graphs.incident_graph import build_incident_graph from stateops.graphs.runtime import GraphIncidentRuntime +_UI_ROOT = Path(__file__).with_name("static") +_UI_SECURITY_HEADERS = { + "Cache-Control": "no-store", + "Content-Security-Policy": ( + "default-src 'none'; script-src 'self'; style-src 'self'; img-src 'self' data:; " + "connect-src 'self'; font-src 'self'; base-uri 'none'; form-action 'self'; " + "frame-ancestors 'none'" + ), + "Referrer-Policy": "no-referrer", + "X-Content-Type-Options": "nosniff", +} + class StateOpsSettings(BaseSettings): """Process configuration with no provider credentials.""" @@ -119,6 +133,17 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]: description="Durable, replayable incident-response state machine.", lifespan=lifespan, ) + app.mount("/ui/assets", StaticFiles(directory=_UI_ROOT), name="ui-assets") + + @app.get("/ui", include_in_schema=False) + @app.get("/ui/", include_in_schema=False) + async def operator_console() -> FileResponse: + """Serve the dependency-free local demonstration console.""" + return FileResponse( + _UI_ROOT / "index.html", + media_type="text/html", + headers=_UI_SECURITY_HEADERS, + ) @app.get("/healthz") async def health() -> dict[str, str]: diff --git a/src/stateops/entrypoints/api/static/app.css b/src/stateops/entrypoints/api/static/app.css new file mode 100644 index 0000000..0c3751f --- /dev/null +++ b/src/stateops/entrypoints/api/static/app.css @@ -0,0 +1,1427 @@ +:root { + --bg: #050810; + --bg-soft: #090e18; + --panel: rgba(12, 18, 30, 0.88); + --panel-strong: #0d1421; + --line: rgba(157, 177, 206, 0.14); + --line-strong: rgba(157, 177, 206, 0.28); + --text: #edf4ff; + --muted: #78869a; + --muted-bright: #a8b4c7; + --mint: #6fffc8; + --mint-soft: rgba(111, 255, 200, 0.12); + --blue: #70aaff; + --blue-soft: rgba(112, 170, 255, 0.13); + --amber: #ffbd66; + --amber-soft: rgba(255, 189, 102, 0.12); + --red: #ff7185; + --red-soft: rgba(255, 113, 133, 0.12); + --radius: 14px; + --mono: "SFMono-Regular", "Cascadia Code", "Roboto Mono", monospace; + --sans: Inter, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; +} + +* { + box-sizing: border-box; +} + +[hidden] { + display: none !important; +} + +html { + min-width: 320px; + background: var(--bg); +} + +body { + min-height: 100vh; + margin: 0; + overflow-x: hidden; + color: var(--text); + background: + linear-gradient(rgba(95, 132, 184, 0.025) 1px, transparent 1px), + linear-gradient(90deg, rgba(95, 132, 184, 0.025) 1px, transparent 1px), + radial-gradient(circle at 28% -15%, rgba(58, 117, 183, 0.12), transparent 38%), + var(--bg); + background-size: 42px 42px, 42px 42px, auto, auto; + font-family: var(--sans); + font-feature-settings: "ss01", "cv11"; + -webkit-font-smoothing: antialiased; +} + +button, +input { + font: inherit; +} + +button { + color: inherit; +} + +button:focus-visible, +input:focus-visible { + outline: 2px solid var(--mint); + outline-offset: 2px; +} + +.ambient { + position: fixed; + z-index: -1; + width: 420px; + height: 420px; + border-radius: 50%; + filter: blur(110px); + opacity: 0.09; + pointer-events: none; +} + +.ambient-one { + top: 18%; + left: 5%; + background: #327dcc; +} + +.ambient-two { + right: 4%; + bottom: 0; + background: #2fd4a0; +} + +.topbar { + position: relative; + z-index: 10; + display: grid; + grid-template-columns: 330px 1fr auto; + align-items: center; + height: 76px; + padding: 0 28px; + border-bottom: 1px solid var(--line); + background: rgba(5, 8, 16, 0.82); + backdrop-filter: blur(20px); +} + +.brand { + display: inline-flex; + align-items: center; + gap: 13px; + width: fit-content; + color: var(--text); + text-decoration: none; +} + +.brand-mark { + position: relative; + display: grid; + grid-template-columns: repeat(2, 7px); + grid-template-rows: repeat(2, 7px); + gap: 7px; + width: 34px; + height: 34px; + padding: 3px; +} + +.brand-mark::before, +.brand-mark::after { + position: absolute; + z-index: -1; + content: ""; + background: rgba(111, 255, 200, 0.34); +} + +.brand-mark::before { + top: 9px; + left: 8px; + width: 18px; + height: 1px; +} + +.brand-mark::after { + top: 8px; + left: 16px; + width: 1px; + height: 18px; +} + +.brand-mark span { + border: 1px solid var(--mint); + border-radius: 2px; + background: var(--bg); + box-shadow: 0 0 10px rgba(111, 255, 200, 0.28); +} + +.brand-mark span:first-child { + background: var(--mint); +} + +.brand strong { + display: block; + letter-spacing: 0.14em; + font-size: 14px; + line-height: 1.1; +} + +.brand small { + display: block; + margin-top: 4px; + color: var(--muted); + letter-spacing: 0.13em; + font-family: var(--mono); + font-size: 8px; +} + +.topbar-context { + padding-left: 25px; + border-left: 1px solid var(--line); +} + +.topbar-context span, +.workspace-meta span, +.outcome-metrics span { + display: block; + color: var(--muted); + letter-spacing: 0.12em; + font-family: var(--mono); + font-size: 8px; +} + +.topbar-context strong { + display: block; + max-width: 480px; + margin-top: 5px; + overflow: hidden; + color: var(--muted-bright); + letter-spacing: 0.04em; + font-family: var(--mono); + font-size: 11px; + text-overflow: ellipsis; + white-space: nowrap; +} + +.topbar-actions { + display: flex; + align-items: center; + gap: 12px; +} + +.health-pill, +.ghost-button { + display: inline-flex; + align-items: center; + gap: 9px; + height: 36px; + padding: 0 13px; + border: 1px solid var(--line); + border-radius: 8px; + background: rgba(12, 18, 30, 0.7); + color: var(--muted-bright); + letter-spacing: 0.04em; + font-family: var(--mono); + font-size: 10px; +} + +.pulse-dot { + width: 6px; + height: 6px; + border-radius: 50%; + background: var(--amber); + box-shadow: 0 0 0 4px rgba(255, 189, 102, 0.1); +} + +.health-pill[data-status="online"] .pulse-dot { + background: var(--mint); + box-shadow: 0 0 0 4px var(--mint-soft), 0 0 12px rgba(111, 255, 200, 0.6); +} + +.health-pill[data-status="offline"] .pulse-dot { + background: var(--red); + box-shadow: 0 0 0 4px var(--red-soft); +} + +.ghost-button { + cursor: pointer; + transition: border-color 160ms ease, color 160ms ease, background 160ms ease; +} + +.ghost-button:hover, +.ghost-button[aria-pressed="true"] { + border-color: rgba(111, 255, 200, 0.38); + background: var(--mint-soft); + color: var(--mint); +} + +.capture-icon { + position: relative; + width: 13px; + height: 10px; + border: 1px solid currentColor; + border-radius: 2px; +} + +.capture-icon::before { + position: absolute; + top: 2px; + left: 4px; + width: 3px; + height: 3px; + border: 1px solid currentColor; + border-radius: 50%; + content: ""; +} + +.shell { + display: grid; + grid-template-columns: 310px minmax(0, 1fr); + gap: 20px; + width: min(1800px, 100%); + margin: 0 auto; + padding: 20px 28px 32px; +} + +.panel { + border: 1px solid var(--line); + border-radius: var(--radius); + background: linear-gradient(145deg, rgba(14, 21, 34, 0.94), rgba(9, 14, 24, 0.91)); + box-shadow: 0 18px 50px rgba(0, 0, 0, 0.14); +} + +.control-panel { + position: sticky; + top: 20px; + align-self: start; + min-height: calc(100vh - 116px); + padding: 22px; +} + +.panel-heading, +.card-heading, +.workspace-header, +.title-row, +.metric-label, +.metric-values { + display: flex; + align-items: center; + justify-content: space-between; +} + +.section-index { + color: var(--blue); + letter-spacing: 0.15em; + font-family: var(--mono); + font-size: 8px; +} + +h1, +h2, +h3, +p { + margin-top: 0; +} + +h1 { + margin-bottom: 0; + font-size: 20px; + font-weight: 570; + letter-spacing: -0.025em; +} + +h2 { + margin: 6px 0 0; + font-size: 21px; + font-weight: 560; + letter-spacing: -0.025em; +} + +h3 { + margin: 5px 0 0; + font-size: 15px; + font-weight: 570; + letter-spacing: -0.01em; +} + +.icon-button { + display: grid; + width: 30px; + height: 30px; + place-items: center; + border: 1px solid var(--line); + border-radius: 7px; + background: transparent; + color: var(--muted); + cursor: pointer; +} + +.icon-button:hover { + border-color: var(--line-strong); + color: var(--mint); +} + +.panel-copy { + margin: 20px 0 22px; + color: var(--muted); + font-size: 12px; + line-height: 1.6; +} + +.field { + display: block; + margin-bottom: 15px; +} + +.field > span:first-child { + display: block; + margin-bottom: 7px; + color: var(--muted-bright); + letter-spacing: 0.02em; + font-size: 10px; + font-weight: 550; +} + +.field input { + width: 100%; + height: 40px; + padding: 0 12px; + border: 1px solid var(--line); + border-radius: 7px; + background: rgba(4, 8, 15, 0.72); + color: var(--text); + font-family: var(--mono); + font-size: 11px; + transition: border-color 150ms ease, background 150ms ease; +} + +.field input:hover, +.field input:focus { + border-color: rgba(112, 170, 255, 0.38); + background: rgba(8, 14, 25, 0.92); +} + +.field-pair { + display: grid; + grid-template-columns: 1fr 1fr; + gap: 10px; +} + +.input-suffix { + position: relative; + display: block; +} + +.input-suffix input { + padding-right: 29px; +} + +.input-suffix b { + position: absolute; + top: 13px; + right: 11px; + color: var(--muted); + font-family: var(--mono); + font-size: 10px; + font-weight: 500; + pointer-events: none; +} + +.critical-input input { + color: var(--red); +} + +.primary-button, +.secondary-button, +.approve-button, +.reject-button { + display: flex; + align-items: center; + justify-content: center; + border-radius: 7px; + letter-spacing: 0.02em; + font-size: 11px; + font-weight: 650; + cursor: pointer; + transition: transform 160ms ease, opacity 160ms ease, box-shadow 160ms ease; +} + +.primary-button { + width: 100%; + height: 44px; + margin-top: 5px; + border: 0; + background: var(--mint); + color: #04110d; + box-shadow: 0 10px 28px rgba(111, 255, 200, 0.14); +} + +.primary-button:hover, +.approve-button:hover { + transform: translateY(-1px); + box-shadow: 0 13px 32px rgba(111, 255, 200, 0.22); +} + +.primary-button:disabled, +.approve-button:disabled, +.reject-button:disabled, +.secondary-button:disabled { + cursor: wait; + opacity: 0.48; + transform: none; +} + +.button-spark { + margin-right: 9px; + font-size: 7px; +} + +.button-arrow { + margin-left: auto; + padding-right: 3px; + font-size: 16px; +} + +.secondary-button { + width: 100%; + height: 38px; + margin-top: 9px; + border: 1px solid var(--line); + background: transparent; + color: var(--muted-bright); +} + +.secondary-button:hover { + border-color: var(--line-strong); + background: rgba(112, 170, 255, 0.06); +} + +.runtime-note { + display: flex; + gap: 11px; + margin-top: 22px; + padding-top: 18px; + border-top: 1px solid var(--line); +} + +.runtime-icon { + display: grid; + flex: 0 0 28px; + width: 28px; + height: 28px; + place-items: center; + border: 1px solid rgba(255, 113, 133, 0.3); + border-radius: 7px; + background: var(--red-soft); + color: var(--red); + font-family: var(--mono); + font-size: 10px; +} + +.runtime-note strong { + display: block; + margin-bottom: 4px; + color: var(--muted-bright); + font-size: 10px; +} + +.runtime-note p { + margin-bottom: 0; + color: var(--muted); + font-size: 9px; + line-height: 1.5; +} + +.workspace { + min-width: 0; +} + +.workspace-header { + min-height: 62px; + padding: 0 4px 14px; +} + +.title-row { + justify-content: flex-start; + gap: 12px; +} + +.phase-badge, +.card-status, +.risk-badge, +.live-indicator { + display: inline-flex; + align-items: center; + min-height: 20px; + padding: 0 8px; + border: 1px solid var(--line); + border-radius: 4px; + background: rgba(120, 134, 154, 0.06); + color: var(--muted); + letter-spacing: 0.09em; + font-family: var(--mono); + font-size: 7px; + font-weight: 650; +} + +.phase-badge[data-phase="running"] { + border-color: rgba(112, 170, 255, 0.26); + background: var(--blue-soft); + color: var(--blue); +} + +.phase-badge[data-phase="waiting_approval"] { + border-color: rgba(255, 189, 102, 0.3); + background: var(--amber-soft); + color: var(--amber); +} + +.phase-badge[data-phase="resolved"] { + border-color: rgba(111, 255, 200, 0.3); + background: var(--mint-soft); + color: var(--mint); +} + +.phase-badge[data-phase="failed"], +.phase-badge[data-phase="escalated"] { + border-color: rgba(255, 113, 133, 0.3); + background: var(--red-soft); + color: var(--red); +} + +.workspace-meta { + display: flex; + gap: 25px; +} + +.workspace-meta div { + min-width: 58px; +} + +.workspace-meta strong { + display: block; + margin-top: 5px; + color: var(--muted-bright); + letter-spacing: 0.04em; + font-family: var(--mono); + font-size: 11px; + text-transform: uppercase; +} + +.workspace-meta strong.critical { + color: var(--red); +} + +.graph-panel { + position: relative; + padding: 18px 18px 16px; + overflow: hidden; +} + +.graph-panel::before { + position: absolute; + top: 0; + left: 0; + width: 100%; + height: 1px; + content: ""; + background: linear-gradient(90deg, transparent, rgba(112, 170, 255, 0.34), transparent); +} + +.graph-track { + display: flex; + align-items: center; + justify-content: space-between; + min-width: 880px; +} + +.graph-node { + position: relative; + display: flex; + flex: 0 0 88px; + flex-direction: column; + align-items: center; + justify-content: center; + min-height: 85px; + padding: 10px 7px; + border: 1px solid var(--line); + border-radius: 8px; + background: rgba(5, 10, 18, 0.62); + text-align: center; + transition: border-color 250ms ease, background 250ms ease, box-shadow 250ms ease; +} + +.node-number { + position: absolute; + top: 5px; + left: 7px; + color: #465166; + font-family: var(--mono); + font-size: 6px; +} + +.node-icon { + display: grid; + width: 24px; + height: 24px; + margin-bottom: 7px; + place-items: center; + border: 1px solid var(--line-strong); + border-radius: 6px; + color: var(--muted); + font-family: var(--mono); + font-size: 11px; +} + +.graph-node strong { + color: var(--muted-bright); + letter-spacing: 0.08em; + font-family: var(--mono); + font-size: 7px; +} + +.graph-node small { + margin-top: 4px; + color: #58657a; + font-size: 7px; +} + +.graph-node small b { + color: inherit; +} + +.graph-node.complete { + border-color: rgba(111, 255, 200, 0.22); + background: rgba(111, 255, 200, 0.04); +} + +.graph-node.complete .node-icon { + border-color: rgba(111, 255, 200, 0.32); + background: var(--mint-soft); + color: var(--mint); +} + +.graph-node.complete strong { + color: #bcebd9; +} + +.graph-node.active { + border-color: rgba(112, 170, 255, 0.62); + background: var(--blue-soft); + box-shadow: 0 0 0 1px rgba(112, 170, 255, 0.08), 0 0 30px rgba(74, 137, 226, 0.12); +} + +.graph-node.active .node-icon { + border-color: var(--blue); + background: rgba(112, 170, 255, 0.12); + color: var(--blue); + box-shadow: 0 0 14px rgba(112, 170, 255, 0.22); +} + +.graph-node.active strong { + color: var(--blue); +} + +.graph-node[data-node="approval"].active { + border-color: rgba(255, 189, 102, 0.68); + background: var(--amber-soft); + box-shadow: 0 0 30px rgba(255, 189, 102, 0.1); +} + +.graph-node[data-node="approval"].active .node-icon, +.graph-node[data-node="approval"].active strong { + border-color: var(--amber); + color: var(--amber); +} + +.graph-node[data-node="verify"].resolved { + border-color: rgba(111, 255, 200, 0.62); + background: var(--mint-soft); + box-shadow: 0 0 30px rgba(111, 255, 200, 0.1); +} + +.graph-node[data-node="verify"].resolved .node-icon, +.graph-node[data-node="verify"].resolved strong { + border-color: var(--mint); + color: var(--mint); +} + +.graph-edge { + position: relative; + flex: 1 1 18px; + min-width: 10px; + max-width: 40px; + height: 1px; + margin: 0 5px; + background: var(--line-strong); +} + +.graph-edge::after { + position: absolute; + top: -2px; + right: -1px; + width: 4px; + height: 4px; + border-top: 1px solid var(--muted); + border-right: 1px solid var(--muted); + content: ""; + transform: rotate(45deg); +} + +.graph-edge.complete { + background: rgba(111, 255, 200, 0.45); +} + +.branch-dots { + position: absolute; + bottom: -5px; + display: flex; + gap: 3px; +} + +.branch-dots i { + width: 5px; + height: 5px; + border: 1px solid rgba(112, 170, 255, 0.5); + border-radius: 50%; + background: var(--panel-strong); +} + +.branch-dots i.complete { + border-color: var(--mint); + background: var(--mint); + box-shadow: 0 0 6px rgba(111, 255, 200, 0.5); +} + +.approval-panel, +.outcome-panel { + display: flex; + align-items: center; + gap: 15px; + margin-top: 14px; + padding: 14px 16px; +} + +.approval-panel { + border-color: rgba(255, 189, 102, 0.32); + background: linear-gradient(100deg, rgba(255, 189, 102, 0.1), rgba(14, 21, 34, 0.94) 58%); +} + +.approval-signal, +.outcome-check { + display: grid; + flex: 0 0 38px; + width: 38px; + height: 38px; + place-items: center; + border: 1px solid rgba(255, 189, 102, 0.42); + border-radius: 9px; + background: var(--amber-soft); + color: var(--amber); + font-family: var(--mono); + font-size: 15px; +} + +.approval-copy { + min-width: 0; +} + +.approval-copy > span, +.outcome-panel > div > span { + color: var(--amber); + letter-spacing: 0.12em; + font-family: var(--mono); + font-size: 7px; +} + +.approval-copy h3, +.outcome-panel h3 { + margin: 3px 0; +} + +.approval-copy p, +.outcome-panel p { + max-width: 680px; + margin: 0; + overflow: hidden; + color: var(--muted); + font-size: 10px; + text-overflow: ellipsis; + white-space: nowrap; +} + +.approval-actions { + display: flex; + gap: 8px; + margin-left: auto; +} + +.approve-button, +.reject-button { + height: 35px; + padding: 0 16px; +} + +.approve-button { + gap: 9px; + border: 0; + background: var(--mint); + color: #04110d; +} + +.reject-button { + border: 1px solid var(--line-strong); + background: rgba(5, 10, 18, 0.45); + color: var(--muted-bright); +} + +.reject-button:hover { + border-color: rgba(255, 113, 133, 0.42); + color: var(--red); +} + +.metrics-grid { + display: grid; + grid-template-columns: repeat(3, minmax(0, 1fr)); + gap: 14px; + margin-top: 14px; +} + +.metric-card { + position: relative; + min-height: 105px; + padding: 15px 17px; + overflow: hidden; +} + +.metric-label span { + color: var(--muted); + letter-spacing: 0.11em; + font-family: var(--mono); + font-size: 7px; +} + +.metric-label b { + padding: 3px 5px; + border-radius: 3px; + background: var(--mint-soft); + color: var(--mint); + letter-spacing: 0.08em; + font-family: var(--mono); + font-size: 6px; + font-weight: 600; +} + +.metric-values { + justify-content: flex-start; + gap: 11px; + margin-top: 12px; +} + +.metric-values strong { + font-family: var(--mono); + font-size: 23px; + font-weight: 500; + letter-spacing: -0.06em; +} + +.metric-delta { + padding: 3px 5px; + border-radius: 3px; + background: var(--red-soft); + color: var(--red); + letter-spacing: 0.06em; + font-family: var(--mono); + font-size: 6px; +} + +.metric-delta.recovered { + background: var(--mint-soft); + color: var(--mint); +} + +.metric-delta.neutral { + background: var(--blue-soft); + color: var(--blue); +} + +.sparkline { + position: absolute; + right: 10px; + bottom: 11px; + left: 17px; + display: flex; + align-items: flex-end; + gap: 3px; + height: 22px; + opacity: 0.5; +} + +.sparkline i { + flex: 1; + min-width: 2px; + border-radius: 1px 1px 0 0; + background: var(--red); +} + +.sparkline i:nth-child(1) { height: 12%; } +.sparkline i:nth-child(2) { height: 17%; } +.sparkline i:nth-child(3) { height: 14%; } +.sparkline i:nth-child(4) { height: 22%; } +.sparkline i:nth-child(5) { height: 29%; } +.sparkline i:nth-child(6) { height: 82%; } +.sparkline i:nth-child(7) { height: 100%; } +.sparkline i:nth-child(8) { height: 94%; } + +.sparkline.recovered i:nth-child(7), +.sparkline.recovered i:nth-child(8) { + height: 15%; + background: var(--mint); +} + +.branch-meter { + display: flex; + gap: 5px; + margin-top: 10px; +} + +.branch-meter i { + height: 4px; + flex: 1; + border-radius: 2px; + background: #1a2638; +} + +.branch-meter i.complete { + background: var(--blue); + box-shadow: 0 0 8px rgba(112, 170, 255, 0.22); +} + +.confidence-track { + height: 3px; + margin-top: 12px; + overflow: hidden; + border-radius: 3px; + background: #1a2638; +} + +.confidence-track span { + display: block; + width: 0; + height: 100%; + border-radius: inherit; + background: linear-gradient(90deg, var(--blue), var(--mint)); + box-shadow: 0 0 9px rgba(111, 255, 200, 0.32); + transition: width 450ms ease; +} + +.details-grid { + display: grid; + grid-template-columns: minmax(0, 1.4fr) minmax(250px, 0.8fr) minmax(285px, 0.9fr); + gap: 14px; + margin-top: 14px; +} + +.detail-card { + min-width: 0; + min-height: 230px; + padding: 17px; +} + +.card-heading { + align-items: flex-start; + margin-bottom: 15px; +} + +.card-status.ready { + border-color: rgba(111, 255, 200, 0.24); + background: var(--mint-soft); + color: var(--mint); +} + +.detail-summary { + display: -webkit-box; + margin-bottom: 16px; + overflow: hidden; + color: #bdc8d8; + font-size: 10px; + line-height: 1.65; + -webkit-box-orient: vertical; + -webkit-line-clamp: 6; +} + +.detail-summary.muted { + color: var(--muted); +} + +.evidence-strip { + display: flex; + gap: 6px; + margin-top: auto; +} + +.evidence-chip { + display: inline-flex; + align-items: center; + gap: 5px; + height: 22px; + padding: 0 7px; + border: 1px solid var(--line); + border-radius: 4px; + color: var(--muted); + font-family: var(--mono); + font-size: 7px; +} + +.evidence-chip::before { + width: 4px; + height: 4px; + border-radius: 50%; + background: var(--red); + content: ""; +} + +.evidence-chip.supports { + border-color: rgba(111, 255, 200, 0.18); + color: #9ebdac; +} + +.evidence-chip.supports::before { + background: var(--mint); +} + +.risk-badge.medium, +.risk-badge.high { + border-color: rgba(255, 189, 102, 0.28); + background: var(--amber-soft); + color: var(--amber); +} + +.risk-badge.low { + border-color: rgba(111, 255, 200, 0.24); + background: var(--mint-soft); + color: var(--mint); +} + +.action-empty { + display: flex; + min-height: 130px; + flex-direction: column; + align-items: center; + justify-content: center; + color: #465166; +} + +.action-empty span { + font-family: var(--mono); + font-size: 22px; +} + +.action-empty p { + margin: 8px 0 0; + font-size: 9px; +} + +.action-name { + display: block; + margin-bottom: 14px; + color: var(--blue); + font-family: var(--mono); + font-size: 12px; + font-weight: 550; + word-break: break-word; +} + +.parameter-list { + margin: 0; +} + +.parameter-row { + display: grid; + grid-template-columns: 92px minmax(0, 1fr); + gap: 8px; + padding: 7px 0; + border-top: 1px solid var(--line); + font-family: var(--mono); + font-size: 7px; + line-height: 1.45; +} + +.parameter-row dt { + color: var(--muted); +} + +.parameter-row dd { + margin: 0; + overflow: hidden; + color: var(--muted-bright); + text-overflow: ellipsis; + white-space: nowrap; +} + +.live-indicator { + border: 0; + background: transparent; + color: var(--mint); +} + +.live-indicator i { + width: 4px; + height: 4px; + margin-right: 5px; + border-radius: 50%; + background: var(--mint); + box-shadow: 0 0 7px rgba(111, 255, 200, 0.7); +} + +.timeline-list { + position: relative; + margin: 0; + padding: 0; + list-style: none; +} + +.timeline-list::before { + position: absolute; + top: 7px; + bottom: 7px; + left: 4px; + width: 1px; + background: var(--line); + content: ""; +} + +.timeline-item { + position: relative; + display: grid; + grid-template-columns: minmax(0, 1fr) auto; + gap: 8px; + padding: 0 0 11px 18px; +} + +.timeline-item:last-child { + padding-bottom: 0; +} + +.timeline-item::before { + position: absolute; + top: 4px; + left: 1px; + width: 7px; + height: 7px; + border: 2px solid var(--panel-strong); + border-radius: 50%; + background: var(--blue); + content: ""; + box-shadow: 0 0 7px rgba(112, 170, 255, 0.42); +} + +.timeline-item:last-child::before { + background: var(--mint); + box-shadow: 0 0 7px rgba(111, 255, 200, 0.52); +} + +.timeline-event { + overflow: hidden; + color: var(--muted-bright); + font-family: var(--mono); + font-size: 7px; + text-overflow: ellipsis; + white-space: nowrap; +} + +.timeline-phase, +.timeline-time { + color: var(--muted); + font-family: var(--mono); + font-size: 6px; +} + +.timeline-phase { + display: block; + margin-top: 2px; +} + +.timeline-empty { + padding: 20px 0 0 18px; + color: var(--muted); + font-size: 9px; +} + +.outcome-panel { + border-color: rgba(111, 255, 200, 0.3); + background: linear-gradient(100deg, rgba(111, 255, 200, 0.1), rgba(14, 21, 34, 0.94) 58%); +} + +.outcome-check { + border-color: rgba(111, 255, 200, 0.42); + background: var(--mint-soft); + color: var(--mint); +} + +.outcome-panel > div > span { + color: var(--mint); +} + +.outcome-metrics { + display: flex; + align-items: center; + gap: 14px; + margin-left: auto; +} + +.outcome-metrics div { + min-width: 55px; +} + +.outcome-metrics strong { + display: block; + margin-top: 3px; + font-family: var(--mono); + font-size: 15px; +} + +.outcome-metrics div:last-child strong { + color: var(--mint); +} + +.outcome-arrow { + color: var(--mint); +} + +.status-message { + min-height: 18px; + margin: 10px 4px 0; + color: var(--muted); + font-family: var(--mono); + font-size: 8px; +} + +.status-message.error { + color: var(--red); +} + +.status-message.running { + color: var(--blue); +} + +.is-busy .graph-node.active .node-icon { + animation: breathe 1.5s ease-in-out infinite; +} + +.capture-mode .topbar { + grid-template-columns: 260px 1fr auto; +} + +.capture-mode .shell { + grid-template-columns: 1fr; +} + +.capture-mode .control-panel { + display: none; +} + +.capture-mode .workspace-header { + padding-top: 4px; +} + +@keyframes breathe { + 0%, 100% { box-shadow: 0 0 8px rgba(112, 170, 255, 0.12); } + 50% { box-shadow: 0 0 20px rgba(112, 170, 255, 0.42); } +} + +@media (max-width: 1240px) { + .graph-panel { + overflow-x: auto; + } + + .details-grid { + grid-template-columns: 1fr 1fr; + } + + .timeline-card { + grid-column: 1 / -1; + min-height: 180px; + } +} + +@media (max-width: 900px) { + .topbar { + grid-template-columns: 1fr auto; + padding: 0 18px; + } + + .topbar-context, + .health-pill { + display: none; + } + + .shell { + grid-template-columns: 1fr; + padding: 16px 18px 28px; + } + + .control-panel { + position: static; + min-height: auto; + } + + .metrics-grid { + grid-template-columns: 1fr; + } +} + +@media (max-width: 640px) { + .topbar { + height: 66px; + } + + .ghost-button { + width: 36px; + padding: 0; + justify-content: center; + font-size: 0; + } + + .workspace-header { + align-items: flex-start; + flex-direction: column; + gap: 14px; + } + + .workspace-meta { + width: 100%; + justify-content: space-between; + } + + .details-grid { + grid-template-columns: 1fr; + } + + .timeline-card { + grid-column: auto; + } + + .approval-panel, + .outcome-panel { + align-items: flex-start; + flex-wrap: wrap; + } + + .approval-actions, + .outcome-metrics { + width: 100%; + margin-left: 53px; + } + + .approval-copy p, + .outcome-panel p { + white-space: normal; + } +} + +@media (prefers-reduced-motion: reduce) { + *, + *::before, + *::after { + scroll-behavior: auto !important; + animation-duration: 0.01ms !important; + animation-iteration-count: 1 !important; + transition-duration: 0.01ms !important; + } +} diff --git a/src/stateops/entrypoints/api/static/app.js b/src/stateops/entrypoints/api/static/app.js new file mode 100644 index 0000000..3e4e028 --- /dev/null +++ b/src/stateops/entrypoints/api/static/app.js @@ -0,0 +1,572 @@ +"use strict"; + +const byId = (id) => document.getElementById(id); + +const ui = { + actionContent: byId("action-content"), + actionEmpty: byId("action-empty"), + actionName: byId("action-name"), + approvalDescription: byId("approval-description"), + approvalPanel: byId("approval-panel"), + approveButton: byId("approve-button"), + attemptValue: byId("attempt-value"), + branchCount: byId("branch-count"), + branchDots: byId("branch-dots"), + branchMeter: byId("branch-meter"), + captureButton: byId("capture-button"), + checkpointValue: byId("checkpoint-value"), + confidenceFill: byId("confidence-fill"), + confidenceValue: byId("confidence-value"), + currentError: byId("current-error"), + deployment: byId("deployment"), + errorAfter: byId("error-after"), + errorBefore: byId("error-before"), + errorDelta: byId("error-delta"), + evidenceCount: byId("evidence-count"), + evidenceStrip: byId("evidence-strip"), + form: byId("incident-form"), + headerIncident: byId("header-incident"), + healthLabel: byId("health-label"), + healthPill: byId("health-pill"), + hypothesisCount: byId("hypothesis-count"), + incidentId: byId("incident-id"), + loadButton: byId("load-button"), + outcomeAfter: byId("outcome-after"), + outcomeBefore: byId("outcome-before"), + outcomeDescription: byId("outcome-description"), + outcomePanel: byId("outcome-panel"), + parameterList: byId("parameter-list"), + phaseBadge: byId("phase-badge"), + reasoningStatus: byId("reasoning-status"), + rejectButton: byId("reject-button"), + resetButton: byId("reset-button"), + riskBadge: byId("risk-badge"), + rootHypothesis: byId("root-hypothesis"), + rootSummary: byId("root-summary"), + service: byId("service"), + severityValue: byId("severity-value"), + sparkline: document.querySelector(".sparkline"), + startButton: byId("start-button"), + statusMessage: byId("status-message"), + timelineList: byId("timeline-list"), +}; + +const graphNodes = Array.from(document.querySelectorAll(".graph-node")); +const graphEdges = Array.from(document.querySelectorAll(".graph-edge")); + +const phasePosition = { + received: 0, + enriching: 0, + classified: 0, + hypotheses_generated: 1, + investigating: 2, + evidence_ready: 3, + planning: 4, + waiting_approval: 5, + executing: 6, + verifying: 7, + resolved: 8, + escalated: 8, + failed: 8, +}; + +let latestState = null; +let busy = false; + +function makeIncidentId() { + const now = new Date(); + const date = now.toISOString().slice(0, 10).replaceAll("-", ""); + const time = now.toTimeString().slice(0, 8).replaceAll(":", ""); + return `INC-DEMO-${date}-${time}`; +} + +function setText(element, value) { + element.textContent = value === null || value === undefined || value === "" ? "—" : String(value); +} + +function setStatus(message, kind = "") { + setText(ui.statusMessage, message); + ui.statusMessage.className = `status-message ${kind}`.trim(); +} + +function setBusy(nextBusy, message = "") { + busy = nextBusy; + document.body.classList.toggle("is-busy", nextBusy); + ui.startButton.disabled = nextBusy; + ui.loadButton.disabled = nextBusy; + ui.approveButton.disabled = nextBusy; + ui.rejectButton.disabled = nextBusy; + if (message) { + setStatus(message, nextBusy ? "running" : ""); + } +} + +function formatPercent(value) { + const numeric = Number(value); + return Number.isFinite(numeric) ? `${numeric.toFixed(numeric < 1 ? 1 : 1)}%` : "—"; +} + +function formatPhase(phase) { + return String(phase || "idle").replaceAll("_", " ").toUpperCase(); +} + +function formatTime(value) { + if (!value) { + return "—"; + } + const date = new Date(value); + if (Number.isNaN(date.getTime())) { + return "—"; + } + return new Intl.DateTimeFormat("pt-BR", { + hour: "2-digit", + minute: "2-digit", + second: "2-digit", + hour12: false, + }).format(date); +} + +function normalizedParameters(parameters) { + if (!Array.isArray(parameters)) { + return []; + } + return parameters.flatMap((parameter) => { + if (Array.isArray(parameter) && parameter.length >= 2) { + return [{ name: parameter[0], value: parameter[1] }]; + } + if (parameter && typeof parameter === "object" && "name" in parameter && "value" in parameter) { + return [{ name: parameter.name, value: parameter.value }]; + } + return []; + }); +} + +function createBranchIndicators(hypotheses, evidence) { + const completed = new Set(evidence.map((item) => item.hypothesis_id)); + ui.branchDots.replaceChildren(); + ui.branchMeter.replaceChildren(); + + hypotheses.forEach((hypothesis) => { + const dot = document.createElement("i"); + dot.classList.toggle("complete", completed.has(hypothesis.id)); + dot.title = String(hypothesis.id || "branch"); + ui.branchDots.append(dot); + + const bar = document.createElement("i"); + bar.classList.toggle("complete", completed.has(hypothesis.id)); + ui.branchMeter.append(bar); + }); +} + +function renderGraph(phase) { + const position = phasePosition[phase] ?? -1; + const terminal = phase === "resolved"; + + graphNodes.forEach((node, index) => { + node.classList.toggle("complete", terminal || index < position); + node.classList.toggle("active", !terminal && index === position); + node.classList.toggle("resolved", terminal && index === graphNodes.length - 1); + }); + graphEdges.forEach((edge, index) => { + edge.classList.toggle("complete", terminal || index < position); + }); +} + +function renderEvidence(evidence) { + ui.evidenceStrip.replaceChildren(); + evidence.slice(0, 5).forEach((item) => { + const chip = document.createElement("span"); + chip.className = "evidence-chip"; + chip.classList.toggle("supports", item.supports === true); + chip.textContent = `${item.hypothesis_id || item.id || "E"} ${item.supports ? "SUPPORTS" : "REFUTES"}`; + ui.evidenceStrip.append(chip); + }); +} + +function renderAction(action) { + const hasAction = Boolean(action && action.id); + ui.actionEmpty.hidden = hasAction; + ui.actionContent.hidden = !hasAction; + ui.parameterList.replaceChildren(); + + if (!hasAction) { + setText(ui.actionName, null); + setText(ui.riskBadge, null); + ui.riskBadge.className = "risk-badge"; + return; + } + + setText(ui.actionName, action.kind); + setText(ui.riskBadge, String(action.risk || "unknown").toUpperCase()); + ui.riskBadge.className = `risk-badge ${String(action.risk || "").toLowerCase()}`; + + normalizedParameters(action.parameters).slice(0, 4).forEach((parameter) => { + const row = document.createElement("div"); + row.className = "parameter-row"; + const term = document.createElement("dt"); + const description = document.createElement("dd"); + setText(term, parameter.name); + setText(description, parameter.value); + description.title = String(parameter.value); + row.append(term, description); + ui.parameterList.append(row); + }); +} + +function renderTimeline(timeline) { + ui.timelineList.replaceChildren(); + if (timeline.length === 0) { + const empty = document.createElement("li"); + empty.className = "timeline-empty"; + empty.textContent = "Os eventos aparecerão durante a execução."; + ui.timelineList.append(empty); + return; + } + + timeline.slice(-7).forEach((item) => { + const entry = document.createElement("li"); + entry.className = "timeline-item"; + const eventBox = document.createElement("div"); + const event = document.createElement("span"); + const phase = document.createElement("span"); + const time = document.createElement("time"); + event.className = "timeline-event"; + phase.className = "timeline-phase"; + time.className = "timeline-time"; + setText(event, item.event); + setText(phase, formatPhase(item.phase)); + setText(time, formatTime(item.occurred_at)); + if (item.occurred_at) { + time.dateTime = String(item.occurred_at); + } + eventBox.append(event, phase); + entry.append(eventBox, time); + ui.timelineList.append(entry); + }); +} + +function renderState(state) { + if (!state || typeof state !== "object") { + return; + } + latestState = state; + + const phase = String(state.phase || "idle"); + const incident = state.incident && typeof state.incident === "object" ? state.incident : {}; + const hypotheses = Array.isArray(state.hypotheses) ? state.hypotheses : []; + const evidence = Array.isArray(state.evidence) ? state.evidence : []; + const rootCause = state.root_cause && typeof state.root_cause === "object" ? state.root_cause : null; + const verification = state.verification_result && typeof state.verification_result === "object" + ? state.verification_result + : null; + const candidates = Array.isArray(state.candidate_actions) ? state.candidate_actions : []; + const action = state.selected_action || candidates[0] || null; + const timeline = Array.isArray(state.timeline) ? state.timeline : []; + + setText(ui.headerIncident, state.incident_id || ui.incidentId.value); + setText(ui.phaseBadge, formatPhase(phase)); + ui.phaseBadge.dataset.phase = ["resolved", "waiting_approval", "failed", "escalated"].includes(phase) + ? phase + : "running"; + setText(ui.severityValue, String(state.severity || "pending").toUpperCase()); + ui.severityValue.classList.toggle("critical", state.severity === "critical"); + setText(ui.attemptValue, state.attempts ?? 0); + + const displayError = verification?.error_rate_after ?? incident.error_rate_after; + setText(ui.currentError, formatPercent(displayError)); + if (verification?.recovered) { + setText(ui.errorDelta, "RECOVERED"); + ui.errorDelta.className = "metric-delta recovered"; + ui.sparkline.classList.add("recovered"); + } else if (incident.error_rate_after !== undefined && incident.error_rate_before !== undefined) { + const delta = Number(incident.error_rate_after) - Number(incident.error_rate_before); + setText(ui.errorDelta, `+${delta.toFixed(1)} PTS`); + ui.errorDelta.className = "metric-delta"; + ui.sparkline.classList.remove("recovered"); + } + + setText(ui.hypothesisCount, hypotheses.length); + setText(ui.branchCount, hypotheses.length); + setText(ui.evidenceCount, `${evidence.length} EVIDENCES`); + createBranchIndicators(hypotheses, evidence); + + if (rootCause) { + const confidence = Number(rootCause.confidence || 0); + setText(ui.confidenceValue, `${Math.round(confidence * 100)}%`); + setText(ui.rootHypothesis, rootCause.hypothesis_id || "SELECTED"); + ui.confidenceFill.style.width = `${Math.max(0, Math.min(100, confidence * 100))}%`; + setText(ui.rootSummary, rootCause.summary); + ui.rootSummary.classList.remove("muted"); + setText(ui.reasoningStatus, "SYNTHESIZED"); + ui.reasoningStatus.classList.add("ready"); + } + renderEvidence(evidence); + renderAction(action); + renderTimeline(timeline); + renderGraph(phase); + + const waiting = phase === "waiting_approval"; + ui.approvalPanel.hidden = !waiting; + if (waiting && action) { + setText( + ui.approvalDescription, + `${action.kind} · risk ${action.risk || "unknown"} · revision ${action.revision || 1}`, + ); + } + + const resolved = phase === "resolved"; + ui.outcomePanel.hidden = !resolved; + if (resolved && verification) { + setText(ui.outcomeBefore, formatPercent(verification.error_rate_before)); + setText(ui.outcomeAfter, formatPercent(verification.error_rate_after)); + const effect = state.execution_result?.synthetic_effect || "simulated action"; + setText(ui.outcomeDescription, `${effect} · recovery confirmed · idempotent execution`); + } + + if (phase === "failed") { + const errors = Array.isArray(state.errors) ? state.errors : []; + const latestError = errors.at(-1); + setStatus(latestError?.message || "O workflow terminou em estado de falha.", "error"); + } else if (waiting) { + setStatus("Checkpoint persistido. Aguardando decisão humana."); + } else if (resolved) { + setStatus("Execução concluída e recuperação verificada."); + } else if (busy) { + setStatus(`Processando fase ${formatPhase(phase)}…`, "running"); + } +} + +async function responseError(response) { + const body = await response.text(); + try { + const parsed = JSON.parse(body); + return parsed.detail || parsed.message || `HTTP ${response.status}`; + } catch { + return body || `HTTP ${response.status}`; + } +} + +async function fetchJson(url, options = {}) { + const response = await fetch(url, options); + if (!response.ok) { + throw new Error(await responseError(response)); + } + return response.json(); +} + +async function refreshHistory(incidentId) { + try { + const history = await fetchJson(`/incidents/${encodeURIComponent(incidentId)}/history`); + setText(ui.checkpointValue, Array.isArray(history) ? history.length : 0); + } catch { + setText(ui.checkpointValue, "—"); + } +} + +function parseSseBlock(block) { + const data = block + .split("\n") + .filter((line) => line.startsWith("data:")) + .map((line) => line.slice(5).trimStart()) + .join("\n"); + if (!data) { + return; + } + renderState(JSON.parse(data)); +} + +async function consumeStateStream(response) { + if (!response.body) { + throw new Error("Este navegador não disponibilizou o stream da resposta."); + } + const reader = response.body.getReader(); + const decoder = new TextDecoder(); + let buffer = ""; + + while (true) { + const { value, done } = await reader.read(); + buffer += decoder.decode(value || new Uint8Array(), { stream: !done }); + let boundary = buffer.indexOf("\n\n"); + while (boundary >= 0) { + parseSseBlock(buffer.slice(0, boundary)); + buffer = buffer.slice(boundary + 2); + boundary = buffer.indexOf("\n\n"); + } + if (done) { + break; + } + } + if (buffer.trim()) { + parseSseBlock(buffer); + } +} + +function incidentPayload() { + return { + incident_id: ui.incidentId.value.trim(), + service: ui.service.value.trim(), + error_rate_before: Number(ui.errorBefore.value), + error_rate_after: Number(ui.errorAfter.value), + deployment: ui.deployment.value.trim(), + started_at: new Date().toISOString(), + }; +} + +async function startIncident(event) { + event.preventDefault(); + if (!ui.form.reportValidity()) { + return; + } + const payload = incidentPayload(); + setBusy(true, "Abrindo stream do grafo…"); + ui.outcomePanel.hidden = true; + ui.approvalPanel.hidden = true; + setText(ui.headerIncident, payload.incident_id); + + try { + const response = await fetch(`/incidents/${encodeURIComponent(payload.incident_id)}/events`, { + method: "POST", + headers: { "Content-Type": "application/json", Accept: "text/event-stream" }, + body: JSON.stringify(payload), + }); + if (!response.ok) { + throw new Error(await responseError(response)); + } + await consumeStateStream(response); + await refreshHistory(payload.incident_id); + } catch (error) { + setStatus(error instanceof Error ? error.message : "Falha ao iniciar incidente.", "error"); + } finally { + setBusy(false); + } +} + +async function loadIncident() { + const incidentId = ui.incidentId.value.trim(); + if (!incidentId) { + ui.incidentId.focus(); + return; + } + setBusy(true, "Carregando checkpoint persistido…"); + try { + const state = await fetchJson(`/incidents/${encodeURIComponent(incidentId)}`); + renderState(state); + await refreshHistory(incidentId); + } catch (error) { + setStatus(error instanceof Error ? error.message : "Thread não encontrada.", "error"); + } finally { + setBusy(false); + } +} + +async function submitDecision(decision) { + const incidentId = String(latestState?.incident_id || ui.incidentId.value).trim(); + const candidates = Array.isArray(latestState?.candidate_actions) ? latestState.candidate_actions : []; + const action = latestState?.selected_action || candidates[0]; + if (!incidentId || !action?.id) { + setStatus("A execução não possui uma ação selecionada.", "error"); + return; + } + + setBusy(true, decision === "approved" ? "Retomando execução aprovada…" : "Replanejando…"); + try { + const result = await fetchJson(`/incidents/${encodeURIComponent(incidentId)}/approval`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + decision, + action_id: action.id, + comment: "Decision submitted from StateOps Control Room", + }), + }); + renderState(result.state); + await refreshHistory(incidentId); + } catch (error) { + setStatus(error instanceof Error ? error.message : "Falha ao enviar decisão.", "error"); + } finally { + setBusy(false); + } +} + +function resetConsole() { + latestState = null; + ui.form.reset(); + ui.incidentId.value = makeIncidentId(); + ui.service.value = "checkout-service"; + ui.errorBefore.value = "0.2"; + ui.errorAfter.value = "18.0"; + ui.deployment.value = "v2.31"; + setText(ui.headerIncident, "NO ACTIVE INCIDENT"); + setText(ui.phaseBadge, "IDLE"); + ui.phaseBadge.dataset.phase = "idle"; + setText(ui.severityValue, null); + setText(ui.checkpointValue, 0); + setText(ui.attemptValue, 0); + setText(ui.currentError, null); + setText(ui.errorDelta, "NO SIGNAL"); + ui.errorDelta.className = "metric-delta"; + setText(ui.hypothesisCount, 0); + setText(ui.evidenceCount, "0 EVIDENCES"); + setText(ui.branchCount, 0); + setText(ui.confidenceValue, null); + setText(ui.rootHypothesis, "PENDING"); + ui.confidenceFill.style.width = "0"; + setText(ui.rootSummary, "Inicie um incidente para visualizar a síntese das evidências produzidas pelos ramos."); + ui.rootSummary.classList.add("muted"); + setText(ui.reasoningStatus, "PENDING"); + ui.reasoningStatus.classList.remove("ready"); + renderEvidence([]); + createBranchIndicators([], []); + renderAction(null); + renderTimeline([]); + renderGraph("idle"); + ui.approvalPanel.hidden = true; + ui.outcomePanel.hidden = true; + ui.sparkline.classList.remove("recovered"); + setStatus("Pronto para iniciar uma nova execução."); +} + +function setCaptureMode(enabled) { + document.body.classList.toggle("capture-mode", enabled); + ui.captureButton.setAttribute("aria-pressed", String(enabled)); + ui.captureButton.title = enabled ? "Sair do modo captura" : "Expandir para captura"; +} + +function toggleCaptureMode() { + setCaptureMode(!document.body.classList.contains("capture-mode")); +} + +async function checkHealth() { + try { + const result = await fetchJson("/healthz"); + const online = result.status === "ok"; + ui.healthPill.dataset.status = online ? "online" : "offline"; + setText(ui.healthLabel, online ? "API online" : "API degradada"); + } catch { + ui.healthPill.dataset.status = "offline"; + setText(ui.healthLabel, "API offline"); + } +} + +ui.form.addEventListener("submit", startIncident); +ui.loadButton.addEventListener("click", loadIncident); +ui.approveButton.addEventListener("click", () => submitDecision("approved")); +ui.rejectButton.addEventListener("click", () => submitDecision("rejected")); +ui.resetButton.addEventListener("click", resetConsole); +ui.captureButton.addEventListener("click", toggleCaptureMode); + +async function initialize() { + resetConsole(); + const parameters = new URLSearchParams(window.location.search); + const incidentId = parameters.get("incident"); + if (parameters.get("capture") === "1") { + setCaptureMode(true); + } + await checkHealth(); + if (incidentId && /^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$/.test(incidentId)) { + ui.incidentId.value = incidentId; + await loadIncident(); + } +} + +initialize(); diff --git a/src/stateops/entrypoints/api/static/index.html b/src/stateops/entrypoints/api/static/index.html new file mode 100644 index 0000000..0fa77c7 --- /dev/null +++ b/src/stateops/entrypoints/api/static/index.html @@ -0,0 +1,300 @@ + + + + + + + + StateOps Control Room + + + + + + + +
+ + + + STATEOPS + GRAPH CONTROL ROOM + + + +
+ THREAD + NO ACTIVE INCIDENT +
+ +
+
+ + Verificando API +
+ +
+
+ +
+ + +
+
+
+ 02 / STATE MACHINE +
+

Incident execution graph

+ IDLE +
+
+
+
SEVERITY—
+
CHECKPOINTS0
+
ATTEMPT0
+
+
+ +
+
+
+ 01 + ↗ + INTAKE + enrich + classify +
+ +
+ 02 + ◇ + HYPOTHESES + structured output +
+ +
+ 03 + ⌘ + FAN-OUT + Send × 0 +
+
+ +
+ 04 + ≡ + REDUCE + evidence synthesis +
+ +
+ 05 + △ + PLAN + select action +
+ +
+ 06 + Ⅱ + HUMAN GATE + interrupt +
+ +
+ 07 + ▷ + EXECUTE + idempotent effect +
+ +
+ 08 + ✓ + VERIFY + Command route +
+
+
+ + + +
+
+
ERROR RATELIVE
+
+ — + NO SIGNAL +
+ +
+ +
+
PARALLEL BRANCHESSEND
+
+ 0 + 0 EVIDENCES +
+ +
+ +
+
ROOT CAUSE CONFIDENCELLM
+
+ — + PENDING +
+
+
+
+ +
+
+
+
+ 03 / REASONING +

Root cause

+
+ PENDING +
+

+ Inicie um incidente para visualizar a síntese das evidências produzidas pelos ramos. +

+
+
+ +
+
+
+ 04 / DECISION +

Selected action

+
+ — +
+
+ △ +

Nenhuma ação selecionada

+
+ +
+ +
+
+
+ 05 / AUDIT TRAIL +

State timeline

+
+ LIVE +
+
    +
  1. Os eventos aparecerão durante a execução.
    2. +
+
+
+ + + +

+
+
+ + diff --git a/src/stateops/graphs/failures.py b/src/stateops/graphs/failures.py new file mode 100644 index 0000000..69ff68f --- /dev/null +++ b/src/stateops/graphs/failures.py @@ -0,0 +1,74 @@ +"""Translate reasoning failures into durable metadata-only graph state.""" + +from stateops.application.ports.clock import Clock +from stateops.application.ports.reasoner import IncidentReasoningError +from stateops.domain.enums import IncidentPhase +from stateops.domain.models import WorkflowError +from stateops.domain.transitions import transition +from stateops.graphs.events import timeline_event +from stateops.graphs.state import IncidentState + + +def reasoning_error( + *, + incident_id: str, + node: str, + error: IncidentReasoningError, + discriminator: str = "0", +) -> WorkflowError: + """Build a stable serializable error without persisting an exception object.""" + return WorkflowError( + id=f"{incident_id}:reasoning.failed:{node}:{discriminator}:{error.code}", + category=error.code, + node=node, + retryable=error.retryable, + ) + + +def failed_reasoning_state( + clock: Clock, + state: IncidentState, + *, + node: str, + error: IncidentReasoningError, + discriminator: str = "0", +) -> IncidentState: + """Terminate a reasoning step as explicit failed state.""" + phase = transition(state["phase"], IncidentPhase.FAILED) + return IncidentState( + phase=phase, + errors=[ + reasoning_error( + incident_id=state["incident_id"], + node=node, + error=error, + discriminator=discriminator, + ) + ], + timeline=[ + timeline_event( + clock, + incident_id=state["incident_id"], + phase=phase, + event="reasoning.failed", + discriminator=f"{node}:{discriminator}:{error.code}", + ) + ], + ) + + +def failed_state_from_branch_errors(clock: Clock, state: IncidentState) -> IncidentState: + """Terminate after parallel investigation when any branch recorded an error.""" + phase = transition(state["phase"], IncidentPhase.FAILED) + return IncidentState( + phase=phase, + timeline=[ + timeline_event( + clock, + incident_id=state["incident_id"], + phase=phase, + event="reasoning.failed", + discriminator="investigation-branches", + ) + ], + ) diff --git a/src/stateops/graphs/incident_graph.py b/src/stateops/graphs/incident_graph.py index c500c1e..0a19d0e 100644 --- a/src/stateops/graphs/incident_graph.py +++ b/src/stateops/graphs/incident_graph.py @@ -74,6 +74,20 @@ def _after_verification(state: IncidentState) -> Literal["remediation"] | str: return END +def _after_investigation(state: IncidentState) -> Literal["remediation"] | str: + """Stop the parent graph when the investigation failed.""" + if IncidentPhase(state["phase"]) is IncidentPhase.FAILED: + return END + return "remediation" + + +def _after_remediation(state: IncidentState) -> Literal["verification"] | str: + """Stop the parent graph when remediation planning failed.""" + if IncidentPhase(state["phase"]) is IncidentPhase.FAILED: + return END + return "verification" + + def build_incident_graph( *, reasoner: IncidentReasoner, @@ -97,8 +111,16 @@ def build_incident_graph( builder.add_edge(START, "enrich") builder.add_edge("enrich", "classify") builder.add_edge("classify", "investigation") - builder.add_edge("investigation", "remediation") - builder.add_edge("remediation", "verification") + builder.add_conditional_edges( + "investigation", + _after_investigation, + {"remediation": "remediation", END: END}, + ) + builder.add_conditional_edges( + "remediation", + _after_remediation, + {"verification": "verification", END: END}, + ) builder.add_conditional_edges( "verification", _after_verification, diff --git a/src/stateops/graphs/investigation/graph.py b/src/stateops/graphs/investigation/graph.py index 61d476f..2658a82 100644 --- a/src/stateops/graphs/investigation/graph.py +++ b/src/stateops/graphs/investigation/graph.py @@ -6,11 +6,16 @@ from langgraph.types import Send from stateops.application.ports.clock import Clock -from stateops.application.ports.reasoner import IncidentReasoner +from stateops.application.ports.reasoner import IncidentReasoner, IncidentReasoningError from stateops.domain.enums import IncidentPhase from stateops.domain.models import Hypothesis from stateops.domain.transitions import transition from stateops.graphs.events import timeline_event +from stateops.graphs.failures import ( + failed_reasoning_state, + failed_state_from_branch_errors, + reasoning_error, +) from stateops.graphs.state import IncidentState @@ -33,8 +38,18 @@ def __init__(self, reasoner: IncidentReasoner, clock: Clock) -> None: async def generate_hypotheses(self, state: IncidentState) -> IncidentState: """Generate hypotheses, then persist them before fan-out.""" + try: + hypotheses = await self._reasoner.generate_hypotheses( + state["incident"], state["context"] + ) + except IncidentReasoningError as exc: + return failed_reasoning_state( + self._clock, + state, + node="generate_hypotheses", + error=exc, + ) phase = transition(state["phase"], IncidentPhase.HYPOTHESES_GENERATED) - hypotheses = await self._reasoner.generate_hypotheses(state["incident"], state["context"]) return IncidentState( phase=phase, hypotheses=list(hypotheses), @@ -87,7 +102,28 @@ async def investigate_hypothesis(self, state: InvestigationTaskState) -> Inciden incident = cast(IncidentRecord, state["incident"]) context = cast(IncidentContext, state["context"]) hypothesis = state["hypothesis"] - evidence = await self._reasoner.investigate(incident, context, hypothesis) + try: + evidence = await self._reasoner.investigate(incident, context, hypothesis) + except IncidentReasoningError as exc: + return IncidentState( + errors=[ + reasoning_error( + incident_id=state["incident_id"], + node="investigate_hypothesis", + error=exc, + discriminator=hypothesis.id, + ) + ], + timeline=[ + timeline_event( + self._clock, + incident_id=state["incident_id"], + phase=IncidentPhase.INVESTIGATING, + event="reasoning.failed", + discriminator=f"investigate_hypothesis:{hypothesis.id}:{exc.code}", + ) + ], + ) return IncidentState( evidence=[evidence], timeline=[ @@ -103,9 +139,19 @@ async def investigate_hypothesis(self, state: InvestigationTaskState) -> Inciden async def synthesize(self, state: IncidentState) -> IncidentState: """Reduce accumulated evidence into one probable root cause.""" - root_cause = await self._reasoner.synthesize( - tuple(state["hypotheses"]), tuple(state["evidence"]) - ) + if state["errors"]: + return failed_state_from_branch_errors(self._clock, state) + try: + root_cause = await self._reasoner.synthesize( + tuple(state["hypotheses"]), tuple(state["evidence"]) + ) + except IncidentReasoningError as exc: + return failed_reasoning_state( + self._clock, + state, + node="synthesize_evidence", + error=exc, + ) phase = transition(state["phase"], IncidentPhase.EVIDENCE_READY) return IncidentState( root_cause=root_cause, @@ -130,8 +176,19 @@ def build_investigation_graph(reasoner: IncidentReasoner, clock: Clock) -> Any: builder.add_node("investigate_hypothesis", nodes.investigate_hypothesis) builder.add_node("synthesize_evidence", nodes.synthesize) builder.add_edge(START, "generate_hypotheses") - builder.add_edge("generate_hypotheses", "mark_investigating") + builder.add_conditional_edges( + "generate_hypotheses", + _after_hypotheses, + {"mark_investigating": "mark_investigating", END: END}, + ) builder.add_conditional_edges("mark_investigating", nodes.fan_out) builder.add_edge("investigate_hypothesis", "synthesize_evidence") builder.add_edge("synthesize_evidence", END) return builder.compile() + + +def _after_hypotheses(state: IncidentState) -> str: + """Stop the subgraph when hypothesis generation failed.""" + if IncidentPhase(state["phase"]) is IncidentPhase.FAILED: + return END + return "mark_investigating" diff --git a/src/stateops/graphs/remediation/graph.py b/src/stateops/graphs/remediation/graph.py index a67f3d1..23625d9 100644 --- a/src/stateops/graphs/remediation/graph.py +++ b/src/stateops/graphs/remediation/graph.py @@ -8,11 +8,12 @@ from stateops.application.approvals import apply_approval, parse_approval from stateops.application.idempotency import remediation_idempotency_key from stateops.application.ports.clock import Clock -from stateops.application.ports.reasoner import IncidentReasoner +from stateops.application.ports.reasoner import IncidentReasoner, IncidentReasoningError from stateops.application.ports.remediation import RemediationExecutor from stateops.domain.enums import ActionKind, ApprovalOutcome, IncidentPhase, Severity from stateops.domain.transitions import transition from stateops.graphs.events import timeline_event +from stateops.graphs.failures import failed_reasoning_state from stateops.graphs.state import IncidentState @@ -29,8 +30,19 @@ def __init__( async def propose_actions(self, state: IncidentState) -> IncidentState: """Generate allowlisted candidates after evidence synthesis or replan.""" + try: + actions = await self._reasoner.propose_remediations( + state["incident"], state["root_cause"] + ) + except IncidentReasoningError as exc: + return failed_reasoning_state( + self._clock, + state, + node="propose_actions", + error=exc, + discriminator=str(state["attempts"]), + ) phase = transition(state["phase"], IncidentPhase.PLANNING) - actions = await self._reasoner.propose_remediations(state["incident"], state["root_cause"]) return IncidentState( phase=phase, candidate_actions=list(actions), @@ -178,7 +190,18 @@ def build_remediation_graph( builder.add_node("execute_action", nodes.execute_action) builder.add_node("replan", nodes.replan) builder.add_edge(START, "propose_actions") - builder.add_edge("propose_actions", "select_action") + builder.add_conditional_edges( + "propose_actions", + _after_proposal, + {"select_action": "select_action", END: END}, + ) builder.add_edge("select_action", "human_approval") builder.add_edge("execute_action", END) return builder.compile(checkpointer=checkpointer) + + +def _after_proposal(state: IncidentState) -> str: + """Stop the subgraph when action generation failed.""" + if IncidentPhase(state["phase"]) is IncidentPhase.FAILED: + return END + return "select_action" diff --git a/src/stateops/graphs/runtime.py b/src/stateops/graphs/runtime.py index 2396b77..ca71945 100644 --- a/src/stateops/graphs/runtime.py +++ b/src/stateops/graphs/runtime.py @@ -146,3 +146,4 @@ async def stream_values( async for value in stream.values: yield _as_mapping(value) await stream.output() + yield await self._latest(_config(incident_id)) diff --git a/tests/unit/test_api.py b/tests/unit/test_api.py index b4ad462..82bece2 100644 --- a/tests/unit/test_api.py +++ b/tests/unit/test_api.py @@ -117,6 +117,27 @@ def test_api_drives_incident_approval_history_and_time_travel() -> None: assert invalid.status_code == 422 +def test_operator_console_is_local_packaged_and_csp_protected() -> None: + with TestClient(create_app(runtime=FakeRuntime())) as client: + page = client.get("/ui") + assert page.status_code == 200 + assert page.headers["content-type"].startswith("text/html") + assert page.headers["content-security-policy"].startswith("default-src 'none'") + assert page.headers["x-content-type-options"] == "nosniff" + assert "StateOps Control Room" in page.text + assert '' in page.text + + stylesheet = client.get("/ui/assets/app.css") + assert stylesheet.status_code == 200 + assert stylesheet.headers["content-type"].startswith("text/css") + + script = client.get("/ui/assets/app.js") + assert script.status_code == 200 + assert "innerHTML" not in script.text + assert "incidents/${encodeURIComponent(incidentId)}/approval" in script.text + assert "URLSearchParams" in script.text + + def test_api_validates_incidents_and_streams_v3_value_projection_as_sse() -> None: with TestClient(create_app(runtime=FakeRuntime())) as client: invalid = _payload() diff --git a/tests/unit/test_governed_gateway.py b/tests/unit/test_governed_gateway.py index 5d56a22..0744579 100644 --- a/tests/unit/test_governed_gateway.py +++ b/tests/unit/test_governed_gateway.py @@ -8,13 +8,15 @@ import pytest from governed_llm_gateway_client import GatewayClient +from governed_llm_gateway_client.errors import GatewayTransportError +from governed_llm_gateway_contracts import ExecutionStatus, GatewayError, StructuredOutputSchema from stateops.adapters.llm.governed_gateway import ( GatewayReasoningError, GovernedGatewayIncidentReasoner, ) from stateops.domain.enums import ActionKind -from stateops.domain.models import IncidentContext, IncidentRecord, Signal +from stateops.domain.models import IncidentContext, IncidentRecord, RootCause, Signal class FakeGatewayClient: @@ -27,7 +29,11 @@ def __init__(self, responses: list[str | None]) -> None: async def generate(self, **kwargs: Any) -> object: self.calls.append(kwargs) - return SimpleNamespace(content=self.responses.pop(0)) + return SimpleNamespace( + status=ExecutionStatus.SUCCEEDED, + content=self.responses.pop(0), + error=None, + ) async def aclose(self) -> None: self.closed = True @@ -64,7 +70,7 @@ async def scenario() -> None: { "id": "rollback-1", "kind": "rollback_deployment", - "parameters": {"target_version": "v2.30"}, + "parameters": [{"name": "target_version", "value": "v2.30"}], "risk": "high", } ] @@ -84,6 +90,131 @@ async def scenario() -> None: assert fake.closed assert all(call["workload"] == "stateops.incident.reasoning" for call in fake.calls) assert all("provider" not in call and "model" not in call for call in fake.calls) + hypotheses_schema = cast(StructuredOutputSchema, fake.calls[0]["structured_output"]) + actions_schema = cast(StructuredOutputSchema, fake.calls[3]["structured_output"]) + hypotheses_properties = cast(dict[str, object], hypotheses_schema.schema["properties"]) + hypotheses_array = cast(dict[str, object], hypotheses_properties["hypotheses"]) + assert "maxItems" not in hypotheses_array + actions_properties = cast(dict[str, object], actions_schema.schema["properties"]) + actions_array = cast(dict[str, object], actions_properties["actions"]) + assert "maxItems" not in actions_array + action_items = actions_array["items"] + action_properties = cast(dict[str, object], action_items)["properties"] + parameters = cast(dict[str, object], action_properties)["parameters"] + parameter_items = cast(dict[str, object], parameters)["items"] + assert cast(dict[str, object], parameter_items)["additionalProperties"] is False + + asyncio.run(scenario()) + + +def test_gateway_reasoner_preserves_failed_terminal_metadata() -> None: + class FailedGatewayClient(FakeGatewayClient): + async def generate(self, **kwargs: Any) -> object: + self.calls.append(kwargs) + return SimpleNamespace( + status=ExecutionStatus.FAILED, + content=None, + error=GatewayError( + code="invalid_request", + message="provider rejected the schema", + retryable=False, + ), + ) + + async def scenario() -> None: + reasoner = GovernedGatewayIncidentReasoner(cast(GatewayClient, FailedGatewayClient([]))) + with pytest.raises(GatewayReasoningError) as raised: + await reasoner.generate_hypotheses(_incident(), _context()) + + assert raised.value.code == "invalid_request" + assert not raised.value.retryable + + asyncio.run(scenario()) + + +def test_gateway_reasoner_marks_transport_failures_retryable() -> None: + class TransportFailureGatewayClient(FakeGatewayClient): + async def generate(self, **kwargs: Any) -> object: + self.calls.append(kwargs) + raise GatewayTransportError("gateway unavailable") + + async def scenario() -> None: + reasoner = GovernedGatewayIncidentReasoner( + cast(GatewayClient, TransportFailureGatewayClient([])) + ) + with pytest.raises(GatewayReasoningError) as raised: + await reasoner.generate_hypotheses(_incident(), _context()) + + assert raised.value.code == "gateway_transport_error" + assert raised.value.retryable + + asyncio.run(scenario()) + + +def test_gateway_reasoner_enforces_collection_limits_after_decoding() -> None: + async def scenario() -> None: + too_many = { + "hypotheses": [ + { + "id": f"hyp-{index}", + "title": "candidate", + "rationale": "signal", + "confidence": 0.5, + } + for index in range(9) + ] + } + reasoner = GovernedGatewayIncidentReasoner( + cast(GatewayClient, FakeGatewayClient([json.dumps(too_many)])) + ) + with pytest.raises(GatewayReasoningError, match="between 1 and 8"): + await reasoner.generate_hypotheses(_incident(), _context()) + + asyncio.run(scenario()) + + +def test_gateway_reasoner_enforces_action_limits_after_decoding() -> None: + async def scenario() -> None: + too_many = { + "actions": [ + { + "id": f"action-{index}", + "kind": "restart_service", + "parameters": [], + "risk": "medium", + } + for index in range(6) + ] + } + reasoner = GovernedGatewayIncidentReasoner( + cast(GatewayClient, FakeGatewayClient([json.dumps(too_many)])) + ) + with pytest.raises(GatewayReasoningError, match="between 1 and 5"): + await reasoner.propose_remediations(_incident(), RootCause("hyp-1", "regression", 0.9)) + + asyncio.run(scenario()) + + +def test_gateway_reasoner_rejects_duplicate_action_parameter_names() -> None: + async def scenario() -> None: + response = { + "actions": [ + { + "id": "rollback-1", + "kind": "rollback_deployment", + "parameters": [ + {"name": "target_version", "value": "v2.30"}, + {"name": "target_version", "value": "v2.29"}, + ], + "risk": "high", + } + ] + } + reasoner = GovernedGatewayIncidentReasoner( + cast(GatewayClient, FakeGatewayClient([json.dumps(response)])) + ) + with pytest.raises(GatewayReasoningError, match="duplicate parameter names"): + await reasoner.propose_remediations(_incident(), RootCause("hyp-1", "regression", 0.9)) asyncio.run(scenario()) diff --git a/tests/unit/test_graph.py b/tests/unit/test_graph.py index 2bf3be5..4ea0706 100644 --- a/tests/unit/test_graph.py +++ b/tests/unit/test_graph.py @@ -4,15 +4,26 @@ from collections.abc import Mapping from datetime import UTC, datetime, timedelta +import pytest from langgraph.types import Command from stateops.adapters.incidents.synthetic_signals import SyntheticIncidentSignals from stateops.adapters.llm.deterministic import DeterministicIncidentReasoner from stateops.adapters.persistence.checkpointer import memory_checkpointer from stateops.adapters.remediation.simulated_executor import SimulatedRemediationExecutor +from stateops.application.ports.reasoner import IncidentReasoningError from stateops.application.runtime_models import RunResult from stateops.domain.enums import IncidentPhase -from stateops.domain.models import ExecutionResult, IncidentRecord, RootCause +from stateops.domain.models import ( + Evidence, + ExecutionResult, + Hypothesis, + IncidentContext, + IncidentRecord, + RemediationAction, + RootCause, + WorkflowError, +) from stateops.graphs.incident_graph import build_incident_graph from stateops.graphs.remediation.graph import build_remediation_graph from stateops.graphs.runtime import GraphIncidentRuntime @@ -31,6 +42,45 @@ def now(self) -> datetime: return value +class FailingReasoner(DeterministicIncidentReasoner): + """Raise a metadata-safe failure at one selected reasoning stage.""" + + def __init__(self, stage: str) -> None: + self._stage = stage + + def _raise_if_selected(self, stage: str) -> None: + if self._stage == stage: + raise IncidentReasoningError( + "reasoning failed", + code=f"{stage}_failed", + retryable=stage == "investigate", + ) + + async def generate_hypotheses( + self, incident: IncidentRecord, context: IncidentContext + ) -> tuple[Hypothesis, ...]: + self._raise_if_selected("generate") + return await super().generate_hypotheses(incident, context) + + async def investigate( + self, incident: IncidentRecord, context: IncidentContext, hypothesis: Hypothesis + ) -> Evidence: + self._raise_if_selected("investigate") + return await super().investigate(incident, context, hypothesis) + + async def synthesize( + self, hypotheses: tuple[Hypothesis, ...], evidence: tuple[Evidence, ...] + ) -> RootCause: + self._raise_if_selected("synthesize") + return await super().synthesize(hypotheses, evidence) + + async def propose_remediations( + self, incident: IncidentRecord, root_cause: RootCause + ) -> tuple[RemediationAction, ...]: + self._raise_if_selected("propose") + return await super().propose_remediations(incident, root_cause) + + def _incident() -> IncidentRecord: return IncidentRecord("checkout", 0.2, 18.0, "v2.31", datetime(2026, 1, 1, tzinfo=UTC)) @@ -89,6 +139,24 @@ async def scenario() -> None: asyncio.run(scenario()) +def test_value_stream_ends_with_the_persisted_interrupt_state() -> None: + async def scenario() -> None: + runtime, _ = _runtime() + + snapshots = [ + snapshot async for snapshot in runtime.stream_values("INC-STREAM", _incident()) + ] + + assert snapshots + assert snapshots[-1]["phase"] is IncidentPhase.WAITING_APPROVAL + evidence = snapshots[-1]["evidence"] + assert isinstance(evidence, list) + assert len(evidence) == 3 + assert snapshots[-1]["selected_action"] is not None + + asyncio.run(scenario()) + + def test_fork_selects_a_controlled_alternative_without_erasing_original_history() -> None: async def scenario() -> None: runtime, executor = _runtime() @@ -155,3 +223,30 @@ async def scenario() -> None: assert replayed["execution_result"].duplicate asyncio.run(scenario()) + + +@pytest.mark.parametrize("stage", ["generate", "investigate", "synthesize", "propose"]) +def test_reasoning_failures_become_terminal_serializable_state(stage: str) -> None: + async def scenario() -> None: + clock = FakeClock() + graph = build_incident_graph( + reasoner=FailingReasoner(stage), + signals=SyntheticIncidentSignals(), + executor=SimulatedRemediationExecutor(clock), + clock=clock, + checkpointer=memory_checkpointer(), + ) + runtime = GraphIncidentRuntime(graph) + + result = await runtime.start(f"INC-FAILED-{stage}", _incident()) + + assert not result.interrupted + assert result.values["phase"] is IncidentPhase.FAILED + errors = result.values["errors"] + assert isinstance(errors, list) + assert errors + assert all(isinstance(error, WorkflowError) for error in errors) + assert {error.category for error in errors} == {f"{stage}_failed"} + assert await runtime.history(f"INC-FAILED-{stage}") + + asyncio.run(scenario())