You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As an 运维操作者, I want one click to represent exactly one submission operation, so that a lost response cannot silently create a second paid Process Run.
As an 运维操作者, I want an ambiguous submission to reuse its original idempotency key, so that retrying after a timeout is safe.
As an 运维操作者, I want the console to distinguish “definitively rejected” from “acceptance unknown,” so that I do not guess whether a paid operation started.
As an 运维操作者, I want the console to show the runId immediately after durable acceptance, so that I can retain a stable recovery handle.
As an 运维操作者, I want the console to show queued, running, succeeded, and failed states, so that I know whether to wait or investigate.
As an 运维操作者, I want a successful terminal Process Run to display its typed result, so that I can verify the delivered artifact.
As an 运维操作者, I want a failed terminal Process Run to display its stable public error, so that I can act without seeing internal exception text.
As an 运维操作者, I want an expired result to be reported explicitly, so that I do not confuse retention expiry with an empty result.
As an 运维操作者, I want a 300-second wait timeout to preserve the accepted runId, so that I can continue querying later.
As an 运维操作者, I want leaving the page to stop browser polling without cancelling the Process Run, so that navigation does not change paid business execution.
As an 运维操作者, I want to resume an accepted Process Run after refreshing the page, so that a browser restart does not lose the recovery path.
As an 运维操作者, I want an explicit action to start a genuinely new submission after an uncertain one, so that duplicate intent is deliberate rather than accidental.
As an 运维操作者, I want transient query failures to recover within the wait deadline, so that a brief gateway or network interruption does not look like Process failure.
As an 运维操作者, I want capacity rejection to respect Retry-After, so that the console does not hot-loop or overload the service.
As an 运维操作者, I want malformed or unexpected server responses to produce a clear protocol error, so that the console never displays fabricated state.
As an 运维操作者, I want the console to link an accepted runId to a recoverable result view, so that I do not need to copy identifiers from free-form text.
As an 运维操作者, I want a clear warning that submission may generate image cost, so that I understand the side effect before starting.
As an 运维操作者, I want the submit button disabled only for the active submission operation, so that repeated clicks cannot create concurrent duplicate intent.
As an authenticated caller, I want all queries scoped to my stable caller identity, so that another caller cannot observe my Process Run.
As an authenticated caller, I want unknown and unauthorized runId values to remain indistinguishable, so that Process Runs cannot be enumerated.
As a security maintainer, I want the browser to send no gateway shared secret, so that trust remains in the gateway Adapter rather than client code.
As a security maintainer, I want the result Location constrained to the same origin and expected resource shape, so that a response cannot redirect polling to an arbitrary host.
As a security maintainer, I want the gateway to remove caller-supplied identity headers before injecting trusted identity, so that callers cannot impersonate one another.
As a service maintainer, I want idempotency, retry, polling and response validation owned by one Client Module, so that protocol fixes have locality.
As a service maintainer, I want the Client Module to expose structured outcomes rather than unknown bodies and error strings, so that every caller handles the same states.
As a service maintainer, I want time, scheduling, transport and pending-submission persistence behind internal seams, so that deterministic tests need no real clock or browser globals.
As a service maintainer, I want tests to use the Client Module’s external Interface, so that Implementation refactors do not rewrite test intent.
As a service maintainer, I want the existing server Async Process Runs Module to remain unchanged unless a failing cross-Seam test proves a server defect, so that mature state-machine behaviour is not duplicated.
As a service maintainer, I want one test to cover queued → running → succeeded, so that repeated polling is proven rather than inferred from a single terminal response.
As a service maintainer, I want terminal failure and expired-result tests, so that every public terminal projection is supported by the Client.
As a service maintainer, I want an ambiguous POST test where the server accepts but the response is lost, so that replay with the same key proves no duplicate Run is created.
As a service maintainer, I want transient GET and 503 recovery tests, so that the 300-second wait is useful during short outages.
As a service maintainer, I want cancellation and timeout tests to prove the server Run continues, so that client lifecycle and Process lifecycle stay separate.
As a service maintainer, I want a real HTTP integration test using the same Client as the browser, so that mocked fetch tests cannot drift from the deployed protocol.
As a service maintainer, I want a test gateway Adapter to inject trusted caller identity, so that integration tests exercise the production trust model without exposing secrets to browser code.
As a service maintainer, I want a minimal headless-browser test of the submit form, progress state and terminal result, so that DOM wiring is verified at the highest useful Seam.
As a service maintainer, I want local development to provide a safe test caller through a development-only gateway Adapter, so that the async console can be exercised without weakening production identity rules.
As a release engineer, I want PostgreSQL and BullMQ integration suites to run in CI, so that a pull request cannot merge after breaking durable acceptance or dispatch.
As a release engineer, I want integration containers and test data cleaned after every CI run, so that retries start from an isolated environment.
As a release engineer, I want the standard synchronous production shape to remain disabled for async requests, so that ordinary deployment does not bypass release gates.
As a release engineer, I want a separate explicit async deployment shape, so that API, Dispatcher, Worker, Webhook Worker and Retention Cleaner can be released and rolled back together without changing the safe default.
As a release engineer, I want every async role to pass deployment preflight and readiness before internal traffic starts, so that missing secrets or mismatched Queue configuration fail before exposure.
As an SRE, I want an internal smoke to submit and query both a successful and failed Process Run, so that status projection and public errors are verified after deployment.
As an SRE, I want a Worker restart during an accepted Run to preserve one terminal outcome, so that rolling deployment is safe.
As an SRE, I want Redis loss followed by Queue Recovery to complete accepted Runs, so that PostgreSQL remains the authority.
As an SRE, I want backlog, stuck Run, Outbox lag and recovery-age readiness gates enforced before canary traffic, so that promotion depends on operational evidence.
As an SRE, I want async observability and alert fields populated during smoke, so that incidents are diagnosable before production traffic.
As an SRE, I want rollback to stop new submissions while preserving owner queries for accepted Runs, so that disabling intake does not strand callers.
As a product owner, I want one controlled paid image smoke through the complete asynchronous path, so that FAL, finalization, OSS persistence and result query are verified together.
As a product owner, I want paid smoke separated from deterministic CI, so that normal validation never incurs hidden external cost.
Implementation Decisions
Preserve the existing Business Process, Process Registration, Process Registry and Async Process Runs domain model. Do not introduce a second Workflow model, client-selected runtime configuration or Queue concepts into the product request.
Keep the server Async Process Runs Module as the authority for acceptance and owner-scoped lookup. Its existing submit/find Interface, PostgreSQL Store, Outbox, Queue Adapter, Worker and recovery behaviour remain the server foundation.
Add one external Seam in the browser layer: the Console Process Run Client Interface. The UI submits a Business Process request to this Interface and receives structured progress/outcomes; it does not manage HTTP headers, polling timers or response parsing itself.
The Client Module owns one stable idempotency key per user submission operation. It persists the pending operation before the first POST and reuses the key for every ambiguous or retryable submission attempt. A fresh key requires a new explicit user operation.
Persist the minimum browser recovery state needed to survive refresh: request fingerprint, idempotency key, optional accepted runId, creation time and current recovery classification. Do not persist business output, gateway credentials or hidden implementation data.
Once a 202 response maps the operation to a runId, preserve that mapping until terminal completion or explicit operator dismissal. Losing the page must not lose the accepted Run’s recovery handle.
Return discriminated structured outcomes for terminal success, terminal failure, expired result, pending timeout/cancellation, definitive submission rejection and protocol/unavailable failure. Free-form text is presentation only and must not be the programmatic Interface.
Keep the default client wait deadline at 300 seconds. The deadline starts after durable acceptance; submission retry has its own bounded policy so an unavailable acceptance path cannot wait forever.
Respect valid server Retry-After values for submission capacity responses and nonterminal queries. Apply safe minimum and maximum bounds to prevent hot polling or an unbounded sleep. Use bounded backoff for transient transport failures when no valid server hint exists.
Treat ambiguous transport failure during POST as acceptance unknown, not rejection. Retry with the same key. Treat stable validation, not-found and idempotency-conflict responses as definitive. Treat admission 429/capacity 503 as retryable without rotating the key because they do not consume it.
Treat transient network failures and retryable gateway/server errors during GET as recoverable until the client deadline. Stable authorization/not-found responses and invalid response shapes end the current wait with a structured failure.
Validate submission and Process Run response shapes before exposing them to the UI. Accept only the documented public states and result-expiry projection.
Resolve Location relative to the current origin, then reject cross-origin or non-Process-Run targets before issuing a query.
Accept an AbortSignal that cancels client retries, timers and polling only. This signal must never call a server cancellation operation; Process Run cancellation remains outside the current public Interface.
Keep transport, clock/scheduler, idempotency-key generation and pending-operation storage as internal seams of the Client Module. Production uses browser Fetch/Crypto/Storage Adapters; tests use scripted/in-memory Adapters. These internal seams are not exposed through the UI Interface.
Keep trusted identity in the gateway Adapter. Browser code uses same-origin HTTP and never reads or sends the gateway shared secret. The production gateway strips untrusted identity headers, authenticates the operator and injects the stable caller subject plus shared credential.
Provide a development-only gateway Adapter or equivalent local proxy configuration that injects a fixed non-production caller and reads any shared secret only on the server side. It must be impossible to include this secret in the browser bundle.
Update the console submission view to show the accepted runId, current public status, terminal structured result and recovery actions. A pending timeout provides “continue querying”; an acceptance-unknown state provides “retry same operation”; only an explicit “new submission” rotates the key.
Keep the standard production Compose shape synchronous and ASYNC_PROCESS_RUNS_ENABLED=false. Add a separate explicit async deployment shape or overlay owned by the async Runbook; do not add PostgreSQL or Redis containers to the base production Compose.
The async production shape starts the API override plus Process Dispatcher, Process Worker, Webhook Worker and Retention Cleaner against externally managed PostgreSQL/Redis. Operations and Queue Recovery remain controlled jobs rather than public HTTP roles.
Add an operator-controlled async release entry that performs per-role environment preflight, migration, full Queue Recovery dry-run, role readiness, internal smoke and staged promotion. Normal synchronous deployment must not enable async routes implicitly.
Rollback disables new asynchronous submissions or returns the release stage to internal while retaining PostgreSQL and the ability to query already accepted Runs. It must not delete accepted Run, owner, idempotency, Outbox or Delivery state.
No new database schema is required for server Process Runs. Browser pending-operation persistence is local client state, not a new production database table.
Keep POST /process-runs and GET /process-runs/{runId} contracts unchanged. Product callers still submit only exact Process identity/version and business input.
Testing Decisions
A good test crosses the highest stable Interface and asserts externally observable outcomes: structured Client outcome, HTTP status/headers, durable Run identity, public Process Run state and absence of duplicate side effects. Tests do not assert private timer calls, internal maps, BullMQ implementation fields or SQL layout.
The primary test Seam is the Console Process Run Client Interface. Deterministic contract tests instantiate it with scripted transport, clock/scheduler and in-memory pending-operation Adapters.
Client contract tests cover: successful submit; queued → running → succeeded; public failure; expired result; malformed submission/query response; missing or cross-origin Location; invalid Retry-After; 300-second timeout; AbortSignal cancellation; transient GET recovery; definitive authorization/not-found failure; admission 429/503; and response bodies that are not JSON.
The critical idempotency test simulates this ordering: the server durably accepts one request, the client loses the first response, the Client retries with the same key, and the final observable result contains the original runId with exactly one Process Run and one paid Business Capability effect.
Add one cross-Seam integration suite where the real Console Process Run Client uses real HTTP through a test gateway Adapter into independently constructed API, Dispatcher and Worker roles backed by temporary PostgreSQL and Redis. It covers success, stable business failure, owner isolation, idempotent replay and client timeout followed by later successful query.
Reuse existing Process Run HTTP, PostgreSQL Store, BullMQ runtime, recovery, Webhook and retention tests as prior art. Extend them only when a new Client-level failure reveals an uncovered server invariant; do not duplicate their internal cases in browser tests.
Add one minimal headless-browser acceptance test that loads the built console, submits through the test gateway, observes accepted runId/progress, and renders the terminal result. Do not add broad snapshot or private frontend component tests.
Add a CI job that starts the repository’s isolated PostgreSQL/Redis integration environment, runs PostgreSQL and async BullMQ integration commands serially, and always cleans containers. Keep deterministic tests as the fast default job; make the real-dependency job required before merge.
The CI integration job uses only the dedicated _test PostgreSQL database and nonzero Redis database. It must refuse shared/production targets under existing safety checks.
Add deployment-shape validation that renders the explicit async production configuration, verifies every role uses the same immutable image revision and Queue settings, and confirms the base production shape still has async disabled.
Staging fault drills cover: API acceptance with Redis unavailable; Dispatcher restart after publish-before-ack; Worker termination during an active claim; Redis Queue loss and full recovery; duplicate Job; expired claim takeover; Webhook endpoint outage; and rollback with owner query preserved.
Internal/canary smoke verifies role health/readiness, one successful Run, one stable failed Run, caller isolation, result retention projection, Queue/Outbox observability and no content leakage in logs.
A paid image smoke runs only with explicit credentials and authorization after deterministic and staging infrastructure gates pass. It executes one approved image Business Process through asynchronous submission, real Worker, FAL/finalizer, OSS storage and owner-scoped result query, then records cost-safe evidence without logging source URLs or credentials.
Existing synchronous /execute, Run Record console, Business Process catalog, activity logging and production build tests remain required regression gates.
Out of Scope
Replacing PostgreSQL as the Process Run authority or BullMQ as the current Queue Adapter.
Rewriting the existing server Async Process Runs state machine, Store, Outbox, Worker, Webhook or Retention Modules without a failing Interface-level test.
Changing Business Process input/output contracts, production catalog membership, Runtime Skill bindings, model/provider selection or object-storage policy.
Exactly-once execution or delivery guarantees. The system retains at-least-once scheduling with explicit downstream idempotency.
Public Process Run cancellation, pause, priority, scheduling, streaming output or caller-selected retry policy.
Public Process Run list/search, Attempt details or Webhook Endpoint self-management.
Putting gateway shared secrets, database/Redis credentials or Queue configuration into browser requests.
Adding PostgreSQL or Redis containers to the standard production Compose.
Automatically promoting traffic to canary or production without the Runbook gates and an authorized operator.
Running paid FAL/OSS smoke in ordinary pull-request CI.
Changing the 300-second client wait default or the existing server Process execution timeout as part of this specification.
Further Notes
This specification follows the delivered console work tracked in 运维控制台可溯源化:活动 trace 持久化、PostgreSQL 记录存储、服务自描述与 Preact 重写 #1. That earlier scope explicitly left asynchronous Process Run enablement out of scope; this Issue closes that next layer rather than reopening the Run Record design.
“Complete integration” currently means real local PostgreSQL/Redis plus independently constructed API, Dispatcher and Worker roles with a controlled Business Capability. It does not mean a real browser, production gateway, production deployment or paid image provider. Completion of this Issue must keep those claims distinct.
The server Async Process Runs Module is already deep: its small submit/find Interface hides Registration acceptance, owner isolation, idempotency, persistence and projection. The design target is to create comparable Depth and Locality on the browser side, not to flatten server Modules into the console.
The standard synchronous deployment remains the safe default. The explicit async deployment shape exists to make release intent visible, reviewable and reversible.
Any production rollout must follow internal → canary → production, preserve query access during rollback and record operational evidence required by the existing async Runbook.
Problem Statement
运维控制台已经能调用异步 Process Run HTTP Interface,并在最多 300 秒内查询结果;服务端 Async Process Runs Module 也已经实现 PostgreSQL durable acceptance、Outbox、BullMQ 调度、独立 Worker、owner 隔离、受控重试、恢复、Webhook 与保留策略。
但当前能力还不是可安全上线的产品闭环:
Idempotency-Key。如果服务端已经接受请求,但202响应在网络中丢失,操作者再次提交可能创建第二个付费 Process Run。这违反了“调用方超时或断线后必须用同一个 key 重试”的既有 Interface invariant。Retry-After、超时和 UI 回调混在一个浅层 Implementation 中,只返回body: unknown或自由文本错误,无法可靠表达已接受但仍待完成的 Process Run。internal → canary → production的部署证据,也没有完成真实网关、角色重启、Redis 恢复、观测告警和付费图片 Process 的 staging smoke。从操作者视角,现在最危险的不是“查询慢”,而是一次不确定提交可能在重试时重复计费;从维护者视角,测试证明了服务端内部链路,却没有证明真实调用方旅程。
Solution
交付一个可恢复、可测试、可受控发布的控制台异步请求闭环。
控制台通过一个深的 Console Process Run Client Module 提交和等待 Process Run。该 Module 为一次用户操作持有稳定幂等键,安全处理响应丢失、容量拒绝和瞬时网络故障;在 durable acceptance 后公开
runId,根据服务端Retry-After查询权威状态,并以结构化结果表达终态、仍待完成、明确拒绝和不可恢复故障。页面导航或 300 秒等待超时只停止客户端查询,不取消服务端 Process Run。测试以 Console Process Run Client Interface 为最高且主要的 Seam。确定性 contract tests 使用本地 Adapter;跨 Seam 集成测试让同一个 Client 穿过测试网关、真实 HTTP、PostgreSQL、Redis、Dispatcher 和 Worker。一个最小浏览器验收覆盖真实表单行为。Production CI/CD 自动运行真实依赖集成门禁;生产保持基础同步 Compose 默认关闭,另提供显式、受控的异步部署形状,按 Runbook 从 internal 分阶段提升。
User Stories
runIdimmediately after durable acceptance, so that I can retain a stable recovery handle.queued,running,succeeded, andfailedstates, so that I know whether to wait or investigate.runId, so that I can continue querying later.Retry-After, so that the console does not hot-loop or overload the service.runIdto a recoverable result view, so that I do not need to copy identifiers from free-form text.runIdvalues to remain indistinguishable, so that Process Runs cannot be enumerated.Locationconstrained to the same origin and expected resource shape, so that a response cannot redirect polling to an arbitrary host.unknownbodies and error strings, so that every caller handles the same states.queued → running → succeeded, so that repeated polling is proven rather than inferred from a single terminal response.503recovery tests, so that the 300-second wait is useful during short outages.Implementation Decisions
submit/findInterface, PostgreSQL Store, Outbox, Queue Adapter, Worker and recovery behaviour remain the server foundation.runId, creation time and current recovery classification. Do not persist business output, gateway credentials or hidden implementation data.202response maps the operation to arunId, preserve that mapping until terminal completion or explicit operator dismissal. Losing the page must not lose the accepted Run’s recovery handle.Retry-Aftervalues for submission capacity responses and nonterminal queries. Apply safe minimum and maximum bounds to prevent hot polling or an unbounded sleep. Use bounded backoff for transient transport failures when no valid server hint exists.429/capacity503as retryable without rotating the key because they do not consume it.Locationrelative to the current origin, then reject cross-origin or non-Process-Run targets before issuing a query.AbortSignalthat cancels client retries, timers and polling only. This signal must never call a server cancellation operation; Process Run cancellation remains outside the current public Interface.runId, current public status, terminal structured result and recovery actions. A pending timeout provides “continue querying”; an acceptance-unknown state provides “retry same operation”; only an explicit “new submission” rotates the key.ASYNC_PROCESS_RUNS_ENABLED=false. Add a separate explicit async deployment shape or overlay owned by the async Runbook; do not add PostgreSQL or Redis containers to the base production Compose.POST /process-runsandGET /process-runs/{runId}contracts unchanged. Product callers still submit only exact Process identity/version and business input.Testing Decisions
queued → running → succeeded; public failure; expired result; malformed submission/query response; missing or cross-originLocation; invalidRetry-After; 300-second timeout; AbortSignal cancellation; transient GET recovery; definitive authorization/not-found failure; admission429/503; and response bodies that are not JSON.runIdwith exactly one Process Run and one paid Business Capability effect.runId/progress, and renders the terminal result. Do not add broad snapshot or private frontend component tests._testPostgreSQL database and nonzero Redis database. It must refuse shared/production targets under existing safety checks./execute, Run Record console, Business Process catalog, activity logging and production build tests remain required regression gates.Out of Scope
Further Notes
submit/findInterface hides Registration acceptance, owner isolation, idempotency, persistence and projection. The design target is to create comparable Depth and Locality on the browser side, not to flatten server Modules into the console.internal → canary → production, preserve query access during rollback and record operational evidence required by the existing async Runbook.