Repository navigation
Conversation
Provider, worker, storage, and API settings currently use different environment/CLI paths. This adds a shared validated configuration source so local processes and hosted roles can use the same settings contract. ## Why is this PR needed? The maintainer requested configurable local/AWS deployments with independently scalable services. On dev at b088553, worker CLI defaults ignore the documented environment settings unless Compose interpolates flags, provider pacing/retry values are not exposed by the environment loader, and loaders duplicate parsing. ## What does this PR change? - Resolve optional TOML configuration, environment overrides, and explicit worker CLI flags through a shared source without mutating the environment. - Reuse each typed value object's defaults and validation across providers, workers, database/archive adapters, budgets, request/export limits, authentication, dispatch, and telemetry. - Add `chess-crawl-admin config validate/show --role ...` without network/SDK/database access, with secret and credential-bearing URL redaction. - Preserve existing environment names, constructors, and CLI flags; expose provider pacing/retries and worker settings consistently. ## Evidence that it solves the reason | Acceptance condition | Baseline evidence | Result and verification | | --- | --- | --- | | Same setting sources across execution paths | Separate getenv/parsing and worker argparse defaults on b088553 | 81 focused configuration/provider/cost cases passed locally, including TOML/environment/CLI precedence, invalid values and URL secret reflection | | API/worker/administration behavior is preserved | Existing PostgreSQL-backed behavior contracts | 141 focused API, worker, operations and configuration tests passed against disposable PostgreSQL 18 | | Shared implementation remains checked | Duplicated loaders and framework-bound export limits | Ruff, Mypy and Bandit passed; export value object now lives in the application layer and existing API imports remain supported | Adversarial inspection found and fixed file-only bootstrap settings, boolean telemetry inconsistency, and credential-bearing URL reflection. Live AWS and Docker runtime validation are not claimed; the PR's CI must pass before reviews and merge. ## Changelog Added an Unreleased entry and a configuration guide with precedence, role checks and integration limits.
## Why is this PR needed? The hosted service needs credentials and quota changes that become effective across API replicas without rebuilding containers. Static token maps and startup defaults currently own both concerns. ## What does this PR change? Adds trusted PostgreSQL administration for workspace provisioning, opaque hashed credentials, rotation/revocation, and versioned quota policies with compare-and-swap updates. Database authentication is explicit and mutually exclusive with static credentials. Managed policies survive default startup settings; existing lifetime run ceilings and usage remain intact. Migration 0017 records policy history and credential hashes. Runtime bootstrap denies credential mutation. Adds `chess-crawl-admin workspaces`, separate managed health credentials, and offline API-role validation that does not build an app or contact PostgreSQL. Billing labels and SaaS account management remain outside this public backend. ## Evidence that it solves the reason | Acceptance condition | Baseline | Verification | | --- | --- | --- | | Cross-replica rotation, immediate revocation and workspace isolation | Static token configuration required restarts | Focused PostgreSQL credential and API tests | | Managed policy changes preserve usage and immutable revisions | Startup policy could tighten shared ceilings | Workspace policy/CAS and existing budget tests | | Offline deployment validation and health credentials | Static-only validation | Configuration, lazy-import, health and CLI tests | Local checks: 30 focused PostgreSQL tests and 96 non-database checks passed; Ruff, Mypy and Bandit passed with zero security findings. Full Python 3.11/3.13 and Compose checks run in CI. No live deployment was performed. ## Changelog Added an Unreleased entry and workspace operator/migration instructions.
## Why is this PR needed? Bounded imports and opponent crawls currently normalize games while holding the shared provider acquisition permit. Hosted deployments need capture and CPU/database processing to scale independently while keeping exact crawl and quota limits. ## What does this PR change? Captures one source unit per acquisition claim and schedules idempotent normalization children. Processing owns opponent-frontier expansion without a provider permit; bounded parents wait for their processing before choosing the next unit. Full/incremental collection keeps its existing pipelining. Shared admission predicates and run locks preserve frontier caps. Shallower rediscovery promotes retained work without refetching; active jobs retain fenced ownership. Unique-game allowances work across duplicate monthly responses, unavailable pinned parsers fail before interpretation, and children retire after parent errors. Captured partial bodies remain pending for replay. Apply migration 0018 after draining older workers. Public API request contracts and original evidence remain intact. ## Evidence that it solves the reason | Acceptance condition | Baseline evidence | Result | | --- | --- | --- | | Release provider permit before normalization/frontier work | Bounded collection performed both inline | Independent stage tests and nine staged acquisition cases | | Exact limits through duplicates and out-of-order discovery | Independent PG probes reproduced missing unique game and lost shallow path | 16 PostgreSQL regression cases plus independent replays passed | | Preserve recovery and source evidence | Partial normalization and parser version dedup must remain durable | Nine bounded-import and six provider-pacing tests passed; retained bytes/status asserted | Ruff, Mypy and Bandit passed. Integration configuration/cloud/health checks: 67 passed. Full Python 3.11/3.13 offline suite and Compose integration run in CI. The local PostgreSQL VM has slower timing than native CI; no local full-suite claim is made. ## Changelog Added staged capture/processing and regression behavior entries under Unreleased, with collection/execution upgrade guidance.
## Why is this PR needed? Polling deployments should not need Mercure, and delivered notifications should not accumulate indefinitely. Durable job/run state and queue dispatch must continue independently of notifications. ## What does this PR change? Adds typed optional event delivery configuration and migration 0019's session-aware outbox guard. Writers continue revisions and dispatch when notifications are disabled. The publisher automatically prunes bounded batches of delivered events; pending notifications remain durable. A trusted `prune-events` command can explicitly expire pending events, under the same exclusive publisher lock. Compose forwards the delivery flag to every database client and retention settings to the publisher. A polling overlay omits the publisher and Mercure hub while disabling writer notifications. The shared runtime catalog validates event policy and rejects a disabled publisher role before loading Mercure credentials. Operators must restart all writers when changing delivery and clients reconcile current API state on subscription recovery. ## Evidence that it solves the reason | Acceptance condition | Baseline evidence | Result and verification | | --- | --- | --- | | Polling without notification infrastructure | Every writer emitted Mercure outbox records | PG regression verifies disabled notifications preserve job revisions and dispatch | | Bounded cleanup preserves retry evidence | Delivered records had no retention | Publisher/retention tests preserve pending records and enforce a combined batch bound | | Safe operator cleanup | HTTP publication can be in flight | Exclusive shared publisher ownership prevents concurrent pruning | 74 PostgreSQL event/configuration cases and 49 final runtime configuration cases passed locally. Ruff, Mypy and Bandit passed. 143 actual Compose/configuration cases and 69 transport contract tests also passed. Full Python 3.11/3.13 offline suites and Compose smoke run in CI. ## Changelog Added optional event delivery and notification retention entries under Unreleased, with `docs/event-delivery.md` and administration command guidance.
## Why is this PR needed? Hosted deployments need to scale acquisition and processing independently while preserving a straightforward local Docker Compose setup. Local and AWS role configuration must express the same operational boundaries. ## What does this PR change? Keeps the default combined-worker Compose experience and adds scalable acquisition/processing profiles, replicated API routing and a managed-auth overlay. External PostgreSQL and the polling overlay now cover every stage role. Shared environment anchors forward provider, archive, quota and usage settings consistently; source access is writable for acquisition and read-only for processing/API. CloudFormation adds independently sized API, acquisition, processing, dispatcher and migration tasks, separate stage queues/DLQs and scoped IAM. Managed-auth wiring preserves the health credential separation introduced in PR42. Existing combined queue upgrade instructions remain explicit; service counts default to zero for operator provisioning. Documents public backend/private hosted product boundaries and local-to-cloud operation without embedding billing plans or private inference behavior. ## Evidence that it solves the reason | Acceptance condition | Baseline evidence | Result and verification | | --- | --- | --- | | Scale stage roles independently | Combined worker/task configuration | Separate stage queue/task and Compose role contracts | | Compose overlays combine correctly | Static credentials can leak through overlays | All overlay combination/rendering and managed-auth tests | | Runtime and cloud settings agree | Numeric bounds and secrets can diverge | Actual owner parsing, CloudFormation lint and configuration tests | 117 real Compose/cloud/health/configuration tests passed locally; cfn-lint, HAProxy syntax, Ruff, Mypy and Bandit passed. Full Python 3.11/3.13 offline suites and live-container Compose smoke run in CI. ## Changelog Added scalable deployment profiles, role configuration and upgrade guidance under Unreleased.
## Why is this PR needed? Independent acquisition and processing workers need observable queue capacity and wait reasons. Hosted operators also need bounded, anonymous usage totals without coupling the backend to a billing plan or metrics vendor. ## What does this PR change? Adds a read-only `chess-crawl-admin metrics` snapshot with counts and oldest ages for stage-eligible work, active leases and exclusive wait reasons. SQL aggregation reuses the same job admission predicates as workers. It reports dispatch/event retry heads and current workspace-budget usage without user identities, targets or credentials. Queries have a bounded deadline and sanitized failures. ## Evidence that it solves the reason | Acceptance condition | Baseline evidence | Result and verification | | --- | --- | --- | | Metrics reflect actual worker eligibility | Wait reasons could differ from job claim rules | Shared admission predicates, stage/cooldown and parent-dependency regressions | | Bounded anonymous operator reads | No operational snapshot command | Timeout/failure and payload privacy tests | | Show backlog and retry delays | Queue size alone hides blocked work | Real PostgreSQL delayed/head and bounded-pipeline cases | 28 metrics/storage-boundary PostgreSQL cases and two full-pipeline cases passed locally. Administration delegation, Ruff, Mypy and Bandit passed. Full Python 3.11/3.13 offline suites and Compose smoke run in CI. ## Changelog Added operational metrics under Unreleased and command/operating guidance.
) ## Why is this PR needed? Large working-set selection and export preparation should run as bounded processing work rather than occupy web requests. Hosted replicas need shared owned artifacts and local deployments need the same contract without cloud services. ## What does this PR change? Adds opt-in workspace-owned archive job endpoints returning 202 with idempotent submission, durable status and owned downloads. Migration 0020 permits providerless jobs only for the two explicit internal operations. Acquisition never claims them; processing reuses existing selection, serializers, budgets, ownership and fair scheduling. Working sets freeze exact game versions at processing time. Exports stream a bounded repeatable snapshot, then publish verified chunk manifests through local or S3 artifact storage. Atomic per-workspace count/byte reservations, attempt-specific namespaces, shared download leases and trusted bounded pruning preserve ownership and retry recovery. Turning admission off still allows draining/status/downloads. Compose and AWS forward the same settings, separate raw evidence from writable artifacts, and scope API/processing storage permissions. Shared offline configuration validates artifact-only S3 dependencies. ## Evidence that it solves the reason | Acceptance condition | Baseline evidence | Result and verification | | --- | --- | --- | | Durable bounded local processing | Synchronous requests owned heavy preparation | Providerless stage, bulk-budget, snapshot and recovery tests | | Workspace isolation and retry-safe artifacts | Multiple replicas can publish/read/delete concurrently | Ownership, exact replay, capacity, attempt generation, lease and corruption regressions | | Same deployment contract locally and on AWS | Artifacts require distinct storage permissions | Actual Compose overlays, cloud IAM/settings and offline SDK validation | Broad PostgreSQL archive/budget/replay checks, 56 combined integration cases, 12 post-quota regressions, 129 deployment/configuration contracts and 61 SDK/configuration cases passed locally. Ruff, Mypy, Bandit, cfn-lint and HAProxy syntax passed. Full Python 3.11/3.13 offline suites and Compose smoke run in CI. ## Changelog Added queued archive operations and artifact storage under Unreleased. `docs/archive-jobs.md` documents processing-time selection, quotas, upgrade, retention and unreferenced-object recovery.
…48) ## Why is this PR needed? Acquisition replicas need shared provider pacing/retry policy that can change without restarting workers, while every process must bound PostgreSQL session usage as replica counts grow. ## What does this PR change? Migration 0021 adds trusted versioned provider policy with optimistic concurrency and append-only history. Provider sessions resolve policy before requests and preserve server cooldowns. Workers recheck pacing, ownership and shutdown after a potentially blocking request-budget reservation; unused reserved bytes are settled if interrupted. A shared per-process admission controller bounds dedicated database sessions, with finite wait and the existing availability/503 behavior. Setup failures and closure release permits once. Worker validation requires independent execution and heartbeat capacity. Shared runtime and deployment configuration forwards session bounds consistently; changing process bounds requires restart. ## Evidence that it solves the reason | Acceptance condition | Baseline evidence | Result and verification | | --- | --- | --- | | Live policy changes govern cached workers | Startup-only provider settings | Real PostgreSQL policy/CAS/cooldown and cached-session tests | | No request after a delayed reservation ignores new policy or shutdown | Deterministic baseline sent HTTP before committed pacing floor | Red/green reservation-delay and cancellation regressions | | Bounded sessions preserve ownership | Unbounded process connection creation | Concurrent admission, timeout, close/setup failure and worker-heartbeat tests | 89 integrated PostgreSQL guards, 35 policy/cooldown/budget cases, 25 provider cases, 25 HTTP budget cases and 177 actual deployment/configuration contracts passed locally. Ruff, Mypy, Bandit and cfn-lint passed. Full Python 3.11/3.13 offline suites and Compose smoke run in CI. ## Changelog Added shared operating policy and process session admission under Unreleased, plus reservation pacing/shutdown fixes and operator guidance.
Summary
Testing
|
xormania
marked this pull request as ready for review
October 6, 2026 07:10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why is this PR needed?
Promote the reviewed backend changes on
devtomaster. These changes prepare Chess-Crawl for a separately maintained hosted product while keeping basic local installations usable.What does this PR change?
The public backend supplies reusable contracts, credentials, work policies and usage. Customer billing and proprietary chess inference/evaluation/training belong in the separate private product layer; see hosted-service boundaries.
Upgrade notes
0017–0021before starting this application version. Restart services after changing deployment configuration. See execution and upgrades.Evidence that it solves the reason
masterlacks the eight merged delivery PRsEvery delivery PR received exactly one
@codex reviewand one@codex security reviewafter CI was green. Findings were resolved and updated checks passed before merge. The promotion's own CI checks validate its merge result separately.Changelog
Carries the substantive
Unreleasedentries already accumulated ondev; no duplicate promotion entry or release version is added.