From 3fe5cd69978ae30a47536c06553103c6c3733341 Mon Sep 17 00:00:00 2001 From: Brian Faust Date: Wed, 26 Aug 2026 16:02:16 +0300 Subject: [PATCH 1/3] chore: finalize standalone documentation Remove archived monorepo links and completed artifacts so users enter through concise, package-owned documentation. Impact: patch --- .ai/GOAL.md | 158 ----------------------------- .ai/GOAL_HARDEN.md | 67 ------------ .golib/scripts/check-module.sh | 2 +- .golib/scripts/repository-check.sh | 2 +- AGENTS.md | 10 +- CHANGELOG.md | 5 + README.md | 6 +- docs/README.md | 55 ++++++---- docs/audit-evidence.md | 94 ----------------- docs/fuzz-report.md | 9 -- docs/hardening.md | 37 ------- docs/mutation-report.md | 19 ---- docs/race-stress-report.md | 22 ---- modules.json | 86 +--------------- scripts/check-docs.sh | 9 +- 15 files changed, 61 insertions(+), 520 deletions(-) delete mode 100644 .ai/GOAL.md delete mode 100644 .ai/GOAL_HARDEN.md delete mode 100644 docs/audit-evidence.md delete mode 100644 docs/fuzz-report.md delete mode 100644 docs/hardening.md delete mode 100644 docs/mutation-report.md delete mode 100644 docs/race-stress-report.md diff --git a/.ai/GOAL.md b/.ai/GOAL.md deleted file mode 100644 index 2c36c0c..0000000 --- a/.ai/GOAL.md +++ /dev/null @@ -1,158 +0,0 @@ -# Goal: Fenced Distributed Leases - -## Objective - -Build a production-grade distributed lease package with explicit ownership, -expiry, renewal, release, and monotonically increasing fencing tokens for -Valkey and PostgreSQL. - -The package MUST provide the coordination primitive required by unique queue -jobs, scheduler overlap prevention, single-owner maintenance work, and bounded -distributed leadership without pretending that an ordinary cache lock provides -stronger guarantees than the backend can deliver. - -## Product Principles - -- A lease is time-bounded ownership, not an indefinitely held mutex. -- Every successful acquisition returns an opaque owner identity and fencing - token that protected resources can use to reject stale owners. -- Expiry, renewal, loss, release, and backend uncertainty are distinct states. -- No process may assume ownership after its lease deadline or uncertain renewal. -- Backend guarantees and clock assumptions are explicit and testable. -- Context cancellation and shutdown never imply successful remote release. - -## Core Model - -- Typed, bounded, namespaced lease keys. -- Immutable acquisition policy: TTL, wait, retry, jitter, renewal, and failure - behavior. -- Lease handle exposing owner, fencing token, acquired time, deadline, state, - renew, validate, and release. -- `TryAcquire`, bounded `Acquire`, explicit `Renew`, and idempotent `Release`. -- Optional managed renewal with explicit goroutine ownership and loss channel. -- Stable errors for contention, timeout, cancellation, lost lease, stale owner, - backend unavailable, invalid state, and ambiguous outcome. -- Test clock and deterministic retry source without production global state. - -## Correctness Semantics - -- Fencing tokens MUST be monotonically increasing for a key within documented - backend continuity guarantees. -- Renewal MUST compare owner identity and current token atomically. -- Release MUST compare ownership atomically and never delete a successor lease. -- A stale handle MUST never renew or release a newer owner's lease. -- TTL and safety margin account for network delay, pauses, scheduling, and - backend clock behavior. -- Acquisition fairness is explicitly documented; no unsupported fairness claim. -- Multi-key atomic leases are out of scope unless a proven backend transaction - model and deadlock policy are added later. - -## Valkey Adapter - -- Native `valkey-go` implementation using atomic server-side scripts/functions. -- Cluster-safe key layout, script loading, `NOSCRIPT`, failover, reconnect, - timeout, ACL, TLS, and rolling-version behavior. -- Backend/server time SHOULD anchor expiry semantics where practical. -- Define fencing continuity after failover, restore, flush, or data loss. -- No Redlock claims or multi-independent-master algorithm by default. - -## PostgreSQL Adapter - -- Native `pgx` implementation with transactional acquisition and renewal. -- Durable lease row and monotonic fencing sequence semantics. -- Indexed schema, cleanup, contention, isolation, deadlock, failover, and - connection-loss behavior. -- Migrations owned through `migrations`. -- PostgreSQL advisory locks MAY be evaluated separately but MUST NOT be confused - with durable TTL leases or reused across pooled sessions unsafely. - -## Integration - -- `queue` middleware for unique jobs and non-overlapping handlers. -- `scheduler` adapter for `onOneServer` and `withoutOverlapping` semantics. -- `idempotency` MAY consume lease/fencing primitives where its stronger - operation state machine remains intact. -- `service` lifecycle integration for managed renewal and shutdown. -- Optional `log` and `telemetry` observations with hashed bounded keys. -- Protected-resource examples MUST demonstrate fencing checks; acquiring a - lease alone is not sufficient safety documentation. -- Valkey MAY expose opaque active-ownership coordinates to a protected writer - that atomically compares the exact lease owner and token with a - reconstructible cache write. - -## Security And Resource Bounds - -- Cryptographically random owner identities with injectable test source. -- Bounded key, owner, waiters, retry attempts, renewal goroutines, observations, - cleanup batches, and backend operations. -- Keys and owner identities MUST not leak through default logs or metric labels. -- Threat-model stale writers, split brain, clock anomalies, replay, token - overflow, key collision, backend rollback, restore, and malicious contention. -- Callbacks MUST not execute while internal locks are held. - -## Non-Goals - -- No distributed transaction, consensus system, membership service, election - platform, semaphore, idempotency state machine, or queue. -- No guarantee that expired work stopped; fencing is required for protected - resources when stale work is dangerous. -- No hidden infinite waiting or retry. -- No in-memory adapter presented as distributed. -- No Redis/Valkey compatibility through one ambiguous adapter. - -## Package Shape - -- Root: keys, policies, handles, states, errors, retry, observations. -- `memory`: deterministic process-local reference and tests only, clearly scoped. -- `valkey` and `postgres`: native distributed adapters. -- `leasequeue`, `leasescheduler`, and `leaseservice`: integrations. -- `leasetest`: conformance, clocks, fault injection, and fencing assertions. - -## Testing And Quality Standard - -Meaningful 100% production statement coverage is mandatory. Required evidence: - -- state-machine and model-based tests for acquisition through final release -- cross-backend lease and fencing conformance -- race/stress tests for contention, renewal, loss, cancellation, and shutdown -- Valkey and PostgreSQL failover, restart, timeout, partition, and fault tests -- stale-owner and successor-protection tests at every operation boundary -- clock skew/jump, process pause, token overflow, and retry fuzzing -- mutation testing of ownership comparisons and stale-owner rejection -- benchmarks for contention, renewal load, latency, allocations, and cleanup - -## Documentation Deliverables - -- Five-minute Valkey and PostgreSQL quickstarts. -- Formal state machine, fencing model, backend guarantees, and API reference. -- Guides for unique jobs, schedulers, protected writes, renewal, loss handling, - shutdown, Kubernetes, failover, and migrations. -- Laravel lock/unique-job migration guide, threat model, operations runbook, - performance, FAQ, troubleshooting, examples, and changelog. -- Every exported API and user-facing scenario MUST be documented. - -## Automation And Release - -Use the latest stable Go release as the minimum at implementation time. Pin all -tools and dependencies. GitHub Actions MUST run formatting, vet, Staticcheck, -strict golangci-lint, advisory NilAway, tests, meaningful 100% coverage, race, -fuzz smoke, mutation checks, Valkey/PostgreSQL matrices, vulnerability scans, -benchmarks, docs, API compatibility, and releases. All blocking commands MUST -be locally reproducible through documented `make` targets. - -## Execution Plan - -1. Specify states, owner identity, fencing, timing, errors, and conformance. -2. Implement deterministic reference behavior and native Valkey adapter. -3. Implement PostgreSQL adapter and migration contract. -4. Add queue, scheduler, service, logging, and telemetry integrations. -5. Complete failover, stale-owner, race, mutation, and performance hardening. -6. Publish complete operational documentation and release v1. - -## Acceptance Criteria - -- Stale owners cannot renew, release, or overwrite protected successor work. -- Every backend satisfies the documented fencing and lease state machine. -- Lease uncertainty and loss always stop ownership-dependent admission. -- Resource use, retries, wait, renewal, and shutdown are bounded. -- Meaningful 100% coverage and every required CI gate pass. diff --git a/.ai/GOAL_HARDEN.md b/.ai/GOAL_HARDEN.md deleted file mode 100644 index 2ec879f..0000000 --- a/.ai/GOAL_HARDEN.md +++ /dev/null @@ -1,67 +0,0 @@ -# Hardening Goal: Fenced Distributed Leases - -## Objective - -Prove that `lease` prevents stale-owner corruption and successor deletion -under contention, pauses, clock anomalies, backend failure, failover, restore, -network uncertainty, cancellation, and rolling deployments. - -## Required Audits - -### State And Fencing Audit - -- Model every acquisition, contention, renewal, expiry, loss, validation, - release, and successor transition. -- Mutation-test owner and token comparisons capable of accepting stale work. -- Prove fencing tokens remain monotonic within documented continuity bounds. -- Verify stale handles cannot affect a successor after every failure point. - -### Timing And Concurrency Audit - -- Inject clock rollback/jump, scheduler delay, process pause, stop-the-world-like - delay, network latency, frozen time, and late responses. -- Race and stress-test acquisition, managed renewal, explicit renewal, release, - cancellation, loss notification, and shutdown. -- Prove callbacks and observations cannot deadlock lease state transitions. -- Enforce wait, retry, jitter, goroutine, and renewal safety margins. - -### Backend Failure Audit - -- Exercise Valkey failover, restart, `NOSCRIPT`, ACL/TLS rotation, disconnect, - restore, flush, and ambiguous write outcomes. -- Exercise PostgreSQL failover, deadlocks, transaction abort, pool churn, - isolation, sequence behavior, restore, and cleanup races. -- Document exactly when fencing continuity can reset and how operators detect it. -- Test rolling clients with compatible and incompatible schema/script versions. - -### Security And Integration Audit - -- Threat-model split brain, stale writer, replay, owner spoofing, key collision, - denial of service, token overflow, and sensitive key leakage. -- Prove queue and scheduler integrations stop ownership-sensitive work on loss. -- Prove protected-write examples reject stale fencing tokens transactionally. -- Verify errors, logs, metrics, traces, and inspection output redact identifiers. - -## Required Deliverables - -- Formal state machine, fencing proof matrix, and backend continuity contract. -- Threat model, failure matrix, resource budgets, and hardening findings. -- Valkey/PostgreSQL failover, restore, fault, and rolling-upgrade evidence. -- Race, fuzz, mutation, stale-owner, and benchmark reports. -- Updated API, operations, security, migration, FAQ, and troubleshooting docs. - -## Release Blockers - -- Any stale owner that can renew, release, or commit a protected write. -- Any successor lease deleted by an earlier handle. -- Any undocumented fencing reset, ambiguous ownership treated as valid, race, - deadlock, panic, leaked renewal goroutine, or unbounded wait/retry. -- Missing meaningful 100% coverage, mutation evidence, or green blocking CI. - -## Completion Criteria - -- State, fencing, timing, backend, restore, and integration suites pass. -- Every ambiguity fails ownership closed and is observable. -- Race, fuzz, vulnerability, compatibility, and performance gates pass. -- NilAway runs visibly as advisory without blocking findings. -- No release blocker remains and the changelog is current. diff --git a/.golib/scripts/check-module.sh b/.golib/scripts/check-module.sh index 2c521e3..84e01a4 100755 --- a/.golib/scripts/check-module.sh +++ b/.golib/scripts/check-module.sh @@ -125,7 +125,7 @@ refresh_owned_sums() { [[ -f go.sum ]] || return 0 local temporary temporary="$(mktemp "${TMPDIR:-/tmp}/golib-go-sum.XXXXXX")" - awk '$1 !~ /^github\.com\/faustbrian\/golib\// { print }' \ + awk '$1 !~ /^github\.com\/faustbrian\/go-/ { print }' \ go.sum >"${temporary}" if cmp -s go.sum "${temporary}"; then rm -f "${temporary}" diff --git a/.golib/scripts/repository-check.sh b/.golib/scripts/repository-check.sh index 353adb2..231c796 100755 --- a/.golib/scripts/repository-check.sh +++ b/.golib/scripts/repository-check.sh @@ -34,7 +34,7 @@ if grep -REnI \ --exclude='go.sum' \ --exclude='CHANGELOG.md' \ --exclude='repository-check.sh' \ - 'github\.com/faustbrian/golib/pkg|/Users/[^/]+/Developer|\.\./go-' \ + 'github\.com/faustbrian/golib|/Users/[^/]+/Developer|\.\./go-' \ "${root}"; then printf 'monorepo or sibling-checkout reference remains\n' >&2 exit 1 diff --git a/AGENTS.md b/AGENTS.md index 6ea988c..c5b3d40 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -11,8 +11,8 @@ shown here. - This file is the canonical policy for the complete repository. - Package policies MAY add stricter domain rules but MUST NOT weaken this file. - `CLAUDE.md` and tool-specific files MUST point here rather than duplicate it. -- Historical `.ai/GOAL*.md` files are requirements and evidence, not proof of - completion. Current executable evidence is REQUIRED. +- Historical implementation plans belong in repository history or issue + tracking, not in the released source tree. Current checks MUST pass. ## Repository Structure @@ -177,7 +177,11 @@ shown here. - Comments MUST explain why a constraint or non-obvious implementation exists; they MUST NOT narrate obvious syntax. - Every public module MUST provide a quick start, API reference, examples, - adoption guidance, tradeoffs, security notes, FAQ, and release notes. + guidance on when to use it, explicit limitations, security notes, FAQ, and + release notes. +- The root README MUST remain a concise entry point. Detailed guides, + operations, audits, and maintainer material belong under `docs/` and MUST be + linked through `docs/README.md`. - Documentation and examples MUST compile and be checked in CI. ## Changelogs diff --git a/CHANGELOG.md b/CHANGELOG.md index bd9f5d4..d3c0ec0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,11 @@ ## Unreleased +### Documentation + +- Replace archived monorepo links and completed execution artifacts with a + standalone, human-oriented documentation structure. + ## 1.0.0 - 2026-08-25 ### Changed diff --git a/README.md b/README.md index 9f0d9d4..06d5210 100644 --- a/README.md +++ b/README.md @@ -62,7 +62,7 @@ the returned error. No acquisition order or starvation guarantee is provided. See the [documentation index](docs/README.md), [security policy](SECURITY.md), and [changelog](CHANGELOG.md). -## Ecosystem +## Documentation -Use the [Golib documentation portal](https://github.com/faustbrian/golib/blob/main/docs/index.md) -to choose companion packages, supported stacks, recipes, and operations guidance. +Use the [documentation index](docs/README.md) for package-owned guides, +operational contracts, examples, and maintainer references. diff --git a/docs/README.md b/docs/README.md index 09224aa..a371ec5 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,20 +1,39 @@ # Documentation -- Concepts: [state machine](state-machine.md), [fencing](fencing.md), - [backend guarantees](backend-guarantees.md), [API](api.md) -- Setup: [Valkey](quickstart-valkey.md), - [PostgreSQL](quickstart-postgres.md), [migrations](migrations.md) -- Integration: [unique jobs](unique-jobs.md), [schedulers](schedulers.md), - [protected writes](protected-writes.md), [service shutdown](shutdown.md) -- Operations: [renewal and loss](renewal-and-loss.md), - [Kubernetes](kubernetes.md), [failover](failover.md), - [operations](operations.md), [troubleshooting](troubleshooting.md) -- Assurance: [threat model](threat-model.md), - [acceptance audit](audit-evidence.md), - [failure matrix](failure-matrix.md), [resource budgets](resource-budgets.md), - [hardening evidence](hardening.md), [performance](performance.md), - [benchmark baseline](benchmark-baseline.md), [fuzz report](fuzz-report.md), - [race and stress report](race-stress-report.md), - [mutation report](mutation-report.md), [compatibility](compatibility.md) -- Migration: [Laravel locks and unique jobs](laravel-migration.md) -- Help: [FAQ](faq.md) +## Getting started + +- [Valkey quickstart](quickstart-valkey.md) +- [PostgreSQL quickstart](quickstart-postgres.md) +- [API reference](api.md) +- [Database migrations](migrations.md) + +## Concepts and integrations + +- [State machine](state-machine.md) +- [Fencing](fencing.md) +- [Backend guarantees](backend-guarantees.md) +- [Protected writes](protected-writes.md) +- [Unique jobs](unique-jobs.md) +- [Schedulers](schedulers.md) +- [Renewal and loss](renewal-and-loss.md) +- [Service shutdown](shutdown.md) + +## Operations + +- [Operations guide](operations.md) +- [Kubernetes](kubernetes.md) +- [Failover](failover.md) +- [Failure matrix](failure-matrix.md) +- [Resource budgets](resource-budgets.md) +- [Threat model](threat-model.md) +- [Performance](performance.md) +- [Troubleshooting](troubleshooting.md) + +## Reference and maintenance + +- [Compatibility](compatibility.md) +- [Benchmark baseline](benchmark-baseline.md) +- [Laravel migration](laravel-migration.md) +- [FAQ](faq.md) +- [Contributing](../CONTRIBUTING.md) +- [Release history](../CHANGELOG.md) diff --git a/docs/audit-evidence.md b/docs/audit-evidence.md deleted file mode 100644 index 45586bd..0000000 --- a/docs/audit-evidence.md +++ /dev/null @@ -1,94 +0,0 @@ -# Acceptance audit evidence - -This audit maps the fenced distributed lease goal to the current local tree. -Hosted CI remains the maintainer's final external verification step. - -## Core model - -| Requirement | Evidence | -|---|---| -| bounded namespaced keys | `key.go`, `TestKeyIsBoundedAndNamespaced` | -| immutable acquisition policy | `policy.go`, `TestPolicyIsValidatedAndImmutable` | -| owner, fence, times, state | `record.go`, `handle.go`, lifecycle tests | -| try, bounded wait, renew, validate, release | `client.go`, `handle.go` | -| managed renewal and loss | `managed.go`, uncertainty and shutdown tests | -| stable errors | `errors.go`, classification tests | -| deterministic clock/retry | `leasetest.Clock`, injected retry tests | -| waiter, operation, goroutine bounds | client capacity and timeout tests | -| backend skew, client rollback/freeze, response pause | dual local deadline tests | - -## Correctness and backends - -| Requirement | Evidence | -|---|---| -| monotonically increasing success fences | shared conformance and model fuzzing | -| atomic owner plus token renewal | Valkey Lua and PostgreSQL conditional SQL | -| successor-safe release | shared conformance and stale release tests | -| backend time | Valkey `TIME`; PostgreSQL `clock_timestamp()` | -| Valkey cluster-safe layout | hashed same-slot lease/counter key test | -| `NOSCRIPT` recovery | live `SCRIPT FLUSH` integration test | -| PostgreSQL durable continuity | separate `lease_fences` table and migration | -| cleanup without fence reset | bounded cleanup tests and retained counters | -| restore/flush continuity limits | backend guarantees and failover guides | - -Disposable local runs passed under `-race` for PostgreSQL 14, 15, 16, 17, and -18 and Valkey 9. Both backends passed after container restart; Valkey also -passed after script-cache flush. Client-owned TLS, ACL, reconnect, pool, and -command-timeout failures surface through fail-closed adapter tests. The -reproducible `make backend-hardening` target seeds and snapshots fixed fences, -proves continuity across restart, detects reuse after older-snapshot restore, -proves destructive reset returns token 1, promotes streaming PostgreSQL and -Valkey replicas, and repeats script-cache and live partition classification -checks in CI. Its secure Valkey phase rotates the CA, server certificate, named -ACL user, and password; old trust and old ACL credentials fail closed before -the new client proves fence continuity. - -The PostgreSQL operational fault phase forces a transaction abort and a real -`40P01` deadlock, races bounded cleanup against successor acquisition, churns -caller-owned pools, runs acquisition at serializable isolation, and exercises -both additive-compatible and fail-closed incompatible rolling schemas. Fence -history must remain monotonic after every phase. PostgreSQL uses transactional -fence rows instead of sequences; the abort phase proves a rolled-back increment -does not create a committed jump and the next acquisition advances exactly -once. - -The physical-replica phase authorizes SCRAM replication only from the -disposable primary's directly connected network before `pg_basebackup`; the -rule is reloaded explicitly and disappears with the fault container. - -Valkey rolling-script tests accept the documented v1 response and reject -added, removed, or changed response fields as unavailable rather than treating -an incompatible response as ownership. - -## Integrations and security - -Queue workers and scheduler callbacks receive the fence; their direct loss -tests advance authoritative time beyond expiry and prove each callback context -is canceled with `ErrLost`. Service lifecycle bounds handles, stops renewers, -and reports remote release failure. The protected-write example race-tests -concurrent writers and rejects stale and replayed tokens atomically. - -Cryptographic 192-bit owners, hashed backend keys, redacted observations, -redacted classified driver errors, observer panic isolation, token overflow -checks, bounded nonblocking observer slots, bounded cleanup, and the threat -model cover spoofing, collision, -leakage, stale writers, split brain, rollback, restore, malicious contention, -and resource exhaustion. - -## Quality and release - -- `make check`: format, vet, unit, race, exact 100.0% production statement - coverage, repeated lifecycle stress, fuzz smoke, benchmarks, docs, examples, - and API baseline -- `make lint staticcheck`: strict analyzer gates -- `make mutation`: 23 Go mutants plus four adapter comparison classes killed, - zero lived, 100% efficacy and mutant coverage -- `make vuln`: no known reachable Go vulnerabilities -- `make workflows`: pinned workflow syntax validation -- `make nilaway`: visible advisory analysis; findings do not block by policy -- CI: PostgreSQL 14-18 and Valkey 9 matrices, mutation, vulnerability, lint, - advisory NilAway, release workflow, and the locally reproducible gates - -No unsupported fairness, consensus, multi-key atomicity, stopped-expired-work, -or distributed-transaction claim is made. Every dangerous stale effect still -requires protected-resource fencing. diff --git a/docs/fuzz-report.md b/docs/fuzz-report.md deleted file mode 100644 index cd00d43..0000000 --- a/docs/fuzz-report.md +++ /dev/null @@ -1,9 +0,0 @@ -# Fuzz report - -`make fuzz` runs bounded smoke fuzzing for canonical key parsing, policy bounds, -and the memory lease state model. The model limits each input to 128 operations -and mixes acquire, renew, release, expiry jumps, and clock rollback. - -Long-running qualification should extend `FUZZ_TIME` and preserve any generated -corpus. A crash, non-monotonic successful fence, accepted out-of-bound policy, -or unexpected state-machine error is a release blocker. diff --git a/docs/hardening.md b/docs/hardening.md deleted file mode 100644 index c2f27fc..0000000 --- a/docs/hardening.md +++ /dev/null @@ -1,37 +0,0 @@ -# Hardening evidence - -| Requirement | Executable evidence | -|---|---| -| successor-safe stale rejection | `leasetest.RunBackendConformance` | -| independent owner/token rejection | shared conformance forged-identity matrix | -| late/corrupt successful response | `TestHandleRejectsMismatchedSuccessfulResponses` | -| state model and clock jumps | `FuzzLeaseStateModel` | -| backend skew, rollback, and frozen client clock | local dual-deadline tests | -| bounded retry and jitter | `TestAcquireUsesInjectedBoundedJitter` | -| contention race | `TestContentionElectsExactlyOneOwner`, `make race` | -| repeated lifecycle concurrency | `make stress` | -| renewal uncertainty | `TestManagedRenewalReportsUncertaintyAndStopsAdmission` | -| observer re-entrancy | `TestObserverCanInspectHandleDuringStateTransition` | -| blocking observer isolation | `TestBlockingObserverCannotDelayLeaseTransition` | -| queue and scheduler loss cancellation | direct integration loss tests | -| concurrent handle operations | `TestHandleRejectsConcurrentOperationsWithoutBlockingState` | -| deadline during remote operation | `TestHandleFailsClosedWhenDeadlinePassesDuringOperation` | -| shutdown semantics | `leaseservice` hardening tests | -| response corruption | Valkey/PostgreSQL hardening tests | -| script owner/token comparison | `TestScriptsUseBackendTimeAndAtomicComparisons` | -| meaningful production coverage | `make coverage` requires exactly 100.0% | -| fuzz smoke | `make fuzz` | -| ownership mutation resistance | `make mutation`, including Lua/SQL predicates | -| live backend parity | `make integration` with backend environment variables | -| restart, restore, reset, partition, backend promotion | `make backend-hardening` | -| PostgreSQL abort, deadlock, cleanup race, pool churn, isolation | `TestLiveOperationalFaults` | -| Valkey TLS and named ACL rotation | secure continuity phases in backend hardening | -| allocation and latency baseline | `make benchmark` | - -Local release verification uses `make check lint staticcheck nilaway mutation -vuln workflows`, `make backend-hardening`, and live PostgreSQL 14-18 and Valkey -9 matrices. The hardening target owns disposable PostgreSQL 18 and Valkey 9 -containers, repeats conformance after restart, exercises script-cache recovery, -and requires partitioned mutations to return `ErrAmbiguousOutcome`. Hosted CI -is the final external verification step and is not used to block local -progress. diff --git a/docs/mutation-report.md b/docs/mutation-report.md deleted file mode 100644 index 183ee38..0000000 --- a/docs/mutation-report.md +++ /dev/null @@ -1,19 +0,0 @@ -# Mutation report - -`make mutation` pins Gremlins v0.6.0, two workers, and timeout coefficient 10. -It targets the memory reference state machine, where mutations to owner/token -comparison, expiry boundaries, counter increments, active state, and release -conditions are exercised by conformance, model, and hardening tests. It then -runs `scripts/check-fence-mutations.sh` against disposable source copies. - -The adapter mutation gate changes Valkey owner/token comparisons from unequal -to equal and PostgreSQL owner/token comparisons from equal to unequal. The -exact contract tests must fail for every mutated copy. This covers the Lua and -SQL predicates that Go mutation tools cannot rewrite directly. -Shared conformance independently forges only the owner and only the token for -renew, validate, and release, then revalidates the successor after every -rejection. This detects an adapter that accidentally compares only one field. - -The current reference run generated 23 Go mutants: 23 killed, zero lived, zero -uncovered, 100.00% efficacy, and 100.00% mutant coverage. The adapter run -generated four comparison mutation classes: four killed and zero lived. diff --git a/docs/race-stress-report.md b/docs/race-stress-report.md deleted file mode 100644 index c3740b1..0000000 --- a/docs/race-stress-report.md +++ /dev/null @@ -1,22 +0,0 @@ -# Race and lifecycle stress report - -`make stress` repeats the ownership-critical concurrency matrix 25 times under -Go's race detector with shuffled test order. The gate covers contended -acquisition, cancellation, explicit renewal and release, concurrent handle -operations, late responses, managed renewal and loss notification, stale-owner -successor protection, queue and scheduler loss cancellation, and service -shutdown races. - -The command is deterministic in scope and bounded in repetitions. Increase -`STRESS_COUNT` for soak runs; the blocking local and hosted gate keeps the -documented default so execution time remains bounded. - -The release gate is: - -```text -go test -race -shuffle=on -count=25 -run ./... -``` - -Passing this report does not replace backend fault tests. PostgreSQL and Valkey -failover, restore, partition, and credential phases remain in -`make backend-hardening`. diff --git a/modules.json b/modules.json index aeb5469..7f23da0 100644 --- a/modules.json +++ b/modules.json @@ -200,89 +200,9 @@ "specifications": [], "conformance_corpora": [], "provenance": [], - "goal_files": [ - ".ai/GOAL.md", - ".ai/GOAL_HARDEN.md" - ], - "goal_status": "implementation-evidence-inventoried", - "goal_evidence": [ - { - "file": ".ai/GOAL.md", - "requirements_sha256": "504032dfaf2d089b625af60b1a0669e1189f10e8e01728795921379d4b2b4d14", - "implementation_evidence": [ - "CHANGELOG.md", - "README.md", - "docs/README.md", - "docs/api.md", - "docs/compatibility.md", - "docs/hardening.md", - "docs/performance.md", - "docs/threat-model.md" - ], - "verification_gates": [ - "format-check", - "tidy-check", - "safety", - "vet", - "test", - "race", - "coverage", - "lint", - "staticcheck", - "vulnerability", - "secrets", - "licenses", - "sbom", - "fuzz", - "mutation", - "nilaway", - "docs", - "api", - "conformance", - "interoperability", - "benchmark" - ], - "implementation_status": "implemented-requires-fresh-verification" - }, - { - "file": ".ai/GOAL_HARDEN.md", - "requirements_sha256": "54066fea5f68f0f83a505da2ce71d3606ad5257722a084e96bde7a1a70a7330d", - "implementation_evidence": [ - "CHANGELOG.md", - "README.md", - "docs/README.md", - "docs/api.md", - "docs/compatibility.md", - "docs/hardening.md", - "docs/performance.md", - "docs/threat-model.md" - ], - "verification_gates": [ - "format-check", - "tidy-check", - "safety", - "vet", - "test", - "race", - "coverage", - "lint", - "staticcheck", - "vulnerability", - "secrets", - "licenses", - "sbom", - "fuzz", - "mutation", - "nilaway", - "docs", - "api", - "conformance", - "interoperability", - "benchmark" - ], - "implementation_status": "implemented-requires-fresh-verification" - } - ], + "goal_files": [], + "goal_status": "not-applicable", + "goal_evidence": [], "gates": { "api_compatibility": true, "benchmarks": true, diff --git a/scripts/check-docs.sh b/scripts/check-docs.sh index deee6e4..5599cbe 100755 --- a/scripts/check-docs.sh +++ b/scripts/check-docs.sh @@ -3,15 +3,14 @@ set -euo pipefail required=( README.md CHANGELOG.md SECURITY.md CONTRIBUTING.md - docs/api.md docs/audit-evidence.md docs/backend-guarantees.md docs/failure-matrix.md - docs/benchmark-baseline.md docs/compatibility.md docs/fuzz-report.md - docs/faq.md docs/fencing.md docs/hardening.md docs/kubernetes.md + docs/api.md docs/backend-guarantees.md docs/failure-matrix.md + docs/benchmark-baseline.md docs/compatibility.md + docs/faq.md docs/fencing.md docs/kubernetes.md docs/laravel-migration.md docs/migrations.md docs/operations.md - docs/mutation-report.md docs/performance.md docs/protected-writes.md docs/quickstart-postgres.md docs/quickstart-valkey.md docs/renewal-and-loss.md docs/resource-budgets.md docs/schedulers.md docs/shutdown.md - docs/state-machine.md docs/race-stress-report.md docs/threat-model.md + docs/state-machine.md docs/threat-model.md docs/troubleshooting.md docs/unique-jobs.md ) From ed797a8e5618cb6949b0b48e3a2d1f239190c76b Mon Sep 17 00:00:00 2001 From: Brian Faust Date: Wed, 26 Aug 2026 17:10:38 +0300 Subject: [PATCH 2/3] docs: finalize standalone documentation Replace archived portal wording with package-owned indexes and keep user-facing README files focused while preserving full technical guides. Impact: patch --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d3c0ec0..5b374af 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,7 +28,7 @@ ### Documentation -- Link the package README to the repository-wide Golib documentation portal. +- Link the package README to package-owned documentation. ### Changed From fcd6ea726e9a3e63ff85f88521d8e67c4be5cbfc Mon Sep 17 00:00:00 2001 From: Brian Faust Date: Sat, 29 Aug 2026 06:32:35 +0300 Subject: [PATCH 3/3] chore(tooling): adopt go-library-tools v1.0.4 Replace copied repository verification scripts with the released shared contract, preserving package checks and exact mutation evidence. Pin the workflow and CLI checksum so hosted verification uses the same tooling contract without changing library behavior. Impact: none --- .github/workflows/ci.yml | 253 +--- .golib.yaml | 65 ++ .golib/documentation-tools.env | 6 - .golib/package.mk | 75 -- .golib/scripts/audit-goals.sh | 109 -- .golib/scripts/build-golib-gremlins.sh | 69 -- .golib/scripts/build-local-proxy.sh | 180 --- .golib/scripts/check-api-baseline.sh | 66 -- .golib/scripts/check-coverage.sh | 133 --- .golib/scripts/check-documentation.sh | 107 -- .golib/scripts/check-fuzz.sh | 61 - .golib/scripts/check-gates.txt | 21 - .golib/scripts/check-go-safety.go | 87 -- .golib/scripts/check-go-safety.sh | 14 - .golib/scripts/check-module.sh | 516 -------- .golib/scripts/check-mutation.sh | 10 - .golib/scripts/codeql-build.sh | 65 -- .../scripts/create-verification-snapshot.sh | 59 - .golib/scripts/discover-mutation.sh | 10 - .golib/scripts/filter-releasable-modules.sh | 12 - .golib/scripts/gate-input-digest.sh | 1035 ----------------- .../internal/configure-mutation-workers.sh | 20 - .golib/scripts/internal/isolated-go.sh | 144 --- .../internal/migrate-api-baseline-evidence.sh | 165 --- .../migrate-runner-isolation-evidence.sh | 168 --- .golib/scripts/internal/mutation-command.sh | 27 - .golib/scripts/internal/mutation-coverage.sh | 31 - .golib/scripts/internal/mutation-scratch.sh | 169 --- .../internal/optional-mutation-digest.sh | 23 - .../reuse-approved-mutation-checkpoint.sh | 118 -- .../internal/reuse-mutation-coverage.sh | 42 - .golib/scripts/internal/run-mutation.sh | 686 ----------- .../internal/run-verification-snapshots.sh | 137 --- .golib/scripts/mutation-verifier-identity.sh | 34 - .golib/scripts/package-source-digest.sh | 42 - .../gremlins-module-relative-diff.patch | 89 -- .../patches/gremlins-run-all-mutants.patch | 60 - .../patches/gremlins-shared-coverage.patch | 51 - .golib/scripts/release.sh | 59 - .golib/scripts/repository-check.sh | 46 - .../scripts/restore-ci-mutation-evidence.sh | 273 ----- .golib/scripts/run-gate-with-evidence.sh | 234 ---- .golib/scripts/run-modules.sh | 45 - .golib/scripts/stage-ci-evidence.sh | 144 --- .golib/scripts/start-services.sh | 296 ----- .golib/scripts/stop-services.sh | 110 -- .../test-oidc-keycloak-interoperability.sh | 81 -- .golib/scripts/update-api-baseline.sh | 48 - .golib/scripts/verify-gate-evidence.sh | 118 -- .golib/scripts/with-disposable-go-cache.sh | 18 - .golib/versions.env | 20 - .../mutation/bootstrap}/root.zip | Bin .../mutation/migration-ledger.json | 0 .../mutation/zero-inventory.json | 0 CHANGELOG.md | 5 + CONTRIBUTING.md | 5 + Makefile | 10 +- scripts/check-api-compat.sh | 5 - scripts/check-coverage.sh | 12 - scripts/check-fuzz.sh | 7 - scripts/check-stress.sh | 7 - verification/package.mk | 10 + 62 files changed, 102 insertions(+), 6410 deletions(-) create mode 100644 .golib.yaml delete mode 100644 .golib/documentation-tools.env delete mode 100644 .golib/package.mk delete mode 100755 .golib/scripts/audit-goals.sh delete mode 100755 .golib/scripts/build-golib-gremlins.sh delete mode 100755 .golib/scripts/build-local-proxy.sh delete mode 100755 .golib/scripts/check-api-baseline.sh delete mode 100755 .golib/scripts/check-coverage.sh delete mode 100755 .golib/scripts/check-documentation.sh delete mode 100755 .golib/scripts/check-fuzz.sh delete mode 100644 .golib/scripts/check-gates.txt delete mode 100644 .golib/scripts/check-go-safety.go delete mode 100755 .golib/scripts/check-go-safety.sh delete mode 100755 .golib/scripts/check-module.sh delete mode 100755 .golib/scripts/check-mutation.sh delete mode 100755 .golib/scripts/codeql-build.sh delete mode 100755 .golib/scripts/create-verification-snapshot.sh delete mode 100755 .golib/scripts/discover-mutation.sh delete mode 100755 .golib/scripts/filter-releasable-modules.sh delete mode 100755 .golib/scripts/gate-input-digest.sh delete mode 100755 .golib/scripts/internal/configure-mutation-workers.sh delete mode 100755 .golib/scripts/internal/isolated-go.sh delete mode 100755 .golib/scripts/internal/migrate-api-baseline-evidence.sh delete mode 100755 .golib/scripts/internal/migrate-runner-isolation-evidence.sh delete mode 100755 .golib/scripts/internal/mutation-command.sh delete mode 100755 .golib/scripts/internal/mutation-coverage.sh delete mode 100644 .golib/scripts/internal/mutation-scratch.sh delete mode 100755 .golib/scripts/internal/optional-mutation-digest.sh delete mode 100755 .golib/scripts/internal/reuse-approved-mutation-checkpoint.sh delete mode 100755 .golib/scripts/internal/reuse-mutation-coverage.sh delete mode 100755 .golib/scripts/internal/run-mutation.sh delete mode 100755 .golib/scripts/internal/run-verification-snapshots.sh delete mode 100755 .golib/scripts/mutation-verifier-identity.sh delete mode 100755 .golib/scripts/package-source-digest.sh delete mode 100644 .golib/scripts/patches/gremlins-module-relative-diff.patch delete mode 100644 .golib/scripts/patches/gremlins-run-all-mutants.patch delete mode 100644 .golib/scripts/patches/gremlins-shared-coverage.patch delete mode 100755 .golib/scripts/release.sh delete mode 100755 .golib/scripts/repository-check.sh delete mode 100755 .golib/scripts/restore-ci-mutation-evidence.sh delete mode 100755 .golib/scripts/run-gate-with-evidence.sh delete mode 100755 .golib/scripts/run-modules.sh delete mode 100755 .golib/scripts/stage-ci-evidence.sh delete mode 100755 .golib/scripts/start-services.sh delete mode 100755 .golib/scripts/stop-services.sh delete mode 100755 .golib/scripts/test-oidc-keycloak-interoperability.sh delete mode 100755 .golib/scripts/update-api-baseline.sh delete mode 100755 .golib/scripts/verify-gate-evidence.sh delete mode 100755 .golib/scripts/with-disposable-go-cache.sh delete mode 100644 .golib/versions.env rename {.golib/mutation-bootstrap => .verification/mutation/bootstrap}/root.zip (100%) rename .golib/mutation-history-migrations.json => .verification/mutation/migration-ledger.json (100%) rename .golib/mutation-zero-inventory.json => .verification/mutation/zero-inventory.json (100%) delete mode 100755 scripts/check-api-compat.sh delete mode 100755 scripts/check-coverage.sh delete mode 100755 scripts/check-fuzz.sh delete mode 100755 scripts/check-stress.sh create mode 100644 verification/package.mk diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1f8df68..3277d6a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -9,261 +9,30 @@ on: workflow_dispatch: inputs: release_dry_run: - description: Run the stable v1 dry-run for every releasable module + description: Run the stable release rehearsal required: false default: false type: boolean permissions: - actions: read contents: read - -concurrency: - group: go-lease-${{ github.event.pull_request.number || github.ref }} - cancel-in-progress: ${{ github.event_name == 'pull_request' }} + security-events: write jobs: - prepare: - name: Select modules - runs-on: ubuntu-24.04 - timeout-minutes: 15 - outputs: - matrix: ${{ steps.selection.outputs.matrix }} - steps: - - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 - - name: Select repository modules - id: selection - env: - RELEASE_DRY_RUN: ${{ inputs.release_dry_run }} - run: | - set -euo pipefail - filter='.' - if [[ "${RELEASE_DRY_RUN}" == true ]]; then - filter='map(select(.releasable == true))' - fi - matrix="$( - jq -c \ - --arg filter "${filter}" ' - .modules - | if $filter == "." then . else map(select(.releasable == true)) end - | map({ - directory, - artifact: ( - if .directory == "." then "root" - else (.directory | gsub("/"; "-")) - end - ) - }) - ' modules.json - )" - [[ "$(jq 'length' <<<"${matrix}")" -gt 0 ]] - echo "matrix=${matrix}" >>"${GITHUB_OUTPUT}" - - repository-contract: - name: Repository contract - runs-on: ubuntu-24.04 - timeout-minutes: 15 - steps: - - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 - with: - fetch-depth: 0 - - run: ./.golib/scripts/repository-check.sh - - quality: - name: Quality / ${{ matrix.directory }} - needs: prepare - runs-on: ubuntu-24.04 - timeout-minutes: 360 - strategy: - fail-fast: false - max-parallel: 8 - matrix: - include: ${{ fromJSON(needs.prepare.outputs.matrix) }} - steps: - - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 - with: - fetch-depth: 0 - - uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6 - with: - go-version-file: .go-version - cache: false - - name: Set up pinned ripgrep - env: - RIPGREP_VERSION: 15.2.0 - RIPGREP_SHA256: 33e15bcf1624b25cdd2a55813a47a2f95dbe126268203e76aa6a585d1e7b149c - run: | - set -euo pipefail - archive="${RUNNER_TEMP}/ripgrep.tar.gz" - root="${RUNNER_TEMP}/ripgrep" - curl --fail --silent --show-error --location \ - --retry 5 --retry-delay 2 --retry-all-errors \ - "https://github.com/BurntSushi/ripgrep/releases/download/${RIPGREP_VERSION}/ripgrep-${RIPGREP_VERSION}-x86_64-unknown-linux-musl.tar.gz" \ - --output "${archive}" - printf '%s %s\n' "${RIPGREP_SHA256}" "${archive}" | sha256sum --check - - mkdir -p "${root}" - tar --extract --gzip --file "${archive}" --directory "${root}" \ - --strip-components 1 - echo "${root}" >>"${GITHUB_PATH}" - "${root}/rg" --version | grep -Eq "^ripgrep ${RIPGREP_VERSION}( |$)" - - name: Set up Node - if: github.repository == 'faustbrian/go-ecma-regexp' || github.repository == 'faustbrian/go-queue-control-plane' - uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6 - with: - node-version: '24.4.1' - package-manager-cache: false - - name: Set up Deno - if: github.repository == 'faustbrian/go-ecma-regexp' - uses: denoland/setup-deno@22d081ff2d3a40755e97629de92e3bcbfa7cf2ed # v2.0.5 - with: - deno-version: '2.9.4' - cache: false - - name: Set up CLI shell runtime - if: github.repository == 'faustbrian/go-cli' - env: - ZSH_DEB_SHA256: bd5cc8dd3a01a6db38c0a815d75202c356a9c7f378674ba7bed9bc86dcba8af0 - run: | - set -euo pipefail - archive="${RUNNER_TEMP}/zsh.deb" - root="${RUNNER_TEMP}/zsh-runtime" - curl --fail --silent --show-error --location \ - --retry 5 --retry-delay 2 --retry-all-errors \ - 'https://archive.ubuntu.com/ubuntu/pool/main/z/zsh/zsh_5.9-6ubuntu2_amd64.deb' \ - --output "${archive}" - printf '%s %s\n' "${ZSH_DEB_SHA256}" "${archive}" | sha256sum --check - - mkdir -p "${root}" - dpkg-deb --extract "${archive}" "${root}" - echo "${root}/bin" >>"${GITHUB_PATH}" - "${root}/bin/zsh" --version | grep -Eq '^zsh 5\.9 ' - - name: Set up unpublished module proxy - if: vars.GOLIB_BOOTSTRAP_PROXY_URL != '' - env: - PROXY_URL: ${{ vars.GOLIB_BOOTSTRAP_PROXY_URL }} - PROXY_SHA256: ${{ vars.GOLIB_BOOTSTRAP_PROXY_SHA256 }} - run: | - set -euo pipefail - [[ "${PROXY_SHA256}" =~ ^[0-9a-f]{64}$ ]] - archive="${RUNNER_TEMP}/golib-proxy.tar.gz" - proxy="${RUNNER_TEMP}/golib-proxy" - curl --fail --silent --show-error --location \ - --retry 5 --retry-delay 2 --retry-all-errors \ - "${PROXY_URL}" --output "${archive}" - printf '%s %s\n' "${PROXY_SHA256}" "${archive}" | sha256sum --check - - mkdir -p "${proxy}" - tar --extract --gzip --file "${archive}" --directory "${proxy}" - echo "GOPROXY=file://${proxy},https://proxy.golang.org,direct" >>"${GITHUB_ENV}" - echo 'GONOSUMDB=github.com/faustbrian/go-*' >>"${GITHUB_ENV}" - - name: Restore content-addressed mutation evidence - if: inputs.release_dry_run != true - env: - GH_TOKEN: ${{ github.token }} - GITHUB_REPOSITORY_ID: ${{ github.repository_id }} - run: | - set -euo pipefail - seed='.golib/mutation-bootstrap/${{ matrix.artifact }}.zip' - if [[ -s "${seed}" ]]; then - ./.golib/scripts/restore-ci-mutation-evidence.sh \ - '${{ matrix.directory }}' "${seed}" - else - ./.golib/scripts/restore-ci-mutation-evidence.sh \ - '${{ matrix.directory }}' - fi - - name: Run strict module contract - id: strict_contract - if: inputs.release_dry_run != true - run: ./.golib/scripts/with-disposable-go-cache.sh ./.golib/scripts/run-modules.sh check --modules '${{ matrix.directory }}' - - name: Run release dry-run - id: release_dry_run - if: inputs.release_dry_run == true - env: - GOLIB_VERIFICATION_SNAPSHOT: '1' - run: | - set -euo pipefail - output='.artifacts/release-dry-run.log' - if [[ '${{ matrix.directory }}' != '.' ]]; then - output='.artifacts/${{ matrix.directory }}/release-dry-run.log' - fi - mkdir -p "$(dirname "${output}")" - ./.golib/scripts/with-disposable-go-cache.sh \ - ./.golib/scripts/run-modules.sh release-dry-run \ - --modules '${{ matrix.directory }}' 2>&1 | tee "${output}" - - name: Stage attributable evidence - if: always() - env: - CONTRACT_OUTCOME: ${{ inputs.release_dry_run == true && steps.release_dry_run.outcome || steps.strict_contract.outcome }} - run: >- - ./.golib/scripts/stage-ci-evidence.sh '${{ matrix.directory }}' - '${{ format('{0}/golib-evidence-{1}', runner.temp, matrix.artifact) }}' - "${CONTRACT_OUTCOME}" - - name: Upload attributable evidence - if: always() - uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 - with: - name: evidence-${{ matrix.artifact }} - path: ${{ format('{0}/golib-evidence-{1}', runner.temp, matrix.artifact) }} - if-no-files-found: error - include-hidden-files: true - retention-days: 30 - - codeql: - name: CodeQL - runs-on: ubuntu-24.04 - timeout-minutes: 120 - permissions: - contents: read - packages: read - security-events: write - steps: - - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 - with: - fetch-depth: 0 - - uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6 - with: - go-version-file: .go-version - cache: false - - name: Set up unpublished module proxy - if: vars.GOLIB_BOOTSTRAP_PROXY_URL != '' - env: - PROXY_URL: ${{ vars.GOLIB_BOOTSTRAP_PROXY_URL }} - PROXY_SHA256: ${{ vars.GOLIB_BOOTSTRAP_PROXY_SHA256 }} - run: | - set -euo pipefail - [[ "${PROXY_SHA256}" =~ ^[0-9a-f]{64}$ ]] - archive="${RUNNER_TEMP}/golib-proxy.tar.gz" - proxy="${RUNNER_TEMP}/golib-proxy" - curl --fail --silent --show-error --location \ - --retry 5 --retry-delay 2 --retry-all-errors \ - "${PROXY_URL}" --output "${archive}" - printf '%s %s\n' "${PROXY_SHA256}" "${archive}" | sha256sum --check - - mkdir -p "${proxy}" - tar --extract --gzip --file "${archive}" --directory "${proxy}" - echo "GOPROXY=file://${proxy},https://proxy.golang.org,direct" >>"${GITHUB_ENV}" - echo 'GONOSUMDB=github.com/faustbrian/go-*' >>"${GITHUB_ENV}" - - uses: github/codeql-action/init@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4 - with: - languages: go - queries: security-extended,security-and-quality - - run: ./.golib/scripts/with-disposable-go-cache.sh ./.golib/scripts/codeql-build.sh - - uses: github/codeql-action/analyze@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4 + ci: + uses: faustbrian/go-library-tools/.github/workflows/library-ci.yml@6028deef86539aa26fcd776e932f9f50ab4d8ad9 # v1.0.4 + with: + tooling_sha: 6028deef86539aa26fcd776e932f9f50ab4d8ad9 + release_dry_run: ${{ inputs.release_dry_run || false }} required: name: Required if: always() - needs: [prepare, repository-contract, quality, codeql] + needs: ci runs-on: ubuntu-24.04 timeout-minutes: 5 steps: - - name: Require every repository result + - name: Require shared workflow env: - PREPARE_RESULT: ${{ needs.prepare.result }} - CONTRACT_RESULT: ${{ needs.repository-contract.result }} - QUALITY_RESULT: ${{ needs.quality.result }} - CODEQL_RESULT: ${{ needs.codeql.result }} - run: | - set -euo pipefail - for result in \ - "${PREPARE_RESULT}" \ - "${CONTRACT_RESULT}" \ - "${QUALITY_RESULT}" \ - "${CODEQL_RESULT}"; do - [[ "${result}" == success ]] - done + CI_RESULT: ${{ needs.ci.result }} + run: test "${CI_RESULT}" = success diff --git a/.golib.yaml b/.golib.yaml new file mode 100644 index 0000000..013585d --- /dev/null +++ b/.golib.yaml @@ -0,0 +1,65 @@ +schema_version: 1 +tool_version: v1.0.4 +tool_checksums_sha256: 6c1ee70e3fa7aa333d12420dc78d7740f4de78717d1979eeb8a5f70a2a124ba0 +manifest: + modules: modules.json + packages: packages.json +evidence: + root: .verification +mutation: + root: .verification/mutation + imports: + - module: . + archive: .verification/mutation/bootstrap/root.zip + ledger: .verification/mutation/migration-ledger.json +api: + baselines: + - module: . + mode: apidiff + path: api/baseline.txt +operations: +- module: . + gate: test + steps: + - type: go-test + packages: ["./..."] + run: '^(TestContentionElectsExactlyOneOwner|TestAcquirePropagatesBackendAndCancellation|TestHandleLifecycleFailsClosed|TestHandleRejectsConcurrentOperationsWithoutBlockingState|TestHandleFailsClosedWhenDeadlinePassesDuringOperation|TestManagedRenewalReportsUncertaintyAndStopsAdmission|TestManagedStopHonorsCallerDeadline|TestStaleOwnerCannotAffectSuccessor|TestAcquireRacingShutdownReleasesReservation|TestManagerBoundsHandlesAndReleasesOnShutdown|TestWorkerCancelsOwnershipSensitiveJobOnLoss|TestCoordinatorCancelsOwnershipSensitiveTaskOnLoss)$' + count: 25 + timeout: 20m + - type: make + makefile: verification/package.mk + target: fence-mutations + timeout: 20m +- module: . + gate: docs + steps: + - type: make + makefile: verification/package.mk + target: docs + timeout: 20m +- module: . + gate: fuzz + steps: + - type: go-test + packages: ["."] + fuzz: '^FuzzKeyParsing$' + budget: 10000x + timeout: 2m + - type: go-test + packages: ["."] + fuzz: '^FuzzPolicyBounds$' + budget: 10000x + timeout: 2m + - type: go-test + packages: ["./memory"] + fuzz: '^FuzzLeaseStateModel$' + budget: 10000x + timeout: 2m +- module: . + gate: benchmark + steps: + - type: go-test + packages: ["./..."] + benchmark: "." + budget: 100ms + timeout: 20m diff --git a/.golib/documentation-tools.env b/.golib/documentation-tools.env deleted file mode 100644 index f58285a..0000000 --- a/.golib/documentation-tools.env +++ /dev/null @@ -1,6 +0,0 @@ -CSPELL_VERSION=10.0.0 -LYCHEE_VERSION=0.24.2 -LYCHEE_SHA256_DARWIN_ARM64=c9d3740ea2d891854d37116c9fba840f37b6e7c89d330e7db84ac333631c4977 -LYCHEE_SHA256_DARWIN_AMD64=887503a9cff667d322b8d0892b40bf49976eb9507af8483220a3706cdad55978 -LYCHEE_SHA256_LINUX_ARM64=91a7bd65685da41b90ccb9bc867a3d649a7818042dae04ff405e55a25bddee4c -LYCHEE_SHA256_LINUX_AMD64=1f4e0ef7f6554a6ed33dd7ac144fb2e1bbed98598e7af973042fc5cd43951c9a diff --git a/.golib/package.mk b/.golib/package.mk deleted file mode 100644 index 0bbb36e..0000000 --- a/.golib/package.mk +++ /dev/null @@ -1,75 +0,0 @@ -GO ?= go -GOLANGCI_LINT ?= golangci-lint -ACTIONLINT ?= $(GO) run github.com/rhysd/actionlint/cmd/actionlint@v1.7.12 -FUZZ_TIME ?= 2s -BENCH_TIME ?= 100ms -GREMLINS_VERSION ?= v0.6.0 -NILAWAY_VERSION ?= v0.0.0-20260710181136-2378218750e4 -STATICCHECK_VERSION ?= v0.8.0-rc.1 - -.PHONY: api-compat backend-hardening benchmark check coverage docs format \ - format-check fuzz integration lint mutation nilaway race staticcheck stress \ - test vet vuln workflows - -format: - gofmt -w . - -format-check: - test -z "$$(gofmt -l .)" - -test: - $(GO) test ./... - -race: - $(GO) test -race ./... - -stress: - ./scripts/check-stress.sh - -coverage: - ./scripts/check-coverage.sh - -vet: - $(GO) vet ./... - -staticcheck: - $(GO) run honnef.co/go/tools/cmd/staticcheck@$(STATICCHECK_VERSION) ./... - -lint: - $(GOLANGCI_LINT) run --timeout=5m - -nilaway: - -$(GO) run go.uber.org/nilaway/cmd/nilaway@$(NILAWAY_VERSION) \ - -include-pkgs='github.com/faustbrian/go-lease' ./... - -fuzz: - ./scripts/check-fuzz.sh "$(FUZZ_TIME)" - -mutation: - $(GO) run github.com/go-gremlins/gremlins/cmd/gremlins@$(GREMLINS_VERSION) \ - unleash --workers 2 --timeout-coefficient 10 \ - --threshold-efficacy 100 --threshold-mcover 100 ./memory - ./scripts/check-fence-mutations.sh - -benchmark: - $(GO) test ./... -run '^$$' -bench . -benchmem -benchtime="$(BENCH_TIME)" - -integration: - $(GO) test -race -count=1 -timeout=15m ./postgres ./valkey - -backend-hardening: - ./scripts/check-backend-faults.sh - -docs: - ./scripts/check-docs.sh - -api-compat: - ./scripts/check-api-compat.sh - -vuln: - $(GO) run golang.org/x/vuln/cmd/govulncheck@v1.6.0 ./... - -workflows: - $(ACTIONLINT) .github/workflows/*.yml - -check: format-check vet test race stress coverage fuzz benchmark docs api-compat diff --git a/.golib/scripts/audit-goals.sh b/.golib/scripts/audit-goals.sh deleted file mode 100755 index 4f6b24e..0000000 --- a/.golib/scripts/audit-goals.sh +++ /dev/null @@ -1,109 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 1 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -module="$1" -artifact="${root}/.artifacts/${module}" -output="${artifact}/goal-traceability.json" -temporary="${output}.tmp.$$" -gate_records="$(mktemp "${TMPDIR:-/tmp}/golib-goal-gates.XXXXXX")" -cleanup() { - rm -f "${temporary}" "${gate_records}" -} -trap cleanup EXIT HUP INT TERM - -module_record="$( - jq -e -c --arg directory "${module}" \ - '.modules[] | select(.directory == $directory)' \ - "${root}/modules.json" -)" || { - printf 'module is absent from modules.json: %s\n' "${module}" >&2 - exit 2 -} -goal_count="$(jq '.goal_evidence | length' <<<"${module_record}")" -mkdir -p "${artifact}" - -if [[ "${goal_count}" -gt 0 ]]; then - while IFS= read -r gate; do - [[ -n "${gate}" ]] || continue - if ! jq -e --arg gate "${gate}" ' - [ - .goal_evidence[] - | select(.implementation_status != "future-not-started") - | .verification_gates[] - ] - | index($gate) != null - ' <<<"${module_record}" >/dev/null; then - continue - fi - "${root}/.golib/scripts/verify-gate-evidence.sh" "${module}" "${gate}" - input_digest="$( - "${root}/.golib/scripts/gate-input-digest.sh" "${gate}" "${module}" - )" - evidence="${artifact}/evidence/by-input/${gate}/${input_digest}.json" - if [[ ! -f "${evidence}" ]]; then - evidence="${artifact}/evidence/${gate}.json" - fi - jq -c '{ - gate, - result, - input_digest, - execution_revision, - completed_at, - revalidated_revision, - revalidated_at - }' "${evidence}" >>"${gate_records}" - done <"${root}/.golib/scripts/check-gates.txt" -fi - -verified_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" -revision="$(git rev-parse HEAD)" -manifest_sha256="$(shasum -a 256 "${root}/modules.json" | awk '{print $1}')" -jq -n \ - --argjson module "${module_record}" \ - --slurpfile gates "${gate_records}" \ - --arg verified_at "${verified_at}" \ - --arg revision "${revision}" \ - --arg manifest_sha256 "${manifest_sha256}" \ - '{ - schema_version: 1, - module: $module.directory, - module_path: $module.module_path, - requirements_manifest_sha256: $manifest_sha256, - implementation_status: $module.goal_status, - verification_status: ( - if ($module.goal_evidence | length) == 0 - then "not-applicable" - elif ([ - $module.goal_evidence[] - | select(.implementation_status != "future-not-started") - ] | length) == 0 - then "deferred" - else "verified" - end - ), - verified_revision: $revision, - verified_at: $verified_at, - goals: [ - $module.goal_evidence[] | - . + { - verification_status: ( - if .implementation_status == "future-not-started" - then "deferred" - else "verified" - end - ) - } - ], - gate_evidence: $gates - }' >"${temporary}" -mv "${temporary}" "${output}" -printf '[%s] goal traceability: %s (%s goals)\n' \ - "${module}" \ - "$(jq -r '.verification_status' "${output}")" \ - "${goal_count}" diff --git a/.golib/scripts/build-golib-gremlins.sh b/.golib/scripts/build-golib-gremlins.sh deleted file mode 100755 index 37179a3..0000000 --- a/.golib/scripts/build-golib-gremlins.sh +++ /dev/null @@ -1,69 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -root="$(git rev-parse --show-toplevel)" - -# Resolved from the verified repository root rather than the caller's cwd. -# shellcheck disable=SC1091 -source "${root}/.golib/versions.env" - -semantic_patch="${root}/.golib/scripts/patches/gremlins-run-all-mutants.patch" -coverage_patch="${root}/.golib/scripts/patches/gremlins-shared-coverage.patch" -diff_patch="${root}/.golib/scripts/patches/gremlins-module-relative-diff.patch" -verifier_identity="$("${root}/.golib/scripts/mutation-verifier-identity.sh")" -platform_identity="$(go env GOOS GOARCH | paste -sd- -)" -artifact="${root}/.artifacts/tooling/gremlins-${verifier_identity}-${platform_identity}" -binary="${artifact}/golib-gremlins" -lock="${artifact}.lock" - -if [[ -x "${binary}" ]]; then - printf '%s\n' "${binary}" - exit 0 -fi - -mkdir -p "$(dirname "${artifact}")" -attempts=0 -while ! mkdir "${lock}" 2>/dev/null; do - if [[ -x "${binary}" ]]; then - printf '%s\n' "${binary}" - exit 0 - fi - attempts=$((attempts + 1)) - if [[ "${attempts}" -ge 120 ]]; then - printf 'timed out waiting for golib-gremlins build lock: %s\n' \ - "${lock}" >&2 - exit 1 - fi - sleep 1 -done -temporary="$(mktemp -d "${TMPDIR:-/tmp}/golib-gremlins.XXXXXX")" -cleanup() { - rm -rf "${temporary}" "${lock}" -} -trap cleanup EXIT HUP INT TERM - -download="$(GOWORK=off go mod download -json \ - "github.com/go-gremlins/gremlins@${GREMLINS_VERSION}")" -source_directory="$(jq -er '.Dir' <<<"${download}")" -jq -e \ - --arg sum "${GREMLINS_SUM}" \ - --arg gomod_sum "${GREMLINS_GOMOD_SUM}" ' - .Sum == $sum and .GoModSum == $gomod_sum - ' <<<"${download}" >/dev/null - -cp -R "${source_directory}" "${temporary}/source" -chmod -R u+w "${temporary}/source" -patch --batch --forward -d "${temporary}/source" -p1 <"${semantic_patch}" >&2 -patch --batch --forward -d "${temporary}/source" -p1 <"${coverage_patch}" >&2 -patch --batch --forward -d "${temporary}/source" -p1 <"${diff_patch}" >&2 -( - cd "${temporary}/source" - GOWORK=off go build -trimpath -buildvcs=false \ - -o "${temporary}/golib-gremlins" ./cmd/gremlins -) - -rm -rf "${artifact}" -mkdir "${artifact}" -mv "${temporary}/golib-gremlins" "${binary}" -chmod 0755 "${binary}" -printf '%s\n' "${binary}" diff --git a/.golib/scripts/build-local-proxy.sh b/.golib/scripts/build-local-proxy.sh deleted file mode 100755 index 40a1b34..0000000 --- a/.golib/scripts/build-local-proxy.sh +++ /dev/null @@ -1,180 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -lt 1 || $# -gt 3 ]]; then - printf 'usage: %s [version] [module-directories]\n' \ - "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -output="$1" -version="${2:-v1.0.0}" -selected="${3:-}" - -if [[ ! "${version}" =~ ^v(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)$ ]]; then - printf 'local proxy version must be canonical semantic version: %s\n' \ - "${version}" >&2 - exit 2 -fi -if [[ -e "${output}" && ! -d "${output}" ]]; then - printf 'local proxy output is not a directory: %s\n' "${output}" >&2 - exit 2 -fi - -mkdir -p "${output}" -if find "${output}" -mindepth 1 -print -quit | grep -q .; then - printf 'local proxy output must be empty: %s\n' "${output}" >&2 - exit 2 -fi - -temporary="$(mktemp -d "${TMPDIR:-/tmp}/golib-local-proxy.XXXXXX")" -mkdir -p "${temporary}/proxy" -cleanup() { - rm -rf "${temporary}" -} -trap cleanup EXIT HUP INT TERM - -rewrite_owned_dependencies() { - GOLIB_PROXY_VERSION="${version}" perl -pi -e \ - 's#(github\.com/faustbrian/go-[a-z0-9/-]+) v0\.0\.0(?:-[0-9]{14}-[0-9a-f]{12})?#$1 $ENV{GOLIB_PROXY_VERSION}#g' \ - "$1" -} - -selection_file="${temporary}/selected-modules" -jq -r --arg selected "${selected}" ' - . as $catalog - | def closure($directories): - ([ - $catalog.modules[] - | select(.directory as $directory | $directories | index($directory)) - | .owned_dependencies[] - ] | unique) as $dependencies - | ([ - $catalog.modules[] - | select(.module_path as $path | $dependencies | index($path)) - | .directory - ] + $directories | unique) as $next - | if $next == $directories then $next else closure($next) end; - ( - if $selected == "" or ($selected | split(",") | index(".") != null) - then [.modules[] | select(.releasable == true) | .directory] - else - ($selected | split(",")) as $selected_directories - | ([ - .modules[] - | select( - .directory as $directory - | any( - $selected_directories[]; - . as $selected_directory - | $directory == $selected_directory or - ($directory | startswith($selected_directory + "/")) - ) - ) - | .directory - ]) as $expanded - | closure($expanded) - end - ) as $directories - | .modules[] - | select( - .releasable == true - and (.directory as $directory | $directories | index($directory)) - ) - | [.module_path, .directory] - | @tsv -' "${root}/modules.json" >"${selection_file}" -if [[ ! -s "${selection_file}" ]]; then - if [[ -n "${selected}" ]] && jq -e --arg selected "${selected}" ' - . as $catalog - | ($selected | split(",")) as $directories - | all( - $directories[]; - . as $directory - | any($catalog.modules[]; .directory == $directory) - ) - ' "${root}/modules.json" >/dev/null; then - exit 0 - fi - printf 'local proxy module selection is empty: %s\n' "${selected}" >&2 - exit 1 -fi - -while IFS=$'\t' read -r module_path module_directory; do - [[ -n "${module_path}" && -n "${module_directory}" ]] || continue - if [[ "${module_path}" =~ [A-Z] ]]; then - printf 'local proxy does not support unescaped uppercase module paths: %s\n' \ - "${module_path}" >&2 - exit 1 - fi - - proxy_directory="${temporary}/proxy/${module_path}/@v" - archive_directory="${temporary}/archive" - archive_root="${archive_directory}/${module_path}@${version}" - zip_file="${proxy_directory}/${version}.zip" - nested_modules="${temporary}/nested-modules" - archive_files="${temporary}/archive-files" - archive_tar="${temporary}/archive.tar" - mkdir -p "${proxy_directory}" "${archive_root}" - jq -r --arg current "${module_directory}" ' - .modules[] - | .directory - | select( - ($current == "." and . != ".") or - ($current != "." and startswith($current + "/")) - ) - ' "${root}/modules.json" >"${nested_modules}" - - cp "${root}/${module_directory}/go.mod" \ - "${proxy_directory}/${version}.mod" - rewrite_owned_dependencies "${proxy_directory}/${version}.mod" - printf '{"Version":"%s","Time":"2000-01-01T00:00:00Z"}\n' \ - "${version}" >"${proxy_directory}/${version}.info" - printf '%s\n' "${version}" >"${proxy_directory}/list" - - : >"${archive_files}" - while IFS= read -r -d '' source; do - if [[ ! -e "${root}/${source}" ]]; then - continue - fi - relative="${source#"${module_directory}/"}" - if [[ "${relative}" == ".golib" || "${relative}" == ".golib/"* ]]; then - continue - fi - nested=0 - while IFS= read -r nested_module; do - [[ -n "${nested_module}" ]] || continue - if [[ "${source}" == "${nested_module}" || - "${source}" == "${nested_module}/"* ]]; then - nested=1 - break - fi - done <"${nested_modules}" - [[ "${nested}" -eq 0 ]] || continue - if [[ -L "${root}/${source}" ]]; then - continue - fi - printf '%s\0' "${relative}" >>"${archive_files}" - done < <( - git -C "${root}" ls-files -z --cached -- "${module_directory}" - ) - ( - cd "${root}/${module_directory}" - tar --null -cf "${archive_tar}" -T "${archive_files}" - ) - tar -xf "${archive_tar}" -C "${archive_root}" - rm -f "${archive_tar}" - rewrite_owned_dependencies "${archive_root}/go.mod" - - find "${archive_directory}" -exec touch -t 200001010000 {} + - ( - cd "${archive_directory}" - LC_ALL=C find "${module_path}@${version}" -type f -print | - LC_ALL=C sort | - zip -q -X "${zip_file}" -@ - ) - rm -rf "${archive_directory}" -done <"${selection_file}" - -cp -R "${temporary}/proxy/." "${output}/" diff --git a/.golib/scripts/check-api-baseline.sh b/.golib/scripts/check-api-baseline.sh deleted file mode 100755 index 9eccb14..0000000 --- a/.golib/scripts/check-api-baseline.sh +++ /dev/null @@ -1,66 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 1 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -# shellcheck disable=SC1091 # Canonical repository tool versions. -source "${root}/.golib/versions.env" -case "$1" in - /*) directory="$1" ;; - *) directory="${root}/$1" ;; -esac -if [[ ! -f "${directory}/go.mod" ]]; then - printf 'module has no go.mod: %s\n' "$1" >&2 - exit 1 -fi - -baseline="${directory}/api/baseline.txt" -if [[ ! -s "${baseline}" ]]; then - printf 'missing API baseline: %s\n' "${baseline}" >&2 - exit 1 -fi -module_path="$( - cd "${directory}" - GOWORK=off go mod edit -json | jq -er '.Module.Path' -)" -current="$(mktemp "${TMPDIR:-/tmp}/golib-api-current.XXXXXX")" -report="$(mktemp "${TMPDIR:-/tmp}/golib-api-report.XXXXXX")" -cleanup() { - rm -f "${current}" "${report}" -} -trap cleanup EXIT HUP INT TERM - -run_apidiff() { - if [[ -n "${GOLIB_APIDIFF:-}" ]]; then - GOWORK=off go exec-tool "${GOLIB_APIDIFF}" "$@" - else - GOWORK=off go run \ - "golang.org/x/exp/cmd/apidiff@${APIDIFF_VERSION}" "$@" - fi -} - -( - cd "${directory}" - run_apidiff -m -w "${current}" "${module_path}" -) -set +e -( - cd "${directory}" - run_apidiff -m -incompatible "${baseline}" "${current}" >"${report}" -) -status=$? -set -e -if [[ -s "${report}" ]]; then - printf 'incompatible exported API changes in %s:\n' \ - "${module_path}" >&2 - cat "${report}" >&2 - exit 1 -fi -if [[ "${status}" -ne 0 ]]; then - printf 'API compatibility tool failed for %s\n' "${module_path}" >&2 - exit "${status}" -fi diff --git a/.golib/scripts/check-coverage.sh b/.golib/scripts/check-coverage.sh deleted file mode 100755 index 02ed3d2..0000000 --- a/.golib/scripts/check-coverage.sh +++ /dev/null @@ -1,133 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 1 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -module="$1" -directory="${root}/${module}" -artifact="${root}/.artifacts/${module}" -profile="${artifact}/coverage.out" -packages="${artifact}/coverage-packages.txt" -report="${artifact}/coverage-report.txt" -identity="${artifact}/coverage-profile.json" -mkdir -p "${artifact}" -rm -f "${identity}" - -cd "${directory}" -jq -r --arg directory "${module}" ' - .modules[] - | select(.directory == $directory) - | .packages[] - | select(.coverage_required == true) - | .import_path -' "${root}/modules.json" | sort -u >"${packages}" -[[ -s "${packages}" ]] || { - printf 'coverage expected packages are empty for %s\n' "${module}" >&2 - exit 1 -} - -tags="$(jq -r --arg directory "${module}" \ - '.modules[] | select(.directory == $directory) | .test_tags | join(",")' \ - "${root}/modules.json")" -test_arguments=( - ./... - -count=1 - -timeout=20m - -covermode=atomic - -coverpkg=./... - -coverprofile="${profile}" -) -if [[ -n "${tags}" ]]; then - test_arguments=(-tags="${tags}" "${test_arguments[@]}") -fi -if [[ "${module}" == "pkg/xsd" ]]; then - xsts_work="${artifact}/xsts" - mkdir -p "${xsts_work}" - XSTS_ROOT="$(./scripts/prepare-xsts.sh "${xsts_work}")" - export XSTS_ROOT -fi -input_digest="$( - "${root}/.golib/scripts/gate-input-digest.sh" coverage "${module}" -)" -started="$(date +%s)" -GOWORK=off go test "${test_arguments[@]}" -finished="$(date +%s)" -elapsed=$((finished - started)) -if [[ "${elapsed}" -lt 1 ]]; then - elapsed=1 -fi -[[ -s "${profile}" ]] || { - printf 'coverage profile is missing for %s\n' "${module}" >&2 - exit 1 -} - -set +e -awk ' - FNR == NR { expected[$1] = 1; next } - FNR == 1 { next } - { - split($1, location, ":") - package = location[1] - sub("/[^/]+$", "", package) - block = $1 - blockPackage[block] = package - blockStatements[block] = $(NF - 1) + 0 - if (($NF + 0) > 0) blockCovered[block] = 1 - } - END { - failed = 0 - for (block in blockStatements) { - package = blockPackage[block] - total[package] += blockStatements[block] - if (blockCovered[block]) covered[package] += blockStatements[block] - } - for (package in expected) { - if (!(package in total) || total[package] == 0) { - printf "%s missing executable coverage evidence\n", package - failed = 1 - continue - } - printf "%s %d/%d statements\n", package, covered[package], total[package] - if (covered[package] != total[package]) failed = 1 - } - exit failed - } -' "${packages}" "${profile}" | sort >"${report}" -coverage_status="${PIPESTATUS[0]}" -set -e - -cat "${report}" -if [[ "${coverage_status}" -eq 0 ]]; then - printf 'all production packages have exact 100%% statement coverage\n' -else - printf 'one or more production packages are below exact 100%% coverage\n' >&2 - exit 1 -fi - -profile_sha256="$(shasum -a 256 "${profile}" | awk '{print $1}')" -completed_input_digest="$( - "${root}/.golib/scripts/gate-input-digest.sh" coverage "${module}" -)" -if [[ "${completed_input_digest}" != "${input_digest}" ]]; then - printf 'coverage inputs changed during execution for %s\n' "${module}" >&2 - exit 1 -fi -identity_tmp="$(mktemp "${identity}.tmp.XXXXXX")" -jq -n \ - --arg input_digest "${input_digest}" \ - --arg test_tags "${tags}" \ - --arg profile_sha256 "${profile_sha256}" \ - --arg elapsed "${elapsed}s" ' - { - schema_version: 1, - input_digest: $input_digest, - test_tags: $test_tags, - profile_sha256: $profile_sha256, - elapsed: $elapsed - } - ' >"${identity_tmp}" -mv "${identity_tmp}" "${identity}" diff --git a/.golib/scripts/check-documentation.sh b/.golib/scripts/check-documentation.sh deleted file mode 100755 index 4eb2e05..0000000 --- a/.golib/scripts/check-documentation.sh +++ /dev/null @@ -1,107 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -root="$(git rev-parse --show-toplevel)" -# shellcheck source=/dev/null -source "${root}/.golib/documentation-tools.env" - -tool_directory="$(mktemp -d "${TMPDIR:-/tmp}/golib-documentation-tools.XXXXXX")" - -cleanup() { - chmod -R u+w "${tool_directory}" 2>/dev/null || true - find "${tool_directory}" -depth -delete 2>/dev/null || true -} -trap cleanup EXIT HUP INT TERM - -cd "${root}" -test -s README.md - -configured_cspell_version="$(jq -r '.devDependencies.cspell // empty' package.json)" -if [[ "${configured_cspell_version}" != "${CSPELL_VERSION}" ]]; then - printf 'cspell version mismatch: package.json=%s versions.env=%s\n' \ - "${configured_cspell_version}" "${CSPELL_VERSION}" >&2 - exit 1 -fi - -cspell="${GOLIB_CSPELL:-}" -if [[ -z "${cspell}" ]]; then - command -v npm >/dev/null || { - printf 'npm is required for the documentation spelling gate\n' >&2 - exit 1 - } - node_directory="${tool_directory}/node" - npm_cache="${tool_directory}/npm-cache" - mkdir -p "${node_directory}" "${npm_cache}" - cp package.json package-lock.json "${node_directory}/" - ( - cd "${node_directory}" - NPM_CONFIG_CACHE="${npm_cache}" npm ci \ - --ignore-scripts \ - --no-audit \ - --no-fund \ - --silent - ) - cspell="${node_directory}/node_modules/.bin/cspell" -fi -[[ -x "${cspell}" ]] || { - printf 'cspell executable is unavailable: %s\n' "${cspell}" >&2 - exit 1 -} -"${cspell}" lint \ - --config cspell.json \ - --no-config-search \ - --validate-directives \ - --no-progress \ - --no-summary \ - README.md 'docs/**/*.md' - -lychee="${GOLIB_LYCHEE:-}" -if [[ -z "${lychee}" ]]; then - case "$(uname -s)/$(uname -m)" in - Darwin/arm64) - target="aarch64-apple-darwin" - checksum="${LYCHEE_SHA256_DARWIN_ARM64}" - ;; - Darwin/x86_64) - target="x86_64-apple-darwin" - checksum="${LYCHEE_SHA256_DARWIN_AMD64}" - ;; - Linux/aarch64|Linux/arm64) - target="aarch64-unknown-linux-gnu" - checksum="${LYCHEE_SHA256_LINUX_ARM64}" - ;; - Linux/x86_64) - target="x86_64-unknown-linux-gnu" - checksum="${LYCHEE_SHA256_LINUX_AMD64}" - ;; - *) - printf 'unsupported lychee platform: %s/%s\n' "$(uname -s)" "$(uname -m)" >&2 - exit 1 - ;; - esac - archive="lychee-${target}.tar.gz" - archive_path="${tool_directory}/${archive}" - curl --fail --silent --show-error --location \ - "https://github.com/lycheeverse/lychee/releases/download/lychee-v${LYCHEE_VERSION}/${archive}" \ - --output "${archive_path}" - printf '%s %s\n' "${checksum}" "${archive_path}" | shasum -a 256 -c - - tar -xzf "${archive_path}" -C "${tool_directory}" - lychee="${tool_directory}/lychee-${target}/lychee" -fi -[[ -x "${lychee}" ]] || { - printf 'lychee executable is unavailable: %s\n' "${lychee}" >&2 - exit 1 -} -# golib-unpublished-pkg-go-dev: public-proxy checks own publication readiness. -"${lychee}" \ - --cache=false \ - --exclude '^https://pkg\.go\.dev/(badge/)?github\.com/faustbrian/go-' \ - --exclude '^https://doi\.org/10\.1145/190314\.190317$' \ - --exclude '^https://service\.unece\.org/trade/' \ - --exclude-private \ - --exclude-loopback \ - --exclude '^https://www\.iso\.org/standard/' \ - --max-concurrency 16 \ - --max-retries 3 \ - --no-progress \ - README.md 'docs/**/*.md' diff --git a/.golib/scripts/check-fuzz.sh b/.golib/scripts/check-fuzz.sh deleted file mode 100755 index 6d31e48..0000000 --- a/.golib/scripts/check-fuzz.sh +++ /dev/null @@ -1,61 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 1 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -module="$1" -fuzz_budget="${GOLIB_FUZZ_SMOKE_BUDGET:-10000x}" -if [[ "${module}" = /* ]]; then - directory="${module}" -else - directory="${root}/${module}" -fi -if [[ ! -f "${directory}/go.mod" ]]; then - printf 'module has no go.mod: %s\n' "${module}" >&2 - exit 1 -fi -targets="$(mktemp)" -trap 'rm -f "${targets}"' EXIT - -cd "${directory}" -while IFS= read -r -d '' file; do - parent="$(dirname "${file#./}")" - nested=0 - while [[ "${parent}" != "." ]]; do - if [[ -f "${parent}/go.mod" ]]; then - nested=1 - break - fi - parent="$(dirname "${parent}")" - done - [[ "${nested}" -eq 0 ]] || continue - names="$( - sed -nE \ - 's/^func (Fuzz[A-Za-z0-9_]+)\([A-Za-z_][A-Za-z0-9_]* \*testing\.F\).*/\1/p' \ - "${file}" - )" - while IFS= read -r target; do - [[ -n "${target}" ]] || continue - printf '%s %s\n' "${file}" "${target}" >>"${targets}" - done <<<"${names}" -done < <(find . -type f -name '*_test.go' -not -path './vendor/*' -print0) -sort -u -o "${targets}" "${targets}" - -count=0 -while read -r file target; do - [[ -n "${file:-}" ]] || continue - package="./$(dirname "${file#./}")" - [[ "${package}" != "./." ]] || package=. - GOWORK=off go test "${package}" -run '^$' -fuzz "^${target}$" \ - -fuzztime="${fuzz_budget}" -parallel=2 - count=$((count + 1)) -done <"${targets}" -if [[ "${count}" -eq 0 ]]; then - printf 'no fuzz targets were executed for %s\n' "${module}" >&2 - exit 1 -fi -printf 'executed %s registered fuzz targets\n' "${count}" diff --git a/.golib/scripts/check-gates.txt b/.golib/scripts/check-gates.txt deleted file mode 100644 index b4c532d..0000000 --- a/.golib/scripts/check-gates.txt +++ /dev/null @@ -1,21 +0,0 @@ -format-check -tidy-check -safety -vet -test -race -coverage -lint -staticcheck -vulnerability -secrets -licenses -sbom -fuzz -mutation -nilaway -docs -api -conformance -interoperability -benchmark diff --git a/.golib/scripts/check-go-safety.go b/.golib/scripts/check-go-safety.go deleted file mode 100644 index a41bec3..0000000 --- a/.golib/scripts/check-go-safety.go +++ /dev/null @@ -1,87 +0,0 @@ -package main - -import ( - "fmt" - "go/parser" - "go/token" - "io/fs" - "os" - "path/filepath" - "sort" - "strconv" - "strings" -) - -func main() { - if len(os.Args) != 2 { - fmt.Fprintln(os.Stderr, "usage: check-go-safety ") - os.Exit(2) - } - violations, err := scan(os.Args[1]) - if err != nil { - fmt.Fprintln(os.Stderr, err) - os.Exit(1) - } - for _, violation := range violations { - fmt.Fprintln(os.Stderr, violation) - } - if len(violations) != 0 { - os.Exit(1) - } -} - -func scan(directory string) ([]string, error) { - violations := make([]string, 0) - err := filepath.WalkDir(directory, func(path string, entry fs.DirEntry, walkErr error) error { - if walkErr != nil { - return walkErr - } - if entry.IsDir() { - if path != directory && excludedDirectory(entry.Name()) { - return filepath.SkipDir - } - return nil - } - if !strings.HasSuffix(path, ".go") || strings.HasSuffix(path, "_test.go") { - return nil - } - - fileSet := token.NewFileSet() - file, err := parser.ParseFile(fileSet, path, nil, parser.ParseComments) - if err != nil { - return fmt.Errorf("parse %s: %w", path, err) - } - for _, imported := range file.Imports { - name, err := strconv.Unquote(imported.Path.Value) - if err != nil { - return fmt.Errorf("parse import in %s: %w", path, err) - } - if name == "unsafe" || name == "C" { - violations = append(violations, fmt.Sprintf( - "%s:%d: forbidden import %q", - path, - fileSet.Position(imported.Pos()).Line, - name, - )) - } - } - for _, group := range file.Comments { - for _, comment := range group.List { - if strings.HasPrefix(strings.TrimSpace(comment.Text), "//go:linkname") { - violations = append(violations, fmt.Sprintf( - "%s:%d: forbidden go:linkname directive", - path, - fileSet.Position(comment.Pos()).Line, - )) - } - } - } - return nil - }) - sort.Strings(violations) - return violations, err -} - -func excludedDirectory(name string) bool { - return name == "vendor" || name == "testdata" || strings.HasPrefix(name, ".") -} diff --git a/.golib/scripts/check-go-safety.sh b/.golib/scripts/check-go-safety.sh deleted file mode 100755 index 840e13b..0000000 --- a/.golib/scripts/check-go-safety.sh +++ /dev/null @@ -1,14 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 1 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -module="$1" -directory="${root}/${module}" - -go run "${root}/.golib/scripts/check-go-safety.go" "${directory}" -printf 'standalone safety policy passed for %s\n' "${module}" diff --git a/.golib/scripts/check-module.sh b/.golib/scripts/check-module.sh deleted file mode 100755 index 84e01a4..0000000 --- a/.golib/scripts/check-module.sh +++ /dev/null @@ -1,516 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 2 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -module="$1" -gate="$2" -directory="${root}/${module}" -local_proxy_owned=0 -local_modcache_owned=0 -isolated_modfiles_owned=0 -GOLIB_GO_TOOL_PATH="" - -cleanup() { - if [[ "${local_proxy_owned}" -eq 1 ]]; then - rm -rf "${GOLIB_LOCAL_PROXY}" - fi - if [[ "${local_modcache_owned}" -eq 1 ]]; then - chmod -R u+w "${GOLIB_LOCAL_MODCACHE}" - rm -rf "${GOLIB_LOCAL_MODCACHE}" - fi - if [[ "${isolated_modfiles_owned}" -eq 1 ]]; then - rm -rf "${GOLIB_ISOLATED_MODFILES_DIRECTORY}" - fi -} -trap cleanup EXIT HUP INT TERM - -if ! jq -e --arg directory "${module}" \ - '.modules[] | select(.directory == $directory)' \ - "${root}/modules.json" >/dev/null; then - printf 'module is absent from modules.json: %s\n' "${module}" >&2 - exit 1 -fi - -if [[ ! -f "${directory}/go.mod" ]]; then - printf 'module has no go.mod: %s\n' "${module}" >&2 - exit 1 -fi - -enable_local_proxy() { - local no_sum_db upstream upstream_flags - - if [[ -z "${GOLIB_REAL_GO:-}" ]]; then - GOLIB_REAL_GO="$(command -v go)" - fi - export GOLIB_REAL_GO - upstream_flags="$( - printf '%s' "${GOLIB_UPSTREAM_GOFLAGS:-$("${GOLIB_REAL_GO}" env GOFLAGS)}" - )" - export GOLIB_UPSTREAM_GOFLAGS="${upstream_flags}" - - if [[ -z "${GOLIB_LOCAL_PROXY:-}" ]]; then - GOLIB_LOCAL_PROXY="$(mktemp -d "${TMPDIR:-/tmp}/golib-proxy.XXXXXX")" - local_proxy_owned=1 - "${root}/.golib/scripts/build-local-proxy.sh" \ - "${GOLIB_LOCAL_PROXY}" v1.0.0 "${module}" - fi - export GOLIB_LOCAL_PROXY - - upstream="${GOLIB_UPSTREAM_GOPROXY:-$(go env GOPROXY)}" - no_sum_db="$(go env GONOSUMDB)" - export GOPROXY="file://${GOLIB_LOCAL_PROXY},${upstream}" - export GONOSUMDB="github.com/faustbrian/go-*${no_sum_db:+,${no_sum_db}}" - if [[ -z "${GOLIB_LOCAL_MODCACHE:-}" ]]; then - GOLIB_LOCAL_MODCACHE="$( - mktemp -d "${TMPDIR:-/tmp}/golib-modcache.XXXXXX" - )" - local_modcache_owned=1 - fi - export GOLIB_LOCAL_MODCACHE - export GOMODCACHE="${GOLIB_LOCAL_MODCACHE}" - if [[ -z "${GOLIB_ISOLATED_MODFILES_DIRECTORY:-}" ]]; then - GOLIB_ISOLATED_MODFILES_DIRECTORY="$( - mktemp -d "${TMPDIR:-/tmp}/golib-modfiles.XXXXXX" - )" - isolated_modfiles_owned=1 - fi - export GOLIB_ISOLATED_MODFILES_DIRECTORY - mkdir -p "${GOLIB_ISOLATED_MODFILES_DIRECTORY}/bin" - ln -sf "${root}/.golib/scripts/internal/isolated-go.sh" \ - "${GOLIB_ISOLATED_MODFILES_DIRECTORY}/bin/go" - case ":${PATH}:" in - *":${GOLIB_ISOLATED_MODFILES_DIRECTORY}/bin:"*) ;; - *) - PATH="${GOLIB_ISOLATED_MODFILES_DIRECTORY}/bin:${PATH}" - export PATH - ;; - esac - export GOFLAGS="${upstream_flags}" -} - -isolated() { - enable_local_proxy - GOWORK=off "$@" -} - -install_go_tool() { - local package="$1" - local executable="$2" - local tool_directory - - enable_local_proxy - tool_directory="${GOLIB_ISOLATED_MODFILES_DIRECTORY}/tools" - mkdir -p "${tool_directory}" - GOBIN="${tool_directory}" GOWORK=off \ - GOFLAGS="${GOLIB_UPSTREAM_GOFLAGS}" \ - "${GOLIB_REAL_GO}" install "${package}" - GOLIB_GO_TOOL_PATH="${tool_directory}/${executable}" -} - -run_go_tool() { - local package="$1" - local executable="$2" - shift 2 - - install_go_tool "${package}" "${executable}" - isolated go exec-tool "${GOLIB_GO_TOOL_PATH}" "$@" -} - -refresh_owned_sums() { - [[ -f go.sum ]] || return 0 - local temporary - temporary="$(mktemp "${TMPDIR:-/tmp}/golib-go-sum.XXXXXX")" - awk '$1 !~ /^github\.com\/faustbrian\/go-/ { print }' \ - go.sum >"${temporary}" - if cmp -s go.sum "${temporary}"; then - rm -f "${temporary}" - else - mv "${temporary}" go.sum - fi -} - -applicable() { - jq -e --arg directory "${module}" --arg gate "$1" \ - '.modules[] | select(.directory == $directory) | .gates[$gate] == true' \ - "${root}/modules.json" >/dev/null -} - -package_makefile="Makefile" -if [[ "${module}" == "." && -f "${root}/.golib/package.mk" ]]; then - package_makefile="${root}/.golib/package.mk" -fi - -package_make() { - make -f "${package_makefile}" "$@" -} - -make_has_target() { - [[ -f "${package_makefile}" ]] && - grep -Eq "^$1([[:space:]]+[^:]*)?:" "${package_makefile}" -} - -find_make_target() { - local target - for target in "$@"; do - if make_has_target "${target}"; then - printf '%s\n' "${target}" - return 0 - fi - done - return 1 -} - -skip_not_applicable() { - printf '[%s] %s: not applicable by catalog policy\n' "${module}" "$1" -} - -test_tags() { - jq -r --arg directory "${module}" \ - '.modules[] | select(.directory == $directory) | .test_tags | join(",")' \ - "${root}/modules.json" -} - -interoperability_declared() { - jq -e --arg directory "${module}" \ - '.modules[] | select(.directory == $directory) | - .interoperability_tools | length > 0' \ - "${root}/modules.json" >/dev/null -} - -run_benchmark() { - local output temporary status target - enable_local_proxy - output="${root}/.artifacts/${module}/benchmark.txt" - temporary="${output}.tmp.$$" - mkdir -p "$(dirname "${output}")" - rm -f "${output}" "${temporary}" - - set +e - if target="$(find_make_target benchmark performance)"; then - package_make GOWORK=off "${target}" 2>&1 | tee "${temporary}" - status=${PIPESTATUS[0]} - else - GOWORK=off go test ./... -run '^$' -bench . -benchmem 2>&1 | - tee "${temporary}" - status=${PIPESTATUS[0]} - fi - set -e - - if [[ "${status}" -ne 0 ]]; then - rm -f "${temporary}" - return "${status}" - fi - if ! grep -Eq '^Benchmark[^[:space:]]*(-[0-9]+)?[[:space:]]+' \ - "${temporary}"; then - printf '[%s] benchmark gate produced no Go benchmark results\n' \ - "${module}" >&2 - rm -f "${temporary}" - return 1 - fi - mv "${temporary}" "${output}" -} - -run_make_evidence() { - local selected="$1" - local target="$2" - local output temporary status - output="${root}/.artifacts/${module}/${selected}.txt" - temporary="${output}.tmp.$$" - mkdir -p "$(dirname "${output}")" - rm -f "${output}" "${temporary}" - - set +e - package_make "${target}" 2>&1 | tee "${temporary}" - status=${PIPESTATUS[0]} - set -e - - if [[ "${status}" -ne 0 ]]; then - rm -f "${temporary}" - return "${status}" - fi - if [[ ! -s "${temporary}" ]]; then - printf '[%s] %s gate produced no attributable output\n' \ - "${module}" "${selected}" >&2 - rm -f "${temporary}" - return 1 - fi - mv "${temporary}" "${output}" -} - -go_test() { - local tags - tags="$(test_tags)" - if [[ -n "${tags}" ]]; then - isolated go test -tags="${tags}" "$@" - else - isolated go test "$@" - fi -} - -run_gate() { - local selected="$1" - printf '\n[%s] %s\n' "${module}" "${selected}" - case "${selected}" in - format) - if target="$(find_make_target format)"; then - package_make GOWORK=off "${target}" - else - local -a format_files=() - while IFS= read -r package_directory; do - while IFS= read -r -d '' source_file; do - format_files+=("${source_file}") - done < <( - find "${package_directory}" -maxdepth 1 -type f \ - -name '*.go' -print0 - ) - done < <( - jq -r --arg directory "${module}" \ - '.modules[] | select(.directory == $directory) | - .packages[].directory' "${root}/modules.json" - ) - if [[ "${#format_files[@]}" -gt 0 ]]; then - gofmt -w "${format_files[@]}" - fi - fi - ;; - format-check) - if target="$(find_make_target format-check)"; then - package_make GOWORK=off "${target}" - else - local -a format_files=() - while IFS= read -r package_directory; do - while IFS= read -r -d '' source_file; do - format_files+=("${source_file}") - done < <( - find "${package_directory}" -maxdepth 1 -type f \ - -name '*.go' -print0 - ) - done < <( - jq -r --arg directory "${module}" \ - '.modules[] | select(.directory == $directory) | - .packages[].directory' "${root}/modules.json" - ) - unformatted="" - if [[ "${#format_files[@]}" -gt 0 ]]; then - unformatted="$(gofmt -l "${format_files[@]}")" - fi - [[ -z "${unformatted}" ]] || { - printf 'unformatted Go files:\n%s\n' "${unformatted}" >&2 - exit 1 - } - fi - ;; - tidy-check) - enable_local_proxy - GOWORK=off GOFLAGS="${GOLIB_UPSTREAM_GOFLAGS}" go mod tidy -diff - ;; - tidy) - enable_local_proxy - refresh_owned_sums - GOWORK=off GOFLAGS="${GOLIB_UPSTREAM_GOFLAGS}" \ - "${GOLIB_REAL_GO}" mod tidy - ;; - test) - applicable tests || { skip_not_applicable tests; return; } - packages="$(isolated go list ./...)" - [[ -n "${packages}" ]] || { - printf '[%s] no Go packages were executed\n' "${module}" >&2 - exit 1 - } - go_test ./... -count=1 -timeout=20m - ;; - workspace-test) - applicable tests || { skip_not_applicable tests; return; } - go test ./... -count=1 -timeout=20m - ;; - race) - applicable race || { skip_not_applicable race; return; } - go_test -race ./... -count=1 -timeout=20m - ;; - coverage) - applicable coverage || { skip_not_applicable coverage; return; } - enable_local_proxy - "${root}/.golib/scripts/check-coverage.sh" "${module}" - ;; - mutation) - applicable mutation || { skip_not_applicable mutation; return; } - enable_local_proxy - "${root}/.golib/scripts/check-mutation.sh" "${module}" - ;; - fuzz) - applicable fuzz || { skip_not_applicable fuzz; return; } - enable_local_proxy - if target="$(find_make_target fuzz fuzz-smoke)"; then - package_make \ - FUZZ_TIME="${GOLIB_FUZZ_SMOKE_BUDGET:-10000x}" \ - FUZZTIME="${GOLIB_FUZZ_SMOKE_BUDGET:-10000x}" \ - "${target}" - else - "${root}/.golib/scripts/check-fuzz.sh" "${module}" - fi - ;; - safety) - "${root}/.golib/scripts/check-go-safety.sh" "${module}" - ;; - vet) - applicable lint || { skip_not_applicable lint; return; } - isolated go vet ./... - ;; - lint) - applicable lint || { skip_not_applicable lint; return; } - enable_local_proxy - GOLANGCI_LINT_CACHE="${GOLIB_ISOLATED_MODFILES_DIRECTORY}/golangci-lint-cache" - export GOLANGCI_LINT_CACHE - mkdir -p "${GOLANGCI_LINT_CACHE}" - run_go_tool \ - "github.com/golangci/golangci-lint/v2/cmd/golangci-lint@${GOLANGCI_LINT_VERSION}" \ - golangci-lint \ - run --allow-parallel-runners --timeout=10m ./... - ;; - staticcheck) - applicable lint || { skip_not_applicable lint; return; } - run_go_tool \ - "honnef.co/go/tools/cmd/staticcheck@${STATICCHECK_VERSION}" \ - staticcheck ./... - ;; - nilaway) - applicable lint || { skip_not_applicable lint; return; } - set +e - run_go_tool \ - "go.uber.org/nilaway/cmd/nilaway@${NILAWAY_VERSION}" \ - nilaway \ - -include-pkgs="$(go mod edit -json | jq -r '.Module.Path')" ./... - status=$? - set -e - printf '[%s] NilAway advisory exit status: %s\n' "${module}" "${status}" - ;; - vulnerability) - applicable security || { skip_not_applicable security; return; } - run_go_tool \ - "golang.org/x/vuln/cmd/govulncheck@${GOVULNCHECK_VERSION}" \ - govulncheck ./... - ;; - secrets) - applicable security || { skip_not_applicable security; return; } - run_go_tool \ - "github.com/zricethezav/gitleaks/v8@${GITLEAKS_VERSION}" \ - gitleaks \ - dir . --config "${root}/.gitleaks.toml" --no-banner --redact - ;; - licenses) - applicable security || { skip_not_applicable security; return; } - test -s "${root}/LICENSE" - module_path="$(go mod edit -json | jq -er '.Module.Path')" - if [[ "${module_path}" != "github.com/faustbrian/go-lease" && - "${module_path}" != github.com/faustbrian/go-lease/* ]]; then - printf '[%s] refusing to ignore non-owned module license: %s\n' \ - "${module}" "${module_path}" >&2 - exit 1 - fi - run_go_tool \ - "github.com/google/go-licenses/v2@${GO_LICENSES_VERSION}" \ - go-licenses \ - check ./... \ - --ignore "github.com/faustbrian/go-lease" - ;; - sbom) - applicable security || { skip_not_applicable security; return; } - mkdir -p "${root}/.artifacts/${module}" - run_go_tool \ - "github.com/CycloneDX/cyclonedx-gomod/cmd/cyclonedx-gomod@${CYCLONEDX_VERSION}" \ - cyclonedx-gomod \ - mod -json -licenses -type library -noserial -notimestamp \ - -output "${root}/.artifacts/${module}/sbom.json" . - test -s "${root}/.artifacts/${module}/sbom.json" - ;; - docs) - applicable documentation || { skip_not_applicable documentation; return; } - enable_local_proxy - if [[ "${module}" == "." ]]; then - GOWORK=off "${root}/.golib/scripts/check-documentation.sh" - fi - if target="$(find_make_target docs documentation)"; then - package_make "${target}" - elif [[ "${module}" != "." ]]; then - GOWORK=off go test ./... -run '^Example' -count=1 - fi - ;; - api) - applicable api_compatibility || { skip_not_applicable api_compatibility; return; } - install_go_tool \ - "golang.org/x/exp/cmd/apidiff@${APIDIFF_VERSION}" \ - apidiff - export GOLIB_APIDIFF="${GOLIB_GO_TOOL_PATH}" - if target="$(find_make_target api-compat api-check api compatibility)"; then - package_make "${target}" - elif [[ -x "./scripts/check-api.sh" ]]; then - GOWORK=off ./scripts/check-api.sh - else - "${root}/.golib/scripts/check-api-baseline.sh" "${module}" - fi - ;; - api-update) - applicable api_compatibility || { skip_not_applicable api_compatibility; return; } - install_go_tool \ - "golang.org/x/exp/cmd/apidiff@${APIDIFF_VERSION}" \ - apidiff - export GOLIB_APIDIFF="${GOLIB_GO_TOOL_PATH}" - "${root}/.golib/scripts/update-api-baseline.sh" "${module}" - ;; - conformance) - applicable conformance || { skip_not_applicable conformance; return; } - enable_local_proxy - if target="$(find_make_target conformance specification)"; then - run_make_evidence conformance "${target}" - else - printf '[%s] conformance is declared but has no command\n' \ - "${module}" >&2 - exit 1 - fi - ;; - interoperability) - enable_local_proxy - if target="$(find_make_target interoperability integration conformance)"; then - run_make_evidence interoperability "${target}" - elif interoperability_declared; then - printf '[%s] interoperability is declared but has no command\n' \ - "${module}" >&2 - exit 1 - else - skip_not_applicable interoperability - fi - ;; - benchmark) - applicable benchmarks || { skip_not_applicable benchmarks; return; } - run_benchmark - ;; - release-dry-run) - "${root}/.golib/scripts/release.sh" --dry-run "${module}" - ;; - release-public) - "${root}/.golib/scripts/release.sh" --dry-run --public "${module}" - ;; - check) - while IFS= read -r required_gate; do - [[ -n "${required_gate}" ]] || continue - run_gate "${required_gate}" - done <"${root}/.golib/scripts/check-gates.txt" - ;; - *) - printf 'unknown gate: %s\n' "${selected}" >&2 - exit 2 - ;; - esac -} - -set -a -# shellcheck disable=SC1091 # Repository-pinned tool versions. -source "${root}/.golib/versions.env" -set +a -cd "${directory}" -run_gate "${gate}" diff --git a/.golib/scripts/check-mutation.sh b/.golib/scripts/check-mutation.sh deleted file mode 100755 index 2329a37..0000000 --- a/.golib/scripts/check-mutation.sh +++ /dev/null @@ -1,10 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 1 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -exec "${root}/.golib/scripts/internal/run-mutation.sh" enforce "$1" diff --git a/.golib/scripts/codeql-build.sh b/.golib/scripts/codeql-build.sh deleted file mode 100755 index 8bc3afb..0000000 --- a/.golib/scripts/codeql-build.sh +++ /dev/null @@ -1,65 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -root="$(git rev-parse --show-toplevel)" -task="$(mktemp -d "${TMPDIR:-/tmp}/golib-codeql-build.XXXXXX")" -cleanup() { - find "${task}" -depth -delete 2>/dev/null || true -} -trap cleanup EXIT HUP INT TERM - -root_version="$(jq -er ' - .modules[] | select(.directory == ".") | .version -' "${root}/modules.json")" -self_proxy="${task}/self-proxy" -"${root}/.golib/scripts/build-local-proxy.sh" \ - "${self_proxy}" "v${root_version}" "." -upstream_proxy="${GOPROXY:-$(go env GOPROXY)}" -export GOPROXY="file://${self_proxy},${upstream_proxy}" -current_no_sum_db="$(go env GONOSUMDB)" -export GONOSUMDB="github.com/faustbrian/go-*${current_no_sum_db:+,${current_no_sum_db}}" - -while IFS= read -r module; do - [[ -n "${module}" ]] || continue - module_root="${root}" - if [[ "${module}" != "." ]]; then - module_root="${root}/${module}" - fi - while IFS= read -r package; do - [[ -n "${package}" ]] || continue - package_tags="$( - jq -r \ - --arg module "${module}" \ - --arg package "${package}" ' - .modules[] - | select(.directory == $module) - | .packages[] - | select(.import_path == $package) - | .build_tags[]? - ' "${root}/modules.json" - )" - slug="$(printf '%s' "${package}" | tr '/.' '--')" - if [[ "${module}" == "benchmarks/platform" && -n "${package_tags}" ]]; then - package_tags=benchmark_disabled - fi - if [[ -z "${package_tags}" ]]; then - (cd "${module_root}" && GOWORK=off go build -o "${task}/${slug}" "${package}") - continue - fi - variant=0 - while IFS= read -r tag; do - [[ -n "${tag}" ]] || continue - (cd "${module_root}" && GOWORK=off go build \ - -tags="${tag}" -o "${task}/${slug}-${variant}" "${package}") - variant=$((variant + 1)) - done <<<"${package_tags}" - done < <( - jq -r --arg module "${module}" ' - .modules[] - | select(.directory == $module) - | .packages[] - | select(.build_required == true) - | .import_path - ' "${root}/modules.json" - ) -done < <(jq -r '.modules[].directory' "${root}/modules.json") diff --git a/.golib/scripts/create-verification-snapshot.sh b/.golib/scripts/create-verification-snapshot.sh deleted file mode 100755 index cceb53d..0000000 --- a/.golib/scripts/create-verification-snapshot.sh +++ /dev/null @@ -1,59 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 2 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -source_repository="$(cd "$1" && pwd)" -snapshot_directory="$2" -if [[ -e "${snapshot_directory}" ]]; then - printf 'snapshot destination already exists: %s\n' \ - "${snapshot_directory}" >&2 - exit 1 -fi - -git -C "${source_repository}" rev-parse --verify HEAD >/dev/null -git clone --shared --no-checkout --quiet \ - "${source_repository}" "${snapshot_directory}" -git -C "${snapshot_directory}" config --local core.fsmonitor false -git -C "${snapshot_directory}" checkout --detach --quiet \ - "$(git -C "${source_repository}" rev-parse HEAD)" - -patch="$(mktemp "${TMPDIR:-/tmp}/golib-snapshot-patch.XXXXXX")" -cleanup() { - rm -f "${patch}" -} -trap cleanup EXIT HUP INT TERM -git -C "${source_repository}" diff --binary --full-index HEAD -- >"${patch}" -if [[ -s "${patch}" ]]; then - git -C "${snapshot_directory}" apply --binary "${patch}" -fi -git -C "${snapshot_directory}" ls-files --deleted -z | - git -C "${snapshot_directory}" update-index --remove -z --stdin - -# A staged addition remains tracked even when a machine-wide ignore rule -# matches its name. Mirror that index classification so verification input -# discovery sees the same files in the snapshot and source repository. -while IFS= read -r -d '' path; do - git -C "${snapshot_directory}" add -N -f -- "${path}" -done < <( - git -C "${source_repository}" \ - diff --cached --name-only --diff-filter=A -z -) - -while IFS= read -r -d '' path; do - mkdir -p "${snapshot_directory}/$(dirname "${path}")" - cp -pP \ - "${source_repository}/${path}" \ - "${snapshot_directory}/${path}" -done < <( - git -C "${source_repository}" \ - ls-files --others --exclude-standard -z -) - -mkdir -p "${source_repository}/.artifacts" -ln -s "${source_repository}/.artifacts" \ - "${snapshot_directory}/.artifacts" -printf '%s\n' '/.artifacts' >>"${snapshot_directory}/.git/info/exclude" diff --git a/.golib/scripts/discover-mutation.sh b/.golib/scripts/discover-mutation.sh deleted file mode 100755 index d84a4ff..0000000 --- a/.golib/scripts/discover-mutation.sh +++ /dev/null @@ -1,10 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 1 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -exec "${root}/.golib/scripts/internal/run-mutation.sh" discover "$1" diff --git a/.golib/scripts/filter-releasable-modules.sh b/.golib/scripts/filter-releasable-modules.sh deleted file mode 100755 index 589eca5..0000000 --- a/.golib/scripts/filter-releasable-modules.sh +++ /dev/null @@ -1,12 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -root="${GOLIB_ROOT:-$(git rev-parse --show-toplevel)}" -selection="$(cat)" - -jq -r --arg selection "${selection}" ' - ($selection | split("\n") | map(select(length > 0)))[] as $directory - | .modules[] - | select(.directory == $directory and .releasable == true) - | .directory -' "${root}/modules.json" diff --git a/.golib/scripts/gate-input-digest.sh b/.golib/scripts/gate-input-digest.sh deleted file mode 100755 index 84a2d91..0000000 --- a/.golib/scripts/gate-input-digest.sh +++ /dev/null @@ -1,1035 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -lt 2 || $# -gt 3 ]]; then - printf 'usage: %s [package-directory]\n' "$0" >&2 - exit 2 -fi - -root="${GOLIB_ROOT:-$(git rev-parse --show-toplevel)}" -gate="$1" -module="$2" -package="${3:-}" -input_policy="${GOLIB_GATE_INPUT_POLICY:-current}" -case "${input_policy}" in - current|legacy-api-baseline|legacy-runner-isolation|legacy-runner-isolation-post) ;; - *) - printf 'unsupported gate input policy: %s\n' "${input_policy}" >&2 - exit 2 - ;; -esac -if ! jq -e --arg directory "${module}" \ - '.modules[] | select(.directory == $directory)' \ - "${root}/modules.json" >/dev/null; then - printf 'module is absent from modules.json: %s\n' "${module}" >&2 - exit 2 -fi - -manifest="$(mktemp "${TMPDIR:-/tmp}/golib-gate-inputs.XXXXXX")" -directories="$(mktemp "${TMPDIR:-/tmp}/golib-gate-directories.XXXXXX")" -input_files="$(mktemp "${TMPDIR:-/tmp}/golib-gate-files.XXXXXX")" -package_data="${manifest}.packages" -relevant_package_data="${manifest}.relevant-packages" -existing_files="${manifest}.existing" -file_hashes="${manifest}.hashes" -nested_directories="${manifest}.nested" -bounded_output="${manifest}.bounded-output" -owned_module_paths="${manifest}.owned-modules" -digest_modfile="" -cleanup() { - rm -f \ - "${manifest}" "${directories}" "${input_files}" "${package_data}" \ - "${relevant_package_data}" \ - "${existing_files}" "${file_hashes}" "${nested_directories}" \ - "${bounded_output}" "${owned_module_paths}" - if [[ -n "${digest_modfile}" ]]; then - rm -f "${digest_modfile}" "${digest_modfile%.mod}.sum" - fi -} -trap cleanup EXIT HUP INT TERM - -append_value() { - printf 'value %s=%s\n' "$1" "$2" >>"${manifest}" -} - -append_version_value() { - local name="$1" - local value - if ! value="$({ - # shellcheck source=/dev/null - source "${root}/.golib/versions.env" - set +u - printf '%s' "${!name}" - })" || [[ -z "${value}" ]]; then - printf 'required version value is missing: %s\n' "${name}" >&2 - exit 1 - fi - append_value "version:${name}" "${value}" -} - -append_required_service_versions() { - local service variable - while IFS= read -r service; do - [[ -n "${service}" ]] || continue - case "${service}" in - postgresql) variable=POSTGRES_IMAGE ;; - valkey) variable=VALKEY_IMAGE ;; - redis) variable=REDIS_IMAGE ;; - nats) variable=NATS_IMAGE ;; - nsq) variable=NSQ_IMAGE ;; - rabbitmq) variable=RABBITMQ_IMAGE ;; - rabbitstream-standalone) - append_file "${root}/.golib/services/rabbitstream/enabled_plugins" - append_file "${root}/.golib/services/rabbitstream/standalone-compose.yaml" - append_file "${root}/.golib/services/rabbitstream/standalone-setup.sh" - append_file "${root}/.golib/services/rabbitstream/standalone.conf" - continue - ;; - rabbitstream) - append_file "${root}/.golib/services/rabbitstream/compose.yaml" - append_file "${root}/.golib/services/rabbitstream/enabled_plugins" - append_file "${root}/.golib/services/rabbitstream/rabbit1.conf" - append_file "${root}/.golib/services/rabbitstream/rabbit2.conf" - append_file "${root}/.golib/services/rabbitstream/rabbit3.conf" - append_file "${root}/.golib/services/rabbitstream/setup.sh" - append_file "${root}/.golib/services/rabbitstream/standalone-compose.yaml" - append_file "${root}/.golib/services/rabbitstream/standalone-setup.sh" - append_file "${root}/.golib/services/rabbitstream/standalone.conf" - append_file "${root}/.golib/services/rabbitstream/tls-compose.yaml" - append_file "${root}/.golib/services/rabbitstream/tls-rabbitmq.conf" - append_file "${root}/.golib/services/rabbitstream/tls-setup.sh" - continue - ;; - opensearch) - append_file "${root}/.golib/services/opensearch/opensearch-images.env" - continue - ;; - *) - printf 'unsupported required service %s for %s\n' \ - "${service}" "${module}" >&2 - exit 1 - ;; - esac - append_version_value "${variable}" - done < <(jq -r --arg directory "${module}" ' - .modules[] - | select(.directory == $directory) - | (.required_services // [])[] - ' "${root}/modules.json" | LC_ALL=C sort -u) -} - -append_gate_tool_versions() { - case "${gate}" in - lint) append_version_value GOLANGCI_LINT_VERSION ;; - staticcheck) append_version_value STATICCHECK_VERSION ;; - nilaway) append_version_value NILAWAY_VERSION ;; - vulnerability) append_version_value GOVULNCHECK_VERSION ;; - secrets) append_version_value GITLEAKS_VERSION ;; - licenses) append_version_value GO_LICENSES_VERSION ;; - sbom) append_version_value CYCLONEDX_VERSION ;; - api|api-update) append_version_value APIDIFF_VERSION ;; - workflow) append_version_value ACTIONLINT_VERSION ;; - # Interoperability scripts may use package-specific pinned tools that - # are not yet represented individually in the module catalog. - interoperability) append_file "${root}/.golib/versions.env" ;; - esac -} - -append_file() { - local file="$1" - local relative digest - [[ -f "${file}" ]] || { - printf 'gate input is missing: %s\n' "${file}" >&2 - exit 1 - } - relative="${file#"${root}/"}" - digest="$(shasum -a 256 "${file}" | awk '{print $1}')" - printf 'file %s %s\n' "${digest}" "${relative}" >>"${manifest}" -} - -append_mutation_module_manifest() { - local file="$1" - local relative digest - if [[ ! -f "${owned_module_paths}" ]]; then - jq -r '.modules[].module_path' "${root}/modules.json" | - LC_ALL=C sort -u >"${owned_module_paths}" - fi - relative="${file#"${root}/"}" - digest="$({ - GOLIB_OWNED_MODULE_PATHS="${owned_module_paths}" perl -pe ' - BEGIN { - open my $paths, "<", $ENV{GOLIB_OWNED_MODULE_PATHS} - or die "open owned module paths: $!"; - chomp(@owned = <$paths>); - } - for my $owned (@owned) { - s/(\Q$owned\E)([ \t]+)v[^\s]+/$1$2v0.0.0/g; - } - ' "${file}" - } | shasum -a 256 | awk '{print $1}')" - printf 'file %s %s\n' "${digest}" "${relative}" >>"${manifest}" -} - -append_repository_files() { - local file - : >"${existing_files}" - while IFS= read -r file; do - [[ -n "${file}" ]] || continue - if [[ -f "${root}/${file}" ]]; then - printf '%s\n' "${file}" >>"${existing_files}" - else - append_value missing-file "${file}" - fi - done - [[ -s "${existing_files}" ]] || return - git -C "${root}" hash-object --stdin-paths \ - <"${existing_files}" >"${file_hashes}" - paste "${file_hashes}" "${existing_files}" | - awk -F '\t' '{ printf "file %s %s\n", $1, $2 }' >>"${manifest}" -} - -append_module_files() { - local directory="$1" - local include_api_baseline=0 - local include_documentation=0 - local include_secret_policy=0 - local include_tests=0 - if [[ "${input_policy}" == "legacy-api-baseline" ]]; then - include_api_baseline=1 - fi - if [[ "${directory}" == "${module}" ]]; then - include_tests=1 - fi - case "${gate}" in - api|api-update) - include_api_baseline=1 - ;; - docs) - include_documentation=1 - ;; - secrets) - include_documentation=1 - include_secret_policy=1 - ;; - esac - : >"${nested_directories}" - jq -r --arg directory "${directory}" ' - .modules[] - | .directory - | select( - $directory != "." and - . != $directory and - startswith($directory + "/") - ) - ' "${root}/modules.json" >"${nested_directories}" - git -C "${root}" ls-files -co --exclude-standard -- "${directory}" | - awk \ - -v include_api_baseline="${include_api_baseline}" \ - -v include_documentation="${include_documentation}" \ - -v include_secret_policy="${include_secret_policy}" \ - -v include_tests="${include_tests}" \ - -v module_directory="${directory}" ' - FILENAME != "-" { - nested[++count] = $0 - next - } - { - relative = tolower($0) - module_prefix = module_directory == "." ? "" : tolower(module_directory) "/" - if (module_prefix != "" && substr(relative, 1, length(module_prefix)) == module_prefix) { - relative = substr(relative, length(module_prefix) + 1) - } - is_markdown = relative ~ /\.(md|markdown)$/ - in_documentation = relative ~ /^(docs|\.ai)\// - in_test_data = relative ~ /(^|\/)(testdata|fixtures|corpus)\// - is_named_documentation = relative ~ /(^|\/)(readme|changelog|contributing|security|code_of_conduct|support)\.(md|markdown)$/ - is_generated_documentation = relative == "llms.txt" || relative == "llms-full.txt" - is_api_baseline = relative == "api/baseline.txt" - is_repository_catalog = relative == "modules.json" || relative == "packages.json" - is_secret_policy = relative == ".gitleaks.toml" - is_test_source = relative ~ /_test\.go$/ - skip_documentation = !include_documentation && (is_generated_documentation || (is_markdown && (in_documentation || (!in_test_data && is_named_documentation)))) - skip_secret_policy = !include_secret_policy && is_secret_policy - if (is_repository_catalog) { - next - } - if (!include_api_baseline && is_api_baseline) { - next - } - if (skip_secret_policy) { - next - } - if (!include_tests && is_test_source) { - next - } - for (position = 1; position <= count; position++) { - prefix = nested[position] "/" - if ($0 == nested[position] || substr($0, 1, length(prefix)) == prefix) { - next - } - } - if (skip_documentation) { - next - } - print - } - ' "${nested_directories}" - | - while IFS= read -r file; do - if [[ -e "${root}/${file}" || -L "${root}/${file}" ]]; then - printf '%s\n' "${file}" - fi - done >>"${input_files}" -} - -append_tool_inputs() { - append_version_value GREMLINS_VERSION - append_required_service_versions - append_file "${root}/.golib/scripts/internal/mutation-command.sh" - append_file "${root}/.golib/scripts/internal/mutation-coverage.sh" - append_file "${root}/.golib/scripts/patches/gremlins-run-all-mutants.patch" - append_file "${root}/.golib/scripts/patches/gremlins-shared-coverage.patch" - append_file "${root}/.golib/scripts/start-services.sh" -} - -append_environment() { - append_value go-version "$(go env GOVERSION)" - append_value goos "$(go env GOOS)" - append_value goarch "$(go env GOARCH)" - append_value cgo-enabled "$(go env CGO_ENABLED)" -} - -append_assurance_environment() { - if [[ -z "${GOLIB_ASSURANCE_GO_VERSION:-}" && - -z "${GOLIB_ASSURANCE_GOOS:-}" && - -z "${GOLIB_ASSURANCE_GOARCH:-}" && - -z "${GOLIB_ASSURANCE_CGO_ENABLED:-}" && - -z "${GOLIB_ASSURANCE_KERNEL:-}" && - -z "${GOLIB_ASSURANCE_NODE:-}" ]]; then - return 1 - fi - if [[ -z "${GOLIB_ASSURANCE_GO_VERSION:-}" || - -z "${GOLIB_ASSURANCE_GOOS:-}" || - -z "${GOLIB_ASSURANCE_GOARCH:-}" || - -z "${GOLIB_ASSURANCE_CGO_ENABLED:-}" || - -z "${GOLIB_ASSURANCE_KERNEL:-}" || - -z "${GOLIB_ASSURANCE_NODE:-}" ]]; then - printf 'operational-assurance environment override is incomplete\n' >&2 - exit 2 - fi - local value - for value in \ - "${GOLIB_ASSURANCE_GO_VERSION}" \ - "${GOLIB_ASSURANCE_GOOS}" \ - "${GOLIB_ASSURANCE_GOARCH}" \ - "${GOLIB_ASSURANCE_CGO_ENABLED}" \ - "${GOLIB_ASSURANCE_KERNEL}" \ - "${GOLIB_ASSURANCE_NODE}"; do - if [[ "${value}" == *$'\n'* || "${value}" == *$'\r'* ]]; then - printf 'operational-assurance environment override contains control characters\n' >&2 - exit 2 - fi - done - if [[ "${GOLIB_ASSURANCE_CGO_ENABLED}" != "0" && - "${GOLIB_ASSURANCE_CGO_ENABLED}" != "1" ]]; then - printf 'operational-assurance cgo override must be 0 or 1\n' >&2 - exit 2 - fi - append_value go-version "${GOLIB_ASSURANCE_GO_VERSION}" - append_value goos "${GOLIB_ASSURANCE_GOOS}" - append_value goarch "${GOLIB_ASSURANCE_GOARCH}" - append_value cgo-enabled "${GOLIB_ASSURANCE_CGO_ENABLED}" - append_value kernel "${GOLIB_ASSURANCE_KERNEL}" - append_value node "${GOLIB_ASSURANCE_NODE}" -} - -bounded_command_output() { - local timeout_seconds="$1" - shift - local command_pid status timer_pid - : >"${bounded_output}" - "$@" >"${bounded_output}" 2>/dev/null & - command_pid=$! - ( - sleep "${timeout_seconds}" - kill -TERM "${command_pid}" 2>/dev/null || true - sleep 1 - kill -KILL "${command_pid}" 2>/dev/null || true - ) & - timer_pid=$! - status=0 - wait "${command_pid}" || status=$? - kill -TERM "${timer_pid}" 2>/dev/null || true - wait "${timer_pid}" 2>/dev/null || true - if [[ "${status}" -eq 0 ]]; then - cat "${bounded_output}" - fi - return "${status}" -} - -append_legacy_docker_environment() { - local docker_timeout docker_value - if ! jq -e --arg directory "${module}" ' - .modules[] - | select(.directory == $directory) - | select((.required_services // []) | length > 0) - ' "${root}/modules.json" >/dev/null; then - append_value docker not-required - return - fi - if [[ -n "${GOLIB_LEGACY_DOCKER_VALUE:-}" ]]; then - append_value docker "${GOLIB_LEGACY_DOCKER_VALUE}" - return - fi - if ! command -v docker >/dev/null 2>&1; then - append_value docker missing - return - fi - docker_timeout="${GOLIB_DOCKER_VERSION_TIMEOUT_SECONDS:-5}" - if [[ ! "${docker_timeout}" =~ ^[1-9][0-9]*$ ]]; then - printf 'invalid Docker version timeout: %s\n' "${docker_timeout}" >&2 - exit 1 - fi - docker_value="$({ - bounded_command_output \ - "${docker_timeout}" \ - docker version --format '{{.Server.Version}}' || - printf unavailable - })" - append_value docker "${docker_value}" -} - -append_verification_environment() { - if [[ "${gate}" == "operational-assurance" ]] && append_assurance_environment; then - return - fi - append_environment - append_value kernel "$(uname -srm)" - # Pinned service images define the runtime contract. Live daemon - # availability is orchestration state and must not invalidate evidence. - # The legacy policy reproduces the old identity solely for verified, - # one-time evidence migration. - if [[ "${input_policy}" == "legacy-api-baseline" ]]; then - append_legacy_docker_environment - fi - if command -v node >/dev/null 2>&1; then - append_value node "$(node --version)" - else - append_value node missing - fi -} - -normalize_irrelevant_formatter_dispatch() { - awk -v selected_gate="${gate}" ' - BEGIN { quote = sprintf("%c", 39) } - selected_gate != "format" && selected_gate != "format-check" && - $0 == " format)" { - print " format)" - print " find . -name " quote "*.go" quote " -not -path " quote "./.tools/*" quote " -print0 | xargs -0 gofmt -w" - print " ;;" - print " format-check)" - print " unformatted=\"$(find . -name " quote "*.go" quote " -not -path " quote "./.tools/*" quote " -print0 | xargs -0 gofmt -l)\"" - print " [[ -z \"${unformatted}\" ]] || {" - print " printf " quote "unformatted Go files:\\n%s\\n" quote " \"${unformatted}\" >&2" - print " exit 1" - print " }" - print " ;;" - skip_formatter_dispatch = 1 - next - } - skip_formatter_dispatch && $0 == " tidy-check)" { - skip_formatter_dispatch = 0 - print - next - } - skip_formatter_dispatch { next } - { print } - ' -} - -append_verification_tool_files() { - local check_module_digest legacy_runner_digest - local paths=( - scripts/start-services.sh - ) - if [[ "${gate}" == "operational-assurance" && "${input_policy}" == "current" ]]; then - paths+=( - scripts/create-verification-snapshot.sh - scripts/run-modules.sh - scripts/stop-services.sh - ) - elif [[ "${input_policy}" == "legacy-runner-isolation" || - "${input_policy}" == "legacy-runner-isolation-post" ]]; then - paths+=( - scripts/create-verification-snapshot.sh - scripts/stop-services.sh - ) - # These are the exact Git blob identities immediately before and after - # process isolation, matching append_repository_files. They identify - # retained evidence without requiring that history to remain available. - legacy_runner_digest='cf841512fc1e48c8c7708259c878028f06a8726f' - if [[ "${input_policy}" == "legacy-runner-isolation-post" ]]; then - legacy_runner_digest='d30bc5a6f7e52b2080a3fe13200dfb2963a1415a' - fi - printf 'file %s %s\n' \ - "${legacy_runner_digest}" \ - 'scripts/run-modules.sh' >>"${manifest}" - else - # Snapshot creation, module selection, and post-gate cleanup do not - # alter a single gate's command or inputs. Their behavior belongs to - # aggregate-run evidence rather than every package checkpoint. - append_value verification-orchestration-contract v1 - fi - append_gate_tool_versions - append_required_service_versions - case "${gate}" in - format-check|workspace-test|safety|benchmark|release-public) ;; - *) - paths+=( - scripts/build-local-proxy.sh - scripts/internal/isolated-go.sh - ) - ;; - esac - case "${gate}" in - coverage) - paths+=(scripts/check-coverage.sh) - ;; - fuzz) - paths+=(scripts/check-fuzz.sh) - ;; - safety) - paths+=(scripts/check-go-safety.sh) - ;; - api) - paths+=(scripts/check-api-baseline.sh) - ;; - api-update) - paths+=(scripts/update-api-baseline.sh) - ;; - licenses) - paths+=(LICENSE) - ;; - release-dry-run|release-public) - paths+=( - scripts/filter-releasable-modules.sh - scripts/release.sh - ) - ;; - esac - git -C "${root}" ls-files -co --exclude-standard -- \ - "${paths[@]}" >>"${input_files}" - - # Runner concurrency changes whether a linter can start, not its findings. - # Keep successful gate evidence bound to the analyzer contract rather than - # invalidating every package when isolated runners are made parallel-safe. - if [[ -f "${root}/.golib/scripts/check-module.sh" ]]; then - if [[ "${gate}" == "docs" ]]; then - check_module_digest="$( - normalize_irrelevant_formatter_dispatch \ - <"${root}/.golib/scripts/check-module.sh" | - sed 's/ --allow-parallel-runners//g' | - git hash-object --stdin - )" - else - # Root-only documentation dispatch cannot alter another gate's - # executable contract or invalidate its retained evidence. - check_module_digest="$( - awk ' - $0 == " if [[ \"${module}\" == \".\" ]]; then" { - print " if target=\"$(find_make_target docs documentation)\"; then" - skip_root_documentation = 1 - next - } - skip_root_documentation && $0 == " elif target=\"$(find_make_target docs documentation)\"; then" { - skip_root_documentation = 0 - next - } - skip_root_documentation { next } - { print } - ' "${root}/.golib/scripts/check-module.sh" | - normalize_irrelevant_formatter_dispatch | - sed 's/ --allow-parallel-runners//g' | - git hash-object --stdin - )" - fi - printf 'file %s %s\n' \ - "${check_module_digest}" \ - 'scripts/check-module.sh' >>"${manifest}" - else - append_value missing-file scripts/check-module.sh - fi -} - -verification_module_policy() { - local directory="$1" - jq -S -c --arg directory "${directory}" ' - .modules[] - | select(.directory == $directory) - | { - directory, - module_path, - owned_dependencies, - required_services, - test_tags, - interoperability_tools, - gates - } - ' "${root}/modules.json" -} - -verification_digest() { - local directory file - local repository_paths=() - append_value gate "${gate}" - append_value module "${module}" - append_verification_environment - - printf '%s\n' "${module}" >"${directories}" - jq -r --arg directory "${module}" ' - . as $catalog - | def closure($seen): - ([ - $catalog.modules[] - | select(.module_path as $path | $seen | index($path)) - | .owned_dependencies[] - ] | unique) as $dependencies - | ($seen + $dependencies | unique) as $next - | if $next == $seen then $next else closure($next) end; - (.modules[] | select(.directory == $directory).owned_dependencies) as $owned - | closure($owned) as $paths - | .modules[] - | select(.module_path as $path | $paths | index($path)) - | .directory - ' "${root}/modules.json" >>"${directories}" - - while IFS= read -r directory; do - [[ -n "${directory}" ]] || continue - append_value "module-policy:${directory}" "$( - verification_module_policy "${directory}" - )" - append_value "package-policy:${directory}" "$( - jq -S -c --arg directory "${directory}" \ - '[.packages[] | select(.module_directory == $directory)]' \ - "${root}/packages.json" - )" - append_module_files "${directory}" - done < <(LC_ALL=C sort -u "${directories}") - append_verification_tool_files - case "${gate}" in - secrets) - repository_paths+=(.gitleaks.toml) - ;; - benchmark|workspace-test) - repository_paths+=(go.work) - ;; - esac - if [[ "${#repository_paths[@]}" -gt 0 ]]; then - git -C "${root}" ls-files -co --exclude-standard -- \ - "${repository_paths[@]}" >>"${input_files}" - fi - - LC_ALL=C sort -u "${input_files}" | append_repository_files -} - -mutation_digest() { - local package_directory package_input_digest - append_value gate mutation - append_value module "${module}" - append_file "${root}/.golib/scripts/check-mutation.sh" - append_file "${root}/.golib/scripts/internal/configure-mutation-workers.sh" - append_file "${root}/.golib/scripts/internal/run-mutation.sh" - append_file "${root}/.golib/scripts/internal/mutation-scratch.sh" - append_file "${root}/.golib/scripts/internal/reuse-mutation-coverage.sh" - while IFS= read -r package_directory; do - [[ -n "${package_directory}" ]] || continue - package_input_digest="$( - "${root}/.golib/scripts/gate-input-digest.sh" \ - mutation "${module}" "${package_directory}" - )" - append_value "package:${package_directory}" "${package_input_digest}" - done < <( - jq -r --arg directory "${module}" ' - .modules[] - | select(.directory == $directory) - | .packages[] - | select(.coverage_required == true) - | .directory - ' "${root}/modules.json" | LC_ALL=C sort - ) -} - -legacy_digest() { - local directory - append_value gate mutation - append_value module "${module}" - append_environment - append_value module-policy "$( - jq -S -c --arg directory "${module}" \ - '.modules[] | select(.directory == $directory)' \ - "${root}/modules.json" - )" - append_value package-policy "$( - jq -S -c --arg directory "${module}" \ - '[.packages[] | select(.module_directory == $directory)]' \ - "${root}/packages.json" - )" - append_value zero-mutant-policy "$( - jq -S -c --arg directory "${module}" \ - '[.packages[] | select(.module_directory == $directory)]' \ - "${root}/.golib/mutation-zero-inventory.json" - )" - - append_file "${root}/.golib/versions.env" - append_file "${root}/.golib/scripts/build-golib-gremlins.sh" - append_file "${root}/.golib/scripts/internal/mutation-command.sh" - append_file "${root}/.golib/scripts/patches/gremlins-run-all-mutants.patch" - append_file "${root}/.golib/scripts/patches/gremlins-shared-coverage.patch" - append_file "${root}/.golib/scripts/start-services.sh" - - printf '%s\n' "${module}" >"${directories}" - jq -r --arg directory "${module}" ' - . as $catalog - | def closure($seen): - ([ - $catalog.modules[] - | select(.module_path as $path | $seen | index($path)) - | .owned_dependencies[] - ] | unique) as $dependencies - | ($seen + $dependencies | unique) as $next - | if $next == $seen then $next else closure($next) end; - (.modules[] | select(.directory == $directory).owned_dependencies) as $owned - | closure($owned) as $paths - | .modules[] - | select(.module_path as $path | $paths | index($path)) - | .directory - ' "${root}/modules.json" >>"${directories}" - - while IFS= read -r directory; do - [[ -n "${directory}" ]] || continue - while IFS= read -r -d '' file; do - append_file "${file}" - done < <( - find "${root}/${directory}" -type f \ - ! -path '*/.git/*' \ - ! -path '*/.artifacts/*' \ - ! -path '*/.tools/*' \ - ! -name '*.coverprofile' \ - ! -name 'coverage.out' \ - -print0 | LC_ALL=C sort -z - ) - done < <(LC_ALL=C sort -u "${directories}") -} - -package_digest() { - local data_name digest_go digest_go_flags digest_workspace flag owned_count - local module_path module_root package_directory resolution tags - local target_import_path - module_root="${root}/${module}" - if ! jq -e --arg directory "${module}" --arg package "${package}" ' - .modules[] - | select(.directory == $directory) - | .packages[] - | select(.directory == $package and .coverage_required == true) - ' "${root}/modules.json" >/dev/null; then - printf 'mutation package is absent from catalog: %s %s\n' \ - "${module}" "${package}" >&2 - exit 2 - fi - - append_value gate mutation - append_value module "${module}" - append_value package "${package}" - append_environment - append_value module-policy "$( - jq -S -c --arg directory "${module}" ' - .modules[] - | select(.directory == $directory) - | { - directory, - module_path, - go_version, - owned_dependencies, - required_services, - test_tags, - mutation: .gates.mutation - } - ' "${root}/modules.json" - )" - append_value package-policy "$( - jq -S -c --arg directory "${module}" --arg package "${package}" ' - .modules[] - | select(.directory == $directory) - | .packages[] - | select(.directory == $package) - | { - module_directory, - directory, - name, - import_path, - kind, - production, - executable, - coverage_required - } - ' "${root}/modules.json" - )" - append_value zero-mutant-policy "$( - jq -S -c --arg directory "${module}" --arg package "${package}" ' - [.packages[] | select( - .module_directory == $directory and - .package_directory == $package - )] - ' "${root}/.golib/mutation-zero-inventory.json" - )" - if [[ "${module}" == "pkg/ecma-regexp" ]]; then - append_value mutation-test-environment "TEST262_ROOT=unset" - fi - # Evidence orchestration does not affect which mutants execute or which - # tests observe them. Campaign semantics are captured by append_tool_inputs. - append_tool_inputs - - module_path="$(jq -r --arg directory "${module}" ' - .modules[] - | select(.directory == $directory) - | .module_path - ' "${root}/modules.json")" - tags="$(jq -r --arg directory "${module}" ' - .modules[] - | select(.directory == $directory) - | .test_tags - | join(",") - ' "${root}/modules.json")" - resolution="${GOLIB_MUTATION_DIGEST_RESOLUTION:-stable}" - if [[ "${resolution}" == "caller" ]]; then - ( - cd "${module_root}" - if [[ -n "${tags}" ]]; then - go list -deps -test -json -tags="${tags}" ./... - else - go list -deps -test -json ./... - fi - ) >"${package_data}" - elif [[ "${resolution}" != "stable" && - "${resolution}" != "observer-v1" && - "${resolution}" != "legacy-stable" ]]; then - printf 'unknown mutation digest resolution: %s\n' \ - "${resolution}" >&2 - exit 2 - else - digest_go="${GOLIB_REAL_GO:-$(command -v go)}" - digest_workspace=off - digest_go_flags="" - for flag in ${GOLIB_UPSTREAM_GOFLAGS:-${GOFLAGS:-}}; do - case "${flag}" in - -mod=*|-modfile=*) ;; - *) - digest_go_flags="$( - printf '%s%s' \ - "${digest_go_flags:+${digest_go_flags} }" "${flag}" - )" - ;; - esac - done - owned_count="$(jq -r --arg directory "${module}" ' - .modules[] | select(.directory == $directory) | .owned_dependencies | length - ' "${root}/modules.json")" - if [[ "${owned_count}" -gt 0 ]]; then - digest_modfile="${manifest}.mutation.mod" - cp "${module_root}/go.mod" "${digest_modfile}" - if [[ -f "${module_root}/go.sum" ]]; then - cp "${module_root}/go.sum" "${digest_modfile%.mod}.sum" - fi - while IFS=$'\t' read -r owned_path owned_directory; do - [[ -n "${owned_path}" && -n "${owned_directory}" ]] || continue - GOWORK=off "${digest_go}" mod edit -modfile="${digest_modfile}" \ - -replace="${owned_path}=${root}/${owned_directory}" - done < <(jq -r --arg directory "${module}" ' - . as $catalog - | def closure($seen): - ([ $catalog.modules[] - | select(.module_path as $path | $seen | index($path)) - | .owned_dependencies[] ] | unique) as $dependencies - | ($seen + $dependencies | unique) as $next - | if $next == $seen then $next else closure($next) end; - (.modules[] | select(.directory == $directory).owned_dependencies) as $owned - | closure($owned) as $paths - | .modules[] - | select(.module_path as $path | $paths | index($path)) - | [.module_path, .directory] - | @tsv - ' "${root}/modules.json") - digest_go_flags="$( - printf '%s%s' "${digest_go_flags:+${digest_go_flags} }" \ - "-modfile=${digest_modfile} -mod=mod" - )" - fi - ( - cd "${module_root}" - if [[ -n "${tags}" ]]; then - GOWORK="${digest_workspace}" GOFLAGS="${digest_go_flags}" \ - "${digest_go}" list -deps -test -json \ - -tags="${tags}" ./... - else - GOWORK="${digest_workspace}" GOFLAGS="${digest_go_flags}" \ - "${digest_go}" list -deps -test -json ./... - fi - ) >"${package_data}" - fi - - if [[ "${resolution}" == "legacy-stable" ]]; then - jq -s \ - --arg root "${root}/" \ - --arg module_path "${module_path}" ' - .[] - | select((.Dir // "") | startswith($root)) - | . + { - GolibMutationObserver: - ((.Module.Path // "") == $module_path) - } - ' "${package_data}" >"${relevant_package_data}" - elif [[ "${resolution}" == "observer-v1" ]]; then - target_import_path="${module_path}" - if [[ "${package}" != "." ]]; then - target_import_path="${module_path}/${package}" - fi - jq -s \ - --arg root "${root}/" \ - --arg target "${target_import_path}" ' - def canonical_import: - (.ImportPath // "" | sub(" \\[.*$"; "")); - [.[] | select( - canonical_import == $target or - (.ForTest // "") == $target - )] as $observers - | ([$observers[].Dir] | unique) as $observer_directories - | ([ - $observers[] - | canonical_import, (.Deps // [])[] - ] | unique) as $relevant_imports - | .[] - | select((.Dir // "") | startswith($root)) - | select( - canonical_import as $import - | ($relevant_imports | index($import)) != null - ) - | .Dir as $directory - | . + { - GolibMutationObserver: - (($observer_directories | index($directory)) != null) - } - ' "${package_data}" >"${relevant_package_data}" - else - target_import_path="${module_path}" - if [[ "${package}" != "." ]]; then - target_import_path="${module_path}/${package}" - fi - jq -s \ - --arg root "${root}/" \ - --arg target "${target_import_path}" ' - def canonical_import: - (.ImportPath // "" | sub(" \\[.*$"; "")); - [.[] | select( - (canonical_import == $target and - (.ForTest // "") == "") or - (.ForTest // "") == $target - )] as $observers - | ([$observers[].Dir] | unique) as $observer_directories - | ([ - $observers[] - | canonical_import, (.Deps // [])[] - ] | unique) as $relevant_imports - | .[] - | select((.Dir // "") | startswith($root)) - | select( - (.ForTest // "") == "" or - (.ForTest // "") == $target - ) - | select( - canonical_import as $import - | ($relevant_imports | index($import)) != null - ) - | .Dir as $directory - | . + { - GolibMutationObserver: - (($observer_directories | index($directory)) != null) - } - ' "${package_data}" >"${relevant_package_data}" - fi - - jq -r --arg root "${root}/" --arg module_path "${module_path}" ' - select(.Dir | startswith($root)) - | .Dir as $directory - | ( - [ - .GoFiles[]?, - .CgoFiles[]?, - .CFiles[]?, - .CXXFiles[]?, - .MFiles[]?, - .HFiles[]?, - .FFiles[]?, - .SFiles[]?, - .SwigFiles[]?, - .SwigCXXFiles[]?, - .SysoFiles[]?, - .EmbedFiles[]? - ] + - ( - if .GolibMutationObserver == true - then [ - .TestGoFiles[]?, - .XTestGoFiles[]?, - .TestEmbedFiles[]?, - .XTestEmbedFiles[]? - ] - else [] - end - ) - )[] - | if startswith("/") then . else "\($directory)/\(.)" end - | select(startswith($root)) - ' "${relevant_package_data}" >>"${input_files}" - - while IFS= read -r file; do - [[ -n "${file}" ]] || continue - append_mutation_module_manifest "${file}" - done < <(jq -r --arg root "${root}/" ' - select( - (.Module.GoMod // "") == $root or - ((.Module.GoMod // "") | startswith($root)) - ) - | .Module.GoMod - ' "${relevant_package_data}" | LC_ALL=C sort -u) - - while IFS= read -r package_directory; do - [[ -n "${package_directory}" ]] || continue - for data_name in corpus fixtures testdata; do - if [[ -d "${package_directory}/${data_name}" ]]; then - find "${package_directory}/${data_name}" -type f \ - -print >>"${input_files}" - fi - done - done < <( - jq -r --arg root "${root}/" ' - select(.Dir | startswith($root)) - | .Dir - ' "${relevant_package_data}" | LC_ALL=C sort -u - ) - - while IFS= read -r file; do - [[ -n "${file}" ]] || continue - append_file "${file}" - done < <(LC_ALL=C sort -u "${input_files}") -} - -if [[ "${gate}" == "mutation" && -z "${package}" ]]; then - mutation_digest -elif [[ "${gate}" == "mutation-legacy" ]]; then - legacy_digest -elif [[ "${gate}" == "mutation" ]]; then - package_digest -else - verification_digest -fi - -LC_ALL=C sort "${manifest}" | shasum -a 256 | awk '{print $1}' diff --git a/.golib/scripts/internal/configure-mutation-workers.sh b/.golib/scripts/internal/configure-mutation-workers.sh deleted file mode 100755 index 0e86726..0000000 --- a/.golib/scripts/internal/configure-mutation-workers.sh +++ /dev/null @@ -1,20 +0,0 @@ -#!/usr/bin/env bash - -configure_mutation_workers() { - local workers="$1" - local index - - if [[ ! "${workers}" =~ ^[1-9][0-9]*$ ]]; then - printf 'mutation workers must be a positive integer\n' >&2 - return 2 - fi - for index in "${!mutation_arguments[@]}"; do - if [[ "${mutation_arguments[${index}]}" == "--workers" ]]; then - mutation_arguments[index + 1]="${workers}" - return - fi - done - - printf 'mutation command is missing --workers\n' >&2 - return 2 -} diff --git a/.golib/scripts/internal/isolated-go.sh b/.golib/scripts/internal/isolated-go.sh deleted file mode 100755 index d84188f..0000000 --- a/.golib/scripts/internal/isolated-go.sh +++ /dev/null @@ -1,144 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -real_go="${GOLIB_REAL_GO:-}" -cache_root="${GOLIB_ISOLATED_MODFILES_DIRECTORY:-}" -if [[ -z "${real_go}" || -z "${cache_root}" ]]; then - printf 'isolated Go requires GOLIB_REAL_GO and ' >&2 - printf 'GOLIB_ISOLATED_MODFILES_DIRECTORY\n' >&2 - exit 2 -fi - -module_root="${PWD}" -while [[ ! -f "${module_root}/go.mod" ]]; do - parent="$(dirname "${module_root}")" - if [[ "${parent}" == "${module_root}" ]]; then - exec "${real_go}" "$@" - fi - module_root="${parent}" -done - -clean_flags="" -for flag in ${GOFLAGS:-}; do - case "${flag}" in - -mod=*|-modfile=*) ;; - *) clean_flags="${clean_flags:+${clean_flags} }${flag}" ;; - esac -done - -identity="$( - { - printf '%s\n' "${module_root}" - cksum "${module_root}/go.mod" - if [[ -f "${module_root}/go.sum" ]]; then - cksum "${module_root}/go.sum" - fi - } | cksum | awk '{print $1 "-" $2}' -)" -state="${cache_root}/${identity}" -modfile="${state}/isolated.mod" -sumfile="${state}/isolated.sum" -ready="${state}/ready" - -if [[ ! -f "${ready}" ]]; then - lock="${state}.lock" - if mkdir "${lock}" 2>/dev/null; then - mkdir -p "${state}" - cp "${module_root}/go.mod" "${modfile}" - if [[ -f "${module_root}/go.sum" ]]; then - awk '$1 !~ /^github\.com\/faustbrian\/golib\// { print }' \ - "${module_root}/go.sum" >"${sumfile}" - else - : >"${sumfile}" - fi - preparation_flags="${clean_flags:+${clean_flags} }-modfile=${modfile}" - if ! GOWORK=off GOFLAGS="${preparation_flags}" \ - "${real_go}" mod download all; then - rm -rf "${state}" "${lock}" - exit 1 - fi - : >"${ready}" - rmdir "${lock}" - else - while [[ ! -f "${ready}" ]]; do - sleep 0.05 - done - fi -fi - -if [[ "${1:-}" == "mod" && "${2:-}" == "tidy" && - " $* " == *" -diff "* ]]; then - tidy_arguments=() - for argument in "$@"; do - [[ "${argument}" == "-diff" ]] || tidy_arguments+=("${argument}") - done - tidy_flags="${clean_flags:+${clean_flags} }-modfile=${modfile}" - GOWORK=off GOFLAGS="${tidy_flags}" \ - "${real_go}" "${tidy_arguments[@]}" - - status=0 - if ! diff -u "${module_root}/go.mod" "${modfile}"; then - status=1 - fi - source_sum="${state}/source-external.sum" - tidy_sum="${state}/tidy-external.sum" - if [[ -f "${module_root}/go.sum" ]]; then - awk '$1 !~ /^github\.com\/faustbrian\/golib\// { print }' \ - "${module_root}/go.sum" >"${source_sum}" - else - : >"${source_sum}" - fi - awk '$1 !~ /^github\.com\/faustbrian\/golib\// { print }' \ - "${sumfile}" >"${tidy_sum}" - if ! diff -u "${source_sum}" "${tidy_sum}"; then - status=1 - fi - exit "${status}" -fi - -case "${1:-}" in - exec-tool) - shift - command_flags="${clean_flags:+${clean_flags} }-modfile=${modfile} -mod=readonly" - GOLIB_ISOLATED_MODFILE="${modfile}" \ - GOWORK=off GOFLAGS="${command_flags}" \ - PATH="$(dirname "${real_go}"):${PATH}" exec "$@" - ;; - run|install) - command="$1" - shift - if [[ " $* " == *"@"* ]]; then - GOWORK=off GOFLAGS="${clean_flags}" \ - "${real_go}" "${command}" "$@" - else - GOLIB_ISOLATED_MODFILE="${modfile}" \ - GOWORK=off GOFLAGS="${clean_flags}" \ - "${real_go}" "${command}" \ - "-modfile=${modfile}" -mod=readonly "$@" - fi - ;; - get) - command_flags="${clean_flags:+${clean_flags} }-modfile=${modfile} -mod=mod" - GOWORK=off GOFLAGS="${command_flags}" "${real_go}" "$@" - ;; - build|clean|fix|fmt|generate|list|test|vet) - command="$1" - shift - GOLIB_ISOLATED_MODFILE="${modfile}" \ - GOWORK=off GOFLAGS="${clean_flags}" \ - "${real_go}" "${command}" \ - "-modfile=${modfile}" -mod=readonly "$@" - ;; - doc) - command_flags="${clean_flags:+${clean_flags} }-modfile=${modfile} -mod=readonly" - GOLIB_ISOLATED_MODFILE="${modfile}" \ - GOWORK=off GOFLAGS="${command_flags}" "${real_go}" "$@" - ;; - mod) - command_flags="${clean_flags:+${clean_flags} }-modfile=${modfile}" - GOWORK=off GOFLAGS="${command_flags}" "${real_go}" "$@" - ;; - *) - GOWORK=off GOFLAGS="${clean_flags}" "${real_go}" "$@" - ;; -esac diff --git a/.golib/scripts/internal/migrate-api-baseline-evidence.sh b/.golib/scripts/internal/migrate-api-baseline-evidence.sh deleted file mode 100755 index 081483a..0000000 --- a/.golib/scripts/internal/migrate-api-baseline-evidence.sh +++ /dev/null @@ -1,165 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 1 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -module="$1" -artifact="${root}/.artifacts/${module}/evidence" -validated_revision="$(git rev-parse HEAD)" -validated_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" -lock="" -lock_acquired=0 -migration_lock="${artifact}/.locks/api-baseline-evidence-migration.lock" -migration_lock_acquired=0 -temporary_evidence="" -temporary_log="" -temporary_legacy_evidence="" -temporary_legacy_log="" - -cleanup_gate() { - rm -f \ - "${temporary_evidence}" "${temporary_log}" \ - "${temporary_legacy_evidence}" "${temporary_legacy_log}" - if [[ "${lock_acquired}" -eq 1 ]]; then - rm -f "${lock}/owner" - rmdir "${lock}" 2>/dev/null || true - fi - lock_acquired=0 - temporary_evidence="" - temporary_log="" - temporary_legacy_evidence="" - temporary_legacy_log="" -} - -cleanup() { - cleanup_gate - if [[ "${migration_lock_acquired}" -eq 1 ]]; then - rm -f "${migration_lock}/owner" - rmdir "${migration_lock}" 2>/dev/null || true - fi -} -trap cleanup EXIT HUP INT TERM - -[[ -f "${root}/${module}/api/baseline.txt" ]] || { - printf 'module has no API baseline: %s\n' "${module}" >&2 - exit 1 -} - -mkdir -p "${artifact}/.locks" -while ! mkdir "${migration_lock}" 2>/dev/null; do - if [[ -f "${migration_lock}/owner" ]]; then - owner="$(cat "${migration_lock}/owner" 2>/dev/null || true)" - if [[ "${owner}" =~ ^[0-9]+$ ]] && ! kill -0 "${owner}" 2>/dev/null; then - rm -f "${migration_lock}/owner" - rmdir "${migration_lock}" 2>/dev/null || true - continue - fi - fi - sleep 0.05 -done -printf '%s\n' "$$" >"${migration_lock}/owner" -migration_lock_acquired=1 - -while IFS= read -r gate; do - [[ -n "${gate}" ]] || continue - source_evidence="${artifact}/${gate}.json" - source_log="${artifact}/${gate}.log" - [[ -f "${source_evidence}" && -f "${source_log}" ]] || continue - - lock_root="${artifact}/.locks" - lock="${lock_root}/${gate}.lock" - mkdir -p "${lock_root}" - while ! mkdir "${lock}" 2>/dev/null; do - if [[ -f "${lock}/owner" ]]; then - owner="$(cat "${lock}/owner" 2>/dev/null || true)" - if [[ "${owner}" =~ ^[0-9]+$ ]] && ! kill -0 "${owner}" 2>/dev/null; then - rm -f "${lock}/owner" - rmdir "${lock}" 2>/dev/null || true - continue - fi - fi - sleep 0.05 - done - printf '%s\n' "$$" >"${lock}/owner" - lock_acquired=1 - - current_digest="$( - "${root}/.golib/scripts/gate-input-digest.sh" "${gate}" "${module}" - )" - source_digest="$(jq -r '.input_digest // empty' "${source_evidence}")" - if [[ "${source_digest}" == "${current_digest}" ]]; then - cleanup_gate - continue - fi - - log_sha256="$(shasum -a 256 "${source_log}" | awk '{print $1}')" - jq -e \ - --arg module "${module}" \ - --arg gate "${gate}" \ - --arg source_digest "${source_digest}" \ - --arg log_sha256 "${log_sha256}" ' - .schema_version == 1 and - .module == $module and - .gate == $gate and - (.result == "passed" or .result == "not_applicable" or - ($gate == "nilaway" and .result == "advisory")) and - .exit_code == 0 and - ($source_digest | length) > 0 and - .input_digest == $source_digest and - .completed_input_digest == $source_digest and - .log_sha256 == $log_sha256 - ' "${source_evidence}" >/dev/null || { - cleanup_gate - continue - } - - legacy_digest="$( - GOLIB_GATE_INPUT_POLICY=legacy-api-baseline \ - "${root}/.golib/scripts/gate-input-digest.sh" "${gate}" "${module}" - )" - if [[ "${source_digest}" != "${legacy_digest}" ]]; then - cleanup_gate - continue - fi - - destination="${artifact}/by-input/${gate}" - destination_evidence="${destination}/${current_digest}.json" - destination_log="${destination}/${current_digest}.log" - temporary_evidence="${destination_evidence}.tmp.$$" - temporary_log="${destination_log}.tmp.$$" - temporary_legacy_evidence="${source_evidence}.tmp.$$" - temporary_legacy_log="${source_log}.tmp.$$" - mkdir -p "${destination}" - jq \ - --arg current_digest "${current_digest}" \ - --arg source_digest "${source_digest}" \ - --arg validated_revision "${validated_revision}" \ - --arg validated_at "${validated_at}" ' - .identity_lineage = (( - (.identity_lineage // []) + - [.input_digest] + - [(.identity_migration.previous_gate_input_digest // "")] - ) | map(select(length > 0)) | unique) - | .input_digest = $current_digest - | .completed_input_digest = $current_digest - | .revalidated_revision = $validated_revision - | .revalidated_at = $validated_at - | .identity_migration = { - reason: "non-semantic-gate-input-scope-narrowing", - previous_gate_input_digest: $source_digest - } - ' "${source_evidence}" >"${temporary_evidence}" - cp "${source_log}" "${temporary_log}" - mv "${temporary_log}" "${destination_log}" - mv "${temporary_evidence}" "${destination_evidence}" - cp "${destination_log}" "${temporary_legacy_log}" - cp "${destination_evidence}" "${temporary_legacy_evidence}" - mv "${temporary_legacy_log}" "${source_log}" - mv "${temporary_legacy_evidence}" "${source_evidence}" - cleanup_gate - printf '[%s] migrated content-identical %s evidence\n' "${module}" "${gate}" -done <"${root}/.golib/scripts/check-gates.txt" diff --git a/.golib/scripts/internal/migrate-runner-isolation-evidence.sh b/.golib/scripts/internal/migrate-runner-isolation-evidence.sh deleted file mode 100755 index f883567..0000000 --- a/.golib/scripts/internal/migrate-runner-isolation-evidence.sh +++ /dev/null @@ -1,168 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -lt 1 || $# -gt 2 ]]; then - printf 'usage: %s [gate]\n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -module="$1" -selected_gate="${2:-}" -artifact="${root}/.artifacts/${module}/evidence" -validated_revision="$(git rev-parse HEAD)" -validated_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" - -if [[ -n "${selected_gate}" ]]; then - gates="${selected_gate}" -else - gates="$(cat "${root}/.golib/scripts/check-gates.txt")" -fi - -while IFS= read -r gate; do - [[ -n "${gate}" ]] || continue - # Aggregate assurance intentionally fingerprints runner implementation. - # Process-isolation changes therefore invalidate rather than migrate it. - [[ "${gate}" != "operational-assurance" ]] || continue - current_digest="$( - "${root}/.golib/scripts/gate-input-digest.sh" "${gate}" "${module}" - )" - destination="${artifact}/by-input/${gate}" - destination_evidence="${destination}/${current_digest}.json" - destination_log="${destination}/${current_digest}.log" - if [[ -f "${destination_evidence}" && -f "${destination_log}" ]]; then - continue - fi - source_evidence="" - source_log="" - source_digest="" - for legacy_policy in \ - legacy-runner-isolation-post \ - legacy-runner-isolation; do - legacy_digest="$( - GOLIB_GATE_INPUT_POLICY="${legacy_policy}" \ - "${root}/.golib/scripts/gate-input-digest.sh" "${gate}" "${module}" - )" - [[ "${current_digest}" != "${legacy_digest}" ]] || continue - candidate_evidence="${artifact}/by-input/${gate}/${legacy_digest}.json" - candidate_log="${artifact}/by-input/${gate}/${legacy_digest}.log" - if [[ ! -f "${candidate_evidence}" || ! -f "${candidate_log}" ]]; then - candidate_evidence="${artifact}/${gate}.json" - candidate_log="${artifact}/${gate}.log" - fi - [[ -f "${candidate_evidence}" && -f "${candidate_log}" ]] || continue - candidate_digest="$(jq -r '.input_digest // empty' "${candidate_evidence}")" - [[ "${candidate_digest}" == "${legacy_digest}" ]] || continue - source_evidence="${candidate_evidence}" - source_log="${candidate_log}" - source_digest="${candidate_digest}" - break - done - [[ -n "${source_evidence}" ]] || continue - log_sha256="$(shasum -a 256 "${source_log}" | awk '{print $1}')" - jq -e \ - --arg module "${module}" \ - --arg gate "${gate}" \ - --arg source_digest "${source_digest}" \ - --arg log_sha256 "${log_sha256}" ' - .schema_version == 1 and - .module == $module and - .gate == $gate and - ( - .result == "passed" or - .result == "not_applicable" or - ($gate == "nilaway" and .result == "advisory") - ) and - .exit_code == 0 and - .input_digest == $source_digest and - .completed_input_digest == $source_digest and - .log_sha256 == $log_sha256 - ' "${source_evidence}" >/dev/null || continue - if [[ "$(jq -r '.result' "${source_evidence}")" == "advisory" ]] && - ! grep -Eq 'NilAway advisory exit status: [1-9][0-9]*$' "${source_log}"; then - continue - fi - if [[ "$(jq -r '.result' "${source_evidence}")" == "not_applicable" ]] && - ! grep -Fq ': not applicable by catalog policy' "${source_log}"; then - continue - fi - - lock_root="${artifact}/.locks" - lock="${lock_root}/${gate}.lock" - mkdir -p "${destination}" "${lock_root}" - while ! ln -s "$$" "${lock}" 2>/dev/null; do - if [[ -L "${lock}" ]]; then - owner="$(readlink "${lock}" 2>/dev/null || true)" - if [[ "${owner}" =~ ^[0-9]+$ ]] && ! kill -0 "${owner}" 2>/dev/null; then - rm -f "${lock}" - continue - fi - elif [[ -d "${lock}" ]]; then - owner="$(cat "${lock}/owner" 2>/dev/null || true)" - if [[ "${owner}" =~ ^[0-9]+$ ]] && ! kill -0 "${owner}" 2>/dev/null; then - rm -f "${lock}/owner" - rmdir "${lock}" 2>/dev/null || true - continue - fi - if [[ -z "${owner}" ]] && rmdir "${lock}" 2>/dev/null; then - continue - fi - elif [[ -f "${lock}" ]]; then - owner="$(cat "${lock}" 2>/dev/null || true)" - if [[ "${owner}" =~ ^[0-9]+$ ]] && ! kill -0 "${owner}" 2>/dev/null; then - rm -f "${lock}" - continue - fi - if [[ -z "${owner}" ]]; then - rm -f "${lock}" - continue - fi - fi - sleep 0.05 - done - - temporary_evidence="${destination_evidence}.tmp.$$" - temporary_log="${destination_log}.tmp.$$" - temporary_pointer_evidence="${artifact}/${gate}.json.tmp.$$" - temporary_pointer_log="${artifact}/${gate}.log.tmp.$$" - cleanup_gate() { - rm -f \ - "${temporary_evidence}" "${temporary_log}" \ - "${temporary_pointer_evidence}" "${temporary_pointer_log}" - if [[ "$(readlink "${lock}" 2>/dev/null || true)" == "$$" ]]; then - rm -f "${lock}" - fi - } - trap cleanup_gate EXIT - trap 'exit 130' HUP INT TERM - - jq \ - --arg current_digest "${current_digest}" \ - --arg source_digest "${source_digest}" \ - --arg validated_revision "${validated_revision}" \ - --arg validated_at "${validated_at}" ' - .identity_lineage = (( - (.identity_lineage // []) + - [.input_digest] + - [(.identity_migration.previous_gate_input_digest // "")] - ) | map(select(length > 0)) | unique) - | .input_digest = $current_digest - | .completed_input_digest = $current_digest - | .revalidated_revision = $validated_revision - | .revalidated_at = $validated_at - | .identity_migration = { - reason: "non-semantic-runner-isolation-scope-narrowing", - previous_gate_input_digest: $source_digest - } - ' "${source_evidence}" >"${temporary_evidence}" - cp "${source_log}" "${temporary_log}" - mv "${temporary_log}" "${destination_log}" - mv "${temporary_evidence}" "${destination_evidence}" - cp "${destination_log}" "${temporary_pointer_log}" - cp "${destination_evidence}" "${temporary_pointer_evidence}" - mv "${temporary_pointer_log}" "${artifact}/${gate}.log" - mv "${temporary_pointer_evidence}" "${artifact}/${gate}.json" - cleanup_gate - trap - EXIT HUP INT TERM - printf '[%s] migrated content-identical %s evidence\n' "${module}" "${gate}" -done <<<"${gates}" diff --git a/.golib/scripts/internal/mutation-command.sh b/.golib/scripts/internal/mutation-command.sh deleted file mode 100755 index 0068cb2..0000000 --- a/.golib/scripts/internal/mutation-command.sh +++ /dev/null @@ -1,27 +0,0 @@ -#!/usr/bin/env bash - -build_mutation_arguments() { - local target="$1" - local output="$2" - local tags="$3" - local discover_only="$4" - - mutation_arguments=( - unleash "${target}" - --integration --coverpkg "${target}" - --exclude-files '^.+/' - --workers 4 --test-cpu 1 --timeout-coefficient 10 - --threshold-efficacy 100 --threshold-mcover 100 - --arithmetic-base --conditionals-boundary --conditionals-negation - --invert-assignments --invert-bitwise --invert-bwassign - --increment-decrement --invert-logical --invert-loopctrl - --invert-negatives --remove-self-assignments - --output-statuses lctvsr --output "${output}" - ) - if [[ -n "${tags}" ]]; then - mutation_arguments+=(--tags "${tags}") - fi - if [[ "${discover_only}" -eq 1 ]]; then - mutation_arguments+=(--dry-run) - fi -} diff --git a/.golib/scripts/internal/mutation-coverage.sh b/.golib/scripts/internal/mutation-coverage.sh deleted file mode 100755 index ab02715..0000000 --- a/.golib/scripts/internal/mutation-coverage.sh +++ /dev/null @@ -1,31 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 2 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -profile="$1" -tags="$2" -started="$(date +%s)" -arguments=( - test - -count=1 - -timeout=20m - -cover - -coverpkg=./... - "-coverprofile=${profile}" -) -if [[ -n "${tags}" ]]; then - arguments+=("-tags=${tags}") -fi -arguments+=(./...) - -go "${arguments[@]}" >&2 -finished="$(date +%s)" -elapsed=$((finished - started)) -if [[ "${elapsed}" -lt 1 ]]; then - elapsed=1 -fi -printf '%ss\n' "${elapsed}" diff --git a/.golib/scripts/internal/mutation-scratch.sh b/.golib/scripts/internal/mutation-scratch.sh deleted file mode 100644 index e2dce95..0000000 --- a/.golib/scripts/internal/mutation-scratch.sh +++ /dev/null @@ -1,169 +0,0 @@ -#!/usr/bin/env bash - -mutation_scratch_process_start() { - LC_ALL=C ps -o lstart= -p "$1" 2>/dev/null | awk '{$1=$1; print}' -} - -mutation_scratch_owner_is_abandoned() { - local candidate="$1" - local expected_host="$2" - local owner_file owner_version owner_host owner_pid owner_start owner_run_id - local current_start - owner_file="${candidate}/.mutation-owner" - [[ -f "${owner_file}" && ! -L "${owner_file}" ]] || return 1 - IFS=$'\t' read -r owner_version owner_host owner_pid owner_start \ - owner_run_id <"${owner_file}" || return 1 - [[ "${owner_version}" == "1" && - "${owner_host}" == "${expected_host}" && - "${owner_pid}" =~ ^[1-9][0-9]*$ && - "${owner_start}" != "" && - "${owner_run_id}" == "$(basename "${candidate}")" ]] || return 1 - current_start="$(mutation_scratch_process_start "${owner_pid}")" - if [[ -n "${current_start}" ]]; then - [[ "${current_start}" != "${owner_start}" ]] - return - fi - if kill -0 "${owner_pid}" 2>/dev/null; then - return 1 - fi - return 0 -} - -mutation_scratch_recover_abandoned() { - local recovery_artifact="$1" - local recovery_host candidate claim - [[ -d "${recovery_artifact}" && ! -L "${recovery_artifact}" ]] || return 0 - recovery_host="$(hostname)" - while IFS= read -r candidate; do - case "${candidate}" in - "${recovery_artifact}"/mutation-run-*) ;; - *) continue ;; - esac - [[ "$(dirname "${candidate}")" == "${recovery_artifact}" && - -d "${candidate}" && ! -L "${candidate}" ]] || continue - mutation_scratch_owner_is_abandoned \ - "${candidate}" "${recovery_host}" || continue - claim="${candidate}/.mutation-recovery-claim" - mkdir "${claim}" 2>/dev/null || continue - if mutation_scratch_owner_is_abandoned \ - "${candidate}" "${recovery_host}"; then - find "${candidate}" -depth -delete - else - rmdir "${claim}" - fi - done < <( - find "${recovery_artifact}" -mindepth 1 -maxdepth 1 -type d \ - -name 'mutation-run-*' -print - ) -} - -mutation_scratch_remove_owned_run() { - local owner_file owner_version owner_host owner_pid owner_start owner_run_id - [[ -n "${artifact:-}" && -n "${run_directory:-}" ]] || return 0 - case "${run_directory}" in - "${artifact}"/mutation-run-*) ;; - *) - printf 'refusing to remove unexpected mutation run: %s\n' \ - "${run_directory}" >&2 - return 1 - ;; - esac - [[ "$(dirname "${run_directory}")" == "${artifact}" && - -d "${run_directory}" && ! -L "${run_directory}" ]] || return 0 - owner_file="${run_directory}/.mutation-owner" - if [[ -e "${owner_file}" || -L "${owner_file}" ]]; then - IFS=$'\t' read -r owner_version owner_host owner_pid owner_start \ - owner_run_id <"${owner_file}" || { - printf 'refusing to remove mutation run without a valid owner: %s\n' \ - "${run_directory}" >&2 - return 1 - } - if [[ "${owner_version}" != "1" || - "${owner_host}" != "${mutation_owner_host}" || - "${owner_pid}" != "$$" || - "${owner_start}" != "${mutation_owner_start}" || - "${owner_run_id}" != "$(basename "${run_directory}")" ]]; then - printf 'refusing to remove mutation run with a different owner: %s\n' \ - "${run_directory}" >&2 - return 1 - fi - elif [[ "${mutation_owner_run_id:-}" != \ - "$(basename "${run_directory}")" ]]; then - printf 'refusing to remove unmarked mutation run: %s\n' \ - "${run_directory}" >&2 - return 1 - fi - find "${run_directory}" -depth -delete - run_directory="" -} - -mutation_scratch_package_cache() { - local slug="$1" - case "${slug}" in - ""|"."|".."|*/*) - printf 'invalid mutation package cache slug: %s\n' "${slug}" >&2 - return 1 - ;; - esac - active_build_cache="$( - mktemp -d "${run_directory}/${slug}.go-cache-XXXXXXXX" - )" -} - -mutation_scratch_cleanup_package_cache() { - [[ -n "${active_build_cache:-}" ]] || return 0 - case "${active_build_cache}" in - "${run_directory}"/*.go-cache-*) ;; - *) - printf 'refusing to remove unexpected mutation cache: %s\n' \ - "${active_build_cache}" >&2 - return 1 - ;; - esac - [[ "$(dirname "${active_build_cache}")" == "${run_directory}" && - -d "${active_build_cache}" && - ! -L "${active_build_cache}" ]] || return 0 - find "${active_build_cache}" -depth -delete - active_build_cache="" -} - -mutation_scratch_on_exit() { - local status=$? - trap - EXIT HUP INT TERM - if ! mutation_scratch_remove_owned_run && [[ "${status}" -eq 0 ]]; then - status=1 - fi - exit "${status}" -} - -mutation_scratch_on_signal() { - exit "$1" -} - -mutation_scratch_install_traps() { - trap mutation_scratch_on_exit EXIT - trap 'mutation_scratch_on_signal 129' HUP - trap 'mutation_scratch_on_signal 130' INT - trap 'mutation_scratch_on_signal 143' TERM -} - -mutation_scratch_initialize() { - local owner_tmp - artifact="$1" - mkdir -p "${artifact}" - mutation_scratch_recover_abandoned "${artifact}" - run_directory="$(mktemp -d "${artifact}/mutation-run-XXXXXXXX")" - mutation_owner_run_id="$(basename "${run_directory}")" - mutation_scratch_install_traps - mutation_owner_host="$(hostname)" - mutation_owner_start="$(mutation_scratch_process_start "$$")" - [[ -n "${mutation_owner_start}" ]] || { - printf 'cannot identify mutation scratch owner process\n' >&2 - return 1 - } - owner_tmp="$(mktemp "${run_directory}/.mutation-owner.XXXXXXXX")" - printf '1\t%s\t%s\t%s\t%s\n' \ - "${mutation_owner_host}" "$$" "${mutation_owner_start}" \ - "$(basename "${run_directory}")" >"${owner_tmp}" - mv "${owner_tmp}" "${run_directory}/.mutation-owner" -} diff --git a/.golib/scripts/internal/optional-mutation-digest.sh b/.golib/scripts/internal/optional-mutation-digest.sh deleted file mode 100755 index 6e71fa3..0000000 --- a/.golib/scripts/internal/optional-mutation-digest.sh +++ /dev/null @@ -1,23 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 3 ]]; then - printf 'usage: %s \n' \ - "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -resolution="$1" -module="$2" -package="$3" -digest="" -if ! digest="$( - GOLIB_MUTATION_DIGEST_RESOLUTION="${resolution}" \ - "${root}/.golib/scripts/gate-input-digest.sh" \ - mutation "${module}" "${package}" 2>/dev/null -)"; then - exit 0 -fi - -printf '%s\n' "${digest}" diff --git a/.golib/scripts/internal/reuse-approved-mutation-checkpoint.sh b/.golib/scripts/internal/reuse-approved-mutation-checkpoint.sh deleted file mode 100755 index 7bd7437..0000000 --- a/.golib/scripts/internal/reuse-approved-mutation-checkpoint.sh +++ /dev/null @@ -1,118 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 9 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -ledger="$1" -checkpoint="$2" -module="$3" -package="$4" -current_input="$5" -tool_version="$6" -verifier_identity="$7" -validated_revision="$8" -output="$9" - -[[ -s "${ledger}" && -s "${checkpoint}" ]] || exit 1 - -report_digest="$({ jq -S -c '.report' "${checkpoint}" || exit 1; } | - shasum -a 256 | awk '{print $1}')" -execution_revision="$(jq -er '.execution_revision' "${checkpoint}")" -previous_input="$(jq -er '.gate_input_digest' "${checkpoint}")" -checkpoint_inputs="$(jq -c ' - ([.gate_input_digest] + - (.identity_lineage // []) + - [(.identity_migration.previous_gate_input_digest // "")]) - | map(select(length > 0)) - | unique -' "${checkpoint}")" - -jq -e \ - --arg module "${module}" \ - --arg package "${package}" \ - --arg execution_revision "${execution_revision}" \ - --arg tool_version "${tool_version}" \ - --arg verifier_identity "${verifier_identity}" \ - --arg report_digest "${report_digest}" \ - --argjson checkpoint_inputs "${checkpoint_inputs}" ' - .schema_version == 3 and - ([.verifier_migrations[]? | select( - .module == $module and - .package == $package and - .gremlins_verifier_sha256 == $verifier_identity and - .execution_revision == $execution_revision and - (.gate_input_digest as $approved_input | - ($checkpoint_inputs | index($approved_input)) != null) and - .gremlins_version == $tool_version and - .report_sha256 == $report_digest - )] | length == 1) - ' "${ledger}" >/dev/null - -jq -e \ - --arg module "${module}" \ - --arg package "${package}" \ - --arg tool_version "${tool_version}" \ - --arg verifier_identity "${verifier_identity}" ' - .schema_version == 3 and - .module == $module and - .package == $package and - .gremlins_version == $tool_version and - ((.gremlins_verifier_sha256 // $verifier_identity) == $verifier_identity) and - (.report.files | type == "array") and - ([.report.files[].mutations[]? | select(.status != "KILLED")] | length == 0) - ' "${checkpoint}" >/dev/null - -if [[ "${previous_input}" == "${current_input}" ]]; then - jq \ - --arg validated_revision "${validated_revision}" \ - --arg verifier_identity "${verifier_identity}" ' - .validated_revision = $validated_revision - | .gremlins_verifier_sha256 = $verifier_identity - | .verifier_identity_source = "approved-semantic-migration" - ' "${checkpoint}" >"${output}" - exit 0 -fi - -jq -e \ - --arg module "${module}" \ - --arg package "${package}" \ - --arg execution_revision "${execution_revision}" \ - --arg previous_input "${previous_input}" \ - --arg current_input "${current_input}" \ - --arg tool_version "${tool_version}" \ - --arg verifier_identity "${verifier_identity}" \ - --arg report_digest "${report_digest}" ' - .schema_version == 3 and - ([.entries[] | select( - .module == $module and - .package == $package and - .execution_revision == $execution_revision and - .gate_input_digest == $previous_input and - .replacement_gate_input_digest == $current_input and - .gremlins_version == $tool_version and - ((.gremlins_verifier_sha256 // $verifier_identity) == $verifier_identity) and - .report_sha256 == $report_digest - )] | length == 1) - ' "${ledger}" >/dev/null - -jq \ - --arg validated_revision "${validated_revision}" \ - --arg current_input "${current_input}" \ - --arg verifier_identity "${verifier_identity}" ' - .identity_lineage = (( - (.identity_lineage // []) + - [.gate_input_digest] + - [(.identity_migration.previous_gate_input_digest // "")] - ) | map(select(length > 0)) | unique) - | .validated_revision = $validated_revision - | .gremlins_verifier_sha256 = $verifier_identity - | .verifier_identity_source = "approved-semantic-migration" - | .identity_migration = { - reason: "approved-input-identity-migration", - previous_gate_input_digest: .gate_input_digest - } - | .gate_input_digest = $current_input - ' "${checkpoint}" >"${output}" diff --git a/.golib/scripts/internal/reuse-mutation-coverage.sh b/.golib/scripts/internal/reuse-mutation-coverage.sh deleted file mode 100755 index c1a0c83..0000000 --- a/.golib/scripts/internal/reuse-mutation-coverage.sh +++ /dev/null @@ -1,42 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 5 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -profile="$1" -identity="$2" -destination="$3" -expected_input_digest="$4" -expected_test_tags="$5" - -[[ -s "${profile}" && -s "${identity}" ]] || exit 1 - -profile_sha256="$(shasum -a 256 "${profile}" | awk '{print $1}')" -elapsed="$(jq -er \ - --arg input_digest "${expected_input_digest}" \ - --arg test_tags "${expected_test_tags}" \ - --arg profile_sha256 "${profile_sha256}" ' - select( - .schema_version == 1 and - .input_digest == $input_digest and - .test_tags == $test_tags and - .profile_sha256 == $profile_sha256 and - (.elapsed | type == "string") and - (.elapsed | test("^[1-9][0-9]*s$")) - ) - | .elapsed - ' "${identity}")" || exit 1 - -temporary="$(mktemp "${destination}.tmp.XXXXXX")" -cleanup() { - rm -f "${temporary}" -} -trap cleanup EXIT HUP INT TERM -cp "${profile}" "${temporary}" -copied_sha256="$(shasum -a 256 "${temporary}" | awk '{print $1}')" -[[ "${copied_sha256}" == "${profile_sha256}" ]] || exit 1 -mv "${temporary}" "${destination}" -printf '%s\n' "${elapsed}" diff --git a/.golib/scripts/internal/run-mutation.sh b/.golib/scripts/internal/run-mutation.sh deleted file mode 100755 index ac46b58..0000000 --- a/.golib/scripts/internal/run-mutation.sh +++ /dev/null @@ -1,686 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 2 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -mode="$1" -module="$2" -if [[ -n "${GOLIB_REAL_GO:-}" ]]; then - PATH="$(dirname "${GOLIB_REAL_GO}"):${PATH}" - export PATH -fi -if [[ "${mode}" != "enforce" && "${mode}" != "discover" ]]; then - printf 'mutation mode must be enforce or discover\n' >&2 - exit 2 -fi -discover_only=0 -if [[ "${mode}" == "discover" ]]; then - discover_only=1 -fi -directory="${root}/${module}" -artifact="${root}/.artifacts/${module}" -report="${artifact}/mutation.json" -checkpoint_directory="${artifact}/mutation-checkpoints" -history_migrations="${root}/.golib/mutation-history-migrations.json" -mkdir -p "${checkpoint_directory}" -active_build_cache="" -# shellcheck disable=SC1091 -source "${root}/.golib/scripts/internal/mutation-scratch.sh" -mutation_scratch_initialize "${artifact}" -mutation_arguments=() -execution_revision="$(git -C "${root}" rev-parse HEAD)" - -# shellcheck disable=SC1091 -source "${root}/.golib/versions.env" -# shellcheck disable=SC1091 -source "${root}/.golib/scripts/internal/mutation-command.sh" -# shellcheck disable=SC1091 -source "${root}/.golib/scripts/internal/configure-mutation-workers.sh" -gremlins_binary="$("${root}/.golib/scripts/build-golib-gremlins.sh")" -gremlins_binary_sha256="$(shasum -a 256 "${gremlins_binary}" | awk '{print $1}')" -gremlins_verifier_sha256="$("${root}/.golib/scripts/mutation-verifier-identity.sh")" -environment_identity="$(go env -json GOVERSION GOOS GOARCH CGO_ENABLED)" -legacy_gate_input_digest="" -# run_directory is initialized by mutation_scratch_initialize. -# shellcheck disable=SC2154 -shared_coverage="${run_directory}/integration.coverage" -shared_coverage_elapsed="" -modfile="" -owned_count="$(jq -r --arg directory "${module}" ' - .modules[] | select(.directory == $directory) | .owned_dependencies | length -' "${root}/modules.json")" -if [[ "${owned_count}" -gt 0 ]]; then - modfile="${run_directory}/golib-mutation.mod" - cp "${directory}/go.mod" "${modfile}" - if [[ -f "${directory}/go.sum" ]]; then - cp "${directory}/go.sum" "${modfile%.mod}.sum" - fi - while IFS=$'\t' read -r owned_path owned_directory; do - [[ -n "${owned_path}" && -n "${owned_directory}" ]] || continue - GOWORK=off go mod edit -modfile="${modfile}" \ - -replace="${owned_path}=${root}/${owned_directory}" - done < <(jq -r '.modules[] - | select(.directory != "." and .kind != "fixture") - | [.module_path, .directory] - | @tsv' "${root}/modules.json") -fi -cd "${directory}" - -tags="$(jq -r --arg directory "${module}" \ - '.modules[] | select(.directory == $directory) - | .test_tags | map(select(. != "interoperability")) | join(",")' \ - "${root}/modules.json")" -mutation_workers=4 -if grep -Eq \ - 'github\.com/testcontainers/testcontainers-go([/[:space:]])' \ - "${directory}/go.mod"; then - mutation_workers=1 -fi -mutation_environment=(env) -if [[ "${module}" == "pkg/ecma-regexp" ]]; then - mutation_environment+=( -u TEST262_ROOT) -fi -packages=() -while IFS= read -r package_directory; do - [[ -n "${package_directory}" ]] && packages+=("${package_directory}") -done < <(jq -r --arg directory "${module}" ' - .modules[] - | select(.directory == $directory) - | .packages[] - | select(.coverage_required == true) - | .directory -' "${root}/modules.json") -[[ "${#packages[@]}" -gt 0 ]] || { - printf 'mutation package selection is empty for %s\n' "${module}" >&2 - exit 1 -} - -expected_packages_file="${run_directory}/expected-packages.json" -printf '%s\n' "${packages[@]}" | jq -R -s \ - 'split("\n") | map(select(length > 0))' >"${expected_packages_file}" -reports=() - -if [[ "${discover_only}" -eq 0 ]]; then - # A new enforced campaign supersedes any older aggregate immediately. - rm -f "${report}" -fi - -ensure_shared_coverage() { - if [[ -s "${shared_coverage}" && -n "${shared_coverage_elapsed}" ]]; then - return - fi - coverage_profile="${artifact}/coverage.out" - coverage_identity="${artifact}/coverage-profile.json" - coverage_input_digest="$( - "${root}/.golib/scripts/gate-input-digest.sh" coverage "${module}" - )" - if shared_coverage_elapsed="$( - "${root}/.golib/scripts/internal/reuse-mutation-coverage.sh" \ - "${coverage_profile}" \ - "${coverage_identity}" \ - "${shared_coverage}" \ - "${coverage_input_digest}" \ - "${tags}" \ - 2>/dev/null - )"; then - printf '[%s] reused content-bound coverage for mutation\n' "${module}" - return - fi - shared_coverage_elapsed="" - if [[ -n "${modfile}" ]]; then - shared_coverage_elapsed="$( - GOWORK=off GOFLAGS="-modfile=${modfile} -mod=mod" \ - "${root}/.golib/scripts/internal/mutation-coverage.sh" \ - "${shared_coverage}" "${tags}" - )" - else - shared_coverage_elapsed="$( - GOWORK=off "${root}/.golib/scripts/internal/mutation-coverage.sh" \ - "${shared_coverage}" "${tags}" - )" - fi - [[ -s "${shared_coverage}" ]] || { - printf 'shared mutation coverage profile is missing for %s\n' \ - "${module}" >&2 - exit 1 - } -} - -write_aggregate() { - local aggregate_tmp validated_revision - aggregate_tmp="$(mktemp "${report}.tmp.XXXXXX")" - validated_revision="$(git -C "${root}" rev-parse HEAD)" - jq -s \ - --arg module "${module}" \ - --arg validated_revision "${validated_revision}" \ - --slurpfile expected "${expected_packages_file}" ' - { - schema_version: 3, - module: $module, - execution_revisions: ([.[].execution_revision] | unique), - validated_revision: $validated_revision, - gate_input_digests: ( - map({key: .package, value: .gate_input_digest}) - | from_entries - ), - gremlins_versions: ([.[].gremlins_version] | unique), - gremlins_verifier_sha256s: ([.[].gremlins_verifier_sha256] | unique), - verifier_identity_sources: ([.[].verifier_identity_source] | unique), - gremlins_binary_sha256s: ( - [.[].gremlins_binary_sha256 | select(. != null)] | unique - ), - environments: ([.[].environment] | unique), - expected_packages: $expected[0], - completed_packages: [.[].package], - complete: (length == ($expected[0] | length)), - packages: . - } - ' "${reports[@]}" >"${aggregate_tmp}" - mv "${aggregate_tmp}" "${report}" -} - -checkpoint_verifier_source_is_valid() { - jq -e ' - ( - .verifier_identity_source == "executed" and - (.gremlins_binary_sha256 | type == "string") and - (.gremlins_binary_sha256 | test("^[0-9a-f]{64}$")) - ) or ( - .verifier_identity_source == "approved-semantic-migration" and - ( - (.gremlins_binary_sha256 // null) == null or - ( - (.gremlins_binary_sha256 | type == "string") and - (.gremlins_binary_sha256 | test("^[0-9a-f]{64}$")) - ) - ) - ) - ' "$1" >/dev/null -} - -for package_directory in "${packages[@]}"; do - target="." - slug="root" - if [[ "${package_directory}" != "." ]]; then - target="./${package_directory}" - slug="${package_directory//\//-}" - fi - package_report="${run_directory}/${slug}.json" - checkpoint="${checkpoint_directory}/${slug}.json" - package_input_digest="$( - "${root}/.golib/scripts/gate-input-digest.sh" \ - mutation "${module}" "${package_directory}" - )" - build_mutation_arguments \ - "${target}" "${package_report}" "${tags}" "${discover_only}" - configure_mutation_workers "${mutation_workers}" - - reviewed_zero_mutant() { - local source_directory source_digest reviewed - source_directory="${module}" - if [[ "${package_directory}" != "." ]]; then - source_directory="${module}/${package_directory}" - fi - source_digest="$("${root}/.golib/scripts/package-source-digest.sh" \ - "${source_directory}")" - reviewed="$(jq --arg module "${module}" \ - --arg package "${package_directory}" \ - --arg digest "${source_digest}" \ - --arg version "${GREMLINS_VERSION}" \ - --arg tool_identity "${gremlins_verifier_sha256}" ' - [.packages[] | select( - .module_directory == $module and - .package_directory == $package and - .source_digest == $digest and - .gremlins_version == $version and - .gremlins_verifier_sha256 == $tool_identity - )] | length - ' "${root}/.golib/mutation-zero-inventory.json")" - [[ "${reviewed}" -eq 1 ]] - } - - if [[ "${discover_only}" -eq 0 && -s "${checkpoint}" ]] && - jq -e \ - --arg module "${module}" \ - --arg package "${package_directory}" \ - --arg digest "${package_input_digest}" \ - --arg version "${GREMLINS_VERSION}" \ - --arg tool_identity "${gremlins_verifier_sha256}" ' - .schema_version == 3 and - .module == $module and - .package == $package and - .gate_input_digest == $digest and - .gremlins_version == $version and - .gremlins_verifier_sha256 == $tool_identity and - (.report.files | type == "array") and - ([.report.files[].mutations[]? | select(.status != "KILLED")] | length == 0) - ' "${checkpoint}" >/dev/null && - checkpoint_verifier_source_is_valid "${checkpoint}"; then - checkpoint_total="$( - jq '[.report.files[].mutations[]?] | length' "${checkpoint}" - )" - if [[ "${checkpoint_total}" -gt 0 ]] || reviewed_zero_mutant; then - checkpoint_tmp="$(mktemp "${checkpoint}.tmp.XXXXXX")" - jq --arg revision "$(git -C "${root}" rev-parse HEAD)" \ - '.validated_revision = $revision' \ - "${checkpoint}" >"${checkpoint_tmp}" - mv "${checkpoint_tmp}" "${checkpoint}" - reports+=("${checkpoint}") - write_aggregate - printf '[%s] %s reused content-identical mutation evidence\n' \ - "${module}" "${target}" - continue - fi - fi - - observer_v1_package_digest="$( - "${root}/.golib/scripts/internal/optional-mutation-digest.sh" \ - observer-v1 "${module}" "${package_directory}" - )" - if [[ "${discover_only}" -eq 0 && - -n "${observer_v1_package_digest}" && -s "${checkpoint}" ]] && - jq -e \ - --arg module "${module}" \ - --arg package "${package_directory}" \ - --arg digest "${observer_v1_package_digest}" \ - --arg version "${GREMLINS_VERSION}" \ - --arg tool_identity "${gremlins_verifier_sha256}" ' - .schema_version == 3 and - .module == $module and - .package == $package and - (( - [.gate_input_digest] + - (.identity_lineage // []) + - [(.identity_migration.previous_gate_input_digest // "")] - ) | index($digest)) != null and - .gremlins_version == $version and - .gremlins_verifier_sha256 == $tool_identity and - (.report.files | type == "array") and - ([.report.files[].mutations[]? | select(.status != "KILLED")] | length == 0) - ' "${checkpoint}" >/dev/null && - checkpoint_verifier_source_is_valid "${checkpoint}"; then - checkpoint_total="$( - jq '[.report.files[].mutations[]?] | length' "${checkpoint}" - )" - if [[ "${checkpoint_total}" -gt 0 ]] || reviewed_zero_mutant; then - checkpoint_tmp="$(mktemp "${checkpoint}.tmp.XXXXXX")" - jq \ - --arg revision "$(git -C "${root}" rev-parse HEAD)" \ - --arg digest "${package_input_digest}" ' - .identity_lineage = (( - (.identity_lineage // []) + - [.gate_input_digest] + - [(.identity_migration.previous_gate_input_digest // "")] - ) | map(select(length > 0)) | unique) - | .validated_revision = $revision - | .identity_migration = { - reason: "dependency-test-isolation", - previous_gate_input_digest: .gate_input_digest - } - | .gate_input_digest = $digest - ' "${checkpoint}" >"${checkpoint_tmp}" - mv "${checkpoint_tmp}" "${checkpoint}" - reports+=("${checkpoint}") - write_aggregate - printf '[%s] %s migrated dependency-test-isolated mutation identity\n' \ - "${module}" "${target}" - continue - fi - fi - - legacy_stable_package_digest="$( - "${root}/.golib/scripts/internal/optional-mutation-digest.sh" \ - legacy-stable "${module}" "${package_directory}" - )" - if [[ "${discover_only}" -eq 0 && - -n "${legacy_stable_package_digest}" && -s "${checkpoint}" ]] && - jq -e \ - --arg module "${module}" \ - --arg package "${package_directory}" \ - --arg digest "${legacy_stable_package_digest}" \ - --arg version "${GREMLINS_VERSION}" \ - --arg tool_identity "${gremlins_verifier_sha256}" ' - .schema_version == 3 and - .module == $module and - .package == $package and - (( - [.gate_input_digest] + - (.identity_lineage // []) + - [(.identity_migration.previous_gate_input_digest // "")] - ) | index($digest)) != null and - .gremlins_version == $version and - .gremlins_verifier_sha256 == $tool_identity and - (.report.files | type == "array") and - ([.report.files[].mutations[]? | select(.status != "KILLED")] | length == 0) - ' "${checkpoint}" >/dev/null && - checkpoint_verifier_source_is_valid "${checkpoint}"; then - checkpoint_total="$( - jq '[.report.files[].mutations[]?] | length' "${checkpoint}" - )" - if [[ "${checkpoint_total}" -gt 0 ]] || reviewed_zero_mutant; then - checkpoint_tmp="$(mktemp "${checkpoint}.tmp.XXXXXX")" - jq \ - --arg revision "$(git -C "${root}" rev-parse HEAD)" \ - --arg digest "${package_input_digest}" ' - .identity_lineage = (( - (.identity_lineage // []) + - [.gate_input_digest] + - [(.identity_migration.previous_gate_input_digest // "")] - ) | map(select(length > 0)) | unique) - | .validated_revision = $revision - | .identity_migration = { - reason: "target-observer-resolution", - previous_gate_input_digest: .gate_input_digest - } - | .gate_input_digest = $digest - ' "${checkpoint}" >"${checkpoint_tmp}" - mv "${checkpoint_tmp}" "${checkpoint}" - reports+=("${checkpoint}") - write_aggregate - printf '[%s] %s migrated module-wide mutation identity\n' \ - "${module}" "${target}" - continue - fi - fi - - legacy_package_digest="$( - "${root}/.golib/scripts/internal/optional-mutation-digest.sh" \ - caller "${module}" "${package_directory}" - )" - if [[ "${discover_only}" -eq 0 && - -n "${legacy_package_digest}" && -s "${checkpoint}" ]] && - jq -e \ - --arg module "${module}" \ - --arg package "${package_directory}" \ - --arg digest "${legacy_package_digest}" \ - --arg version "${GREMLINS_VERSION}" \ - --arg tool_identity "${gremlins_verifier_sha256}" ' - .schema_version == 3 and - .module == $module and - .package == $package and - (( - [.gate_input_digest] + - (.identity_lineage // []) + - [(.identity_migration.previous_gate_input_digest // "")] - ) | index($digest)) != null and - .gremlins_version == $version and - .gremlins_verifier_sha256 == $tool_identity and - (.report.files | type == "array") and - ([.report.files[].mutations[]? | select(.status != "KILLED")] | length == 0) - ' "${checkpoint}" >/dev/null && - checkpoint_verifier_source_is_valid "${checkpoint}"; then - checkpoint_total="$( - jq '[.report.files[].mutations[]?] | length' "${checkpoint}" - )" - if [[ "${checkpoint_total}" -gt 0 ]] || reviewed_zero_mutant; then - checkpoint_tmp="$(mktemp "${checkpoint}.tmp.XXXXXX")" - jq \ - --arg revision "$(git -C "${root}" rev-parse HEAD)" \ - --arg digest "${package_input_digest}" ' - .identity_lineage = (( - (.identity_lineage // []) + - [.gate_input_digest] + - [(.identity_migration.previous_gate_input_digest // "")] - ) | map(select(length > 0)) | unique) - | .validated_revision = $revision - | .identity_migration = { - reason: "canonical-workspace-resolution", - previous_gate_input_digest: .gate_input_digest - } - | .gate_input_digest = $digest - ' "${checkpoint}" >"${checkpoint_tmp}" - mv "${checkpoint_tmp}" "${checkpoint}" - reports+=("${checkpoint}") - write_aggregate - printf '[%s] %s migrated caller-dependent mutation identity\n' \ - "${module}" "${target}" - continue - fi - fi - - if [[ "${discover_only}" -eq 0 && -s "${checkpoint}" ]] && - jq -e \ - --arg module "${module}" \ - --arg package "${package_directory}" \ - --arg version "${GREMLINS_VERSION}" \ - --arg tool_identity "${gremlins_verifier_sha256}" ' - .schema_version == 2 and - .module == $module and - .package == $package and - .gremlins_version == $version and - .gremlins_verifier_sha256 == $tool_identity and - (.report.files | type == "array") and - ([.report.files[].mutations[]? | select(.status != "KILLED")] | length == 0) - ' "${checkpoint}" >/dev/null && - checkpoint_verifier_source_is_valid "${checkpoint}"; then - if [[ -z "${legacy_gate_input_digest}" ]]; then - legacy_gate_input_digest="$( - "${root}/.golib/scripts/gate-input-digest.sh" \ - mutation-legacy "${module}" - )" - fi - if jq -e --arg digest "${legacy_gate_input_digest}" \ - '.gate_input_digest == $digest' "${checkpoint}" >/dev/null; then - checkpoint_total="$( - jq '[.report.files[].mutations[]?] | length' "${checkpoint}" - )" - if [[ "${checkpoint_total}" -gt 0 ]] || reviewed_zero_mutant; then - checkpoint_tmp="$(mktemp "${checkpoint}.tmp.XXXXXX")" - jq \ - --arg revision "$(git -C "${root}" rev-parse HEAD)" \ - --arg digest "${package_input_digest}" ' - .schema_version = 3 - | .validated_revision = $revision - | .legacy_module_gate_input_digest = .gate_input_digest - | .gate_input_digest = $digest - ' "${checkpoint}" >"${checkpoint_tmp}" - mv "${checkpoint_tmp}" "${checkpoint}" - reports+=("${checkpoint}") - write_aggregate - printf '[%s] %s migrated content-identical mutation evidence\n' \ - "${module}" "${target}" - continue - fi - fi - fi - - if [[ "${discover_only}" -eq 0 && -s "${checkpoint}" && - -s "${history_migrations}" ]]; then - checkpoint_tmp="$(mktemp "${checkpoint}.tmp.XXXXXX")" - if "${root}/.golib/scripts/internal/reuse-approved-mutation-checkpoint.sh" \ - "${history_migrations}" \ - "${checkpoint}" \ - "${module}" \ - "${package_directory}" \ - "${package_input_digest}" \ - "${GREMLINS_VERSION}" \ - "${gremlins_verifier_sha256}" \ - "$(git -C "${root}" rev-parse HEAD)" \ - "${checkpoint_tmp}"; then - checkpoint_total="$( - jq '[.report.files[].mutations[]?] | length' \ - "${checkpoint_tmp}" - )" - if [[ "${checkpoint_total}" -gt 0 ]] || reviewed_zero_mutant; then - mv "${checkpoint_tmp}" "${checkpoint}" - reports+=("${checkpoint}") - write_aggregate - printf '[%s] %s reused approved content-identical mutation evidence\n' \ - "${module}" "${target}" - continue - fi - fi - rm -f "${checkpoint_tmp}" - fi - - printf '[%s] mutation package %s\n' "${module}" "${target}" - mutation_scratch_package_cache "${slug}" - ensure_shared_coverage - status=0 - if [[ "${discover_only}" -eq 1 ]]; then - if [[ -n "${modfile}" ]]; then - "${mutation_environment[@]}" \ - GOCACHE="${active_build_cache}" GOWORK=off \ - GOLIB_GREMLINS_COVERAGE_PROFILE="${shared_coverage}" \ - GOLIB_GREMLINS_COVERAGE_ELAPSED="${shared_coverage_elapsed}" \ - GOFLAGS="-modfile=${modfile} -mod=mod" \ - "${gremlins_binary}" "${mutation_arguments[@]}" \ - >"${run_directory}/${slug}.log" 2>&1 || status=$? - else - "${mutation_environment[@]}" \ - GOCACHE="${active_build_cache}" GOWORK=off \ - GOLIB_GREMLINS_COVERAGE_PROFILE="${shared_coverage}" \ - GOLIB_GREMLINS_COVERAGE_ELAPSED="${shared_coverage_elapsed}" \ - "${gremlins_binary}" "${mutation_arguments[@]}" \ - >"${run_directory}/${slug}.log" 2>&1 || status=$? - fi - elif [[ -n "${modfile}" ]]; then - "${mutation_environment[@]}" \ - GOCACHE="${active_build_cache}" GOWORK=off \ - GOLIB_GREMLINS_COVERAGE_PROFILE="${shared_coverage}" \ - GOLIB_GREMLINS_COVERAGE_ELAPSED="${shared_coverage_elapsed}" \ - GOFLAGS="-modfile=${modfile} -mod=mod" \ - "${gremlins_binary}" "${mutation_arguments[@]}" || status=$? - else - "${mutation_environment[@]}" \ - GOCACHE="${active_build_cache}" GOWORK=off \ - GOLIB_GREMLINS_COVERAGE_PROFILE="${shared_coverage}" \ - GOLIB_GREMLINS_COVERAGE_ELAPSED="${shared_coverage_elapsed}" \ - "${gremlins_binary}" "${mutation_arguments[@]}" || status=$? - fi - mutation_scratch_cleanup_package_cache - if [[ "${status}" -ne 0 ]]; then - if [[ "${discover_only}" -eq 1 ]]; then - cat "${run_directory}/${slug}.log" >&2 - fi - printf 'mutation tool failed for %s %s (status %s)\n' \ - "${module}" "${target}" "${status}" >&2 - exit 1 - fi - if [[ "${discover_only}" -eq 1 && ! -s "${package_report}" ]]; then - source_directory="${module}" - if [[ "${package_directory}" != "." ]]; then - source_directory="${module}/${package_directory}" - fi - source_digest="$("${root}/.golib/scripts/package-source-digest.sh" \ - "${source_directory}")" - printf 'ZERO\t%s\t%s\t%s\n' \ - "${module}" "${package_directory}" "${source_digest}" - continue - fi - if [[ ! -s "${package_report}" ]]; then - if reviewed_zero_mutant; then - printf '{"files":[]}\n' >"${package_report}" - else - printf 'mutation report is missing for %s %s\n' \ - "${module}" "${target}" >&2 - exit 1 - fi - fi - if ! jq -e '.files | type == "array"' "${package_report}" >/dev/null; then - printf 'mutation report is malformed for %s %s\n' "${module}" "${target}" >&2 - exit 1 - fi - total="$(jq '[.files[].mutations[]] | length' "${package_report}")" - if [[ "${discover_only}" -eq 1 ]]; then - printf 'MUTANTS\t%s\t%s\t%s\n' \ - "${module}" "${package_directory}" "${total}" - continue - fi - if [[ "${total}" -eq 0 ]]; then - if ! reviewed_zero_mutant; then - printf 'mutation report unexpectedly contains no reviewed mutants for %s %s\n' \ - "${module}" "${target}" >&2 - exit 1 - fi - fi - unexpected="$(jq '[.files[].mutations[] | select(.status != "KILLED")] | length' \ - "${package_report}")" - if [[ "${status}" -ne 0 || "${unexpected}" -ne 0 ]]; then - jq -r '.files[] as $file | $file.mutations[] | select(.status != "KILLED") | "\($file.file_name):\(.line):\(.column) \(.type) \(.status)"' \ - "${package_report}" >&2 - printf 'mutation gate failed for %s %s (tool status %s)\n' \ - "${module}" "${target}" "${status}" >&2 - exit 1 - fi - current_input_digest="$( - "${root}/.golib/scripts/gate-input-digest.sh" \ - mutation "${module}" "${package_directory}" - )" - if [[ "${current_input_digest}" != "${package_input_digest}" ]]; then - printf 'mutation inputs changed while running %s %s\n' \ - "${module}" "${target}" >&2 - exit 1 - fi - checkpoint_tmp="$(mktemp "${checkpoint}.tmp.XXXXXX")" - jq \ - --arg module "${module}" \ - --arg package "${package_directory}" \ - --arg execution_revision "${execution_revision}" \ - --arg validated_revision "$(git -C "${root}" rev-parse HEAD)" \ - --arg gate_input_digest "${package_input_digest}" \ - --arg gremlins_version "${GREMLINS_VERSION}" \ - --arg gremlins_verifier_sha256 "${gremlins_verifier_sha256}" \ - --arg gremlins_binary_sha256 "${gremlins_binary_sha256}" \ - --argjson environment "${environment_identity}" ' - { - schema_version: 3, - module: $module, - package: $package, - execution_revision: $execution_revision, - validated_revision: $validated_revision, - gate_input_digest: $gate_input_digest, - gremlins_version: $gremlins_version, - gremlins_verifier_sha256: $gremlins_verifier_sha256, - verifier_identity_source: "executed", - gremlins_binary_sha256: $gremlins_binary_sha256, - environment: $environment, - report: . - } - ' "${package_report}" >"${checkpoint_tmp}" - mv "${checkpoint_tmp}" "${checkpoint}" - reports+=("${checkpoint}") - write_aggregate - if [[ "${total}" -eq 0 ]]; then - printf '[%s] %s has a hash-pinned zero-viable-mutant review\n' \ - "${module}" "${target}" - else - printf '[%s] %s killed %s/%s viable mutants\n' \ - "${module}" "${target}" "${total}" "${total}" - fi -done - -if [[ "${discover_only}" -eq 1 ]]; then - exit 0 -fi - -if ! jq -e \ - ' - .schema_version == 3 and - .complete == true and - (.gremlins_verifier_sha256s | length == 1) and - ([.packages[] | select(.schema_version != 3)] | length == 0) and - all(.packages[]; - ( - .verifier_identity_source == "executed" and - (.gremlins_binary_sha256 | type == "string") and - (.gremlins_binary_sha256 | test("^[0-9a-f]{64}$")) - ) or ( - .verifier_identity_source == "approved-semantic-migration" and - ( - (.gremlins_binary_sha256 // null) == null or - ( - (.gremlins_binary_sha256 | type == "string") and - (.gremlins_binary_sha256 | test("^[0-9a-f]{64}$")) - ) - ) - ) - ) - ' "${report}" >/dev/null; then - printf 'aggregate mutation report is incomplete for %s\n' "${module}" >&2 - exit 1 -fi diff --git a/.golib/scripts/internal/run-verification-snapshots.sh b/.golib/scripts/internal/run-verification-snapshots.sh deleted file mode 100755 index 59de60e..0000000 --- a/.golib/scripts/internal/run-verification-snapshots.sh +++ /dev/null @@ -1,137 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 4 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$1" -gate="$2" -jobs="$3" -selection="$4" -snapshot_parents=() -snapshot_pids=() -snapshot_process_groups=() -lane_files=() -snapshot_processes_complete=0 -cleanup_started=0 - -# shellcheck disable=SC2329 # Called from the EXIT cleanup trap. -terminate_snapshot_processes() { - local attempt group running - for group in "${snapshot_process_groups[@]}"; do - kill -TERM -- "-${group}" 2>/dev/null || true - done - attempt=0 - while [[ "${attempt}" -lt 100 ]]; do - running=0 - for group in "${snapshot_process_groups[@]}"; do - if kill -0 -- "-${group}" 2>/dev/null; then - running=1 - break - fi - done - [[ "${running}" -eq 1 ]] || break - sleep 0.05 - attempt=$((attempt + 1)) - done - for group in "${snapshot_process_groups[@]}"; do - if kill -0 -- "-${group}" 2>/dev/null; then - kill -KILL -- "-${group}" 2>/dev/null || true - fi - done -} - -# shellcheck disable=SC2329 # Invoked by the EXIT trap. -cleanup_snapshots() { - local status=$? lane_file parent pid - if [[ "${cleanup_started}" -eq 1 ]]; then - return "${status}" - fi - cleanup_started=1 - trap '' HUP INT TERM - if [[ "${snapshot_processes_complete}" -eq 0 ]]; then - terminate_snapshot_processes - fi - for pid in "${snapshot_pids[@]}"; do - wait "${pid}" 2>/dev/null || true - done - for parent in "${snapshot_parents[@]}"; do - if [[ -d "${parent}" ]]; then - find "${parent}" -depth -delete - fi - done - for lane_file in "${lane_files[@]}"; do - rm -f "${lane_file}" - done - return "${status}" -} - -trap cleanup_snapshots EXIT -trap 'exit 129' HUP -trap 'exit 130' INT -trap 'exit 143' TERM - -lane=0 -while [[ "${lane}" -lt "${jobs}" ]]; do - lane_files+=("$(mktemp "${TMPDIR:-/tmp}/golib-lane.XXXXXX")") - lane=$((lane + 1)) -done -lane=0 -while IFS= read -r module; do - [[ -n "${module}" ]] || continue - printf '%s\n' "${module}" >>"${lane_files[${lane}]}" - lane=$(((lane + 1) % jobs)) -done <<<"${selection}" - -printf 'parallel-safe verification snapshot jobs=%s\n' "${jobs}" -# Non-interactive job control gives every background lane its own process group. -set -m -lane=0 -while [[ "${lane}" -lt "${jobs}" ]]; do - snapshot_parent="$( - mktemp -d "${TMPDIR:-/tmp}/golib-verification.XXXXXX" - )" - snapshot="${snapshot_parent}/repository" - snapshot_parents+=("${snapshot_parent}") - "${root}/.golib/scripts/create-verification-snapshot.sh" \ - "${root}" "${snapshot}" - selected_modules="$(paste -sd, - <"${lane_files[${lane}]}")" - ( - cd "${snapshot}" - GOLIB_VERIFICATION_SNAPSHOT=1 \ - ./scripts/run-modules.sh \ - "${gate}" --jobs 1 --modules "${selected_modules}" - ) & - snapshot_pids+=("$!") - snapshot_process_groups+=("$!") - lane=$((lane + 1)) -done - -status=0 -for pid in "${snapshot_pids[@]}"; do - if ! wait "${pid}"; then - status=1 - fi -done -snapshot_processes_complete=1 -for group in "${snapshot_process_groups[@]}"; do - if kill -0 -- "-${group}" 2>/dev/null; then - snapshot_processes_complete=0 - status=1 - break - fi -done -if [[ "${status}" -eq 0 ]]; then - while IFS= read -r module; do - [[ -n "${module}" ]] || continue - if [[ "${gate}" == "check" ]]; then - "${root}/.golib/scripts/audit-goals.sh" "${module}" >/dev/null - else - "${root}/.golib/scripts/verify-gate-evidence.sh" \ - "${module}" "${gate}" - fi - done <<<"${selection}" -fi -exit "${status}" diff --git a/.golib/scripts/mutation-verifier-identity.sh b/.golib/scripts/mutation-verifier-identity.sh deleted file mode 100755 index df9c609..0000000 --- a/.golib/scripts/mutation-verifier-identity.sh +++ /dev/null @@ -1,34 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -root="${GOLIB_ROOT:-$(git rev-parse --show-toplevel)}" - -# shellcheck disable=SC1091 -source "${root}/.golib/versions.env" - -: "${GREMLINS_VERSION:?GREMLINS_VERSION is required}" -: "${GREMLINS_SUM:?GREMLINS_SUM is required}" -: "${GREMLINS_GOMOD_SUM:?GREMLINS_GOMOD_SUM is required}" - -inputs=( - scripts/internal/mutation-command.sh - scripts/internal/mutation-coverage.sh - scripts/patches/gremlins-run-all-mutants.patch - scripts/patches/gremlins-shared-coverage.patch - scripts/patches/gremlins-module-relative-diff.patch -) - -{ - printf 'gremlins-version\t%s\n' "${GREMLINS_VERSION}" - printf 'gremlins-sum\t%s\n' "${GREMLINS_SUM}" - printf 'gremlins-gomod-sum\t%s\n' "${GREMLINS_GOMOD_SUM}" - for input in "${inputs[@]}"; do - [[ -f "${root}/.golib/${input}" ]] || { - printf 'mutation verifier input is missing: %s\n' "${input}" >&2 - exit 1 - } - printf 'file\t%s\t%s\n' "${input}" "$( - shasum -a 256 "${root}/.golib/${input}" | awk '{print $1}' - )" - done -} | shasum -a 256 | awk '{print $1}' diff --git a/.golib/scripts/package-source-digest.sh b/.golib/scripts/package-source-digest.sh deleted file mode 100755 index 3c49ed0..0000000 --- a/.golib/scripts/package-source-digest.sh +++ /dev/null @@ -1,42 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 1 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -package_directory="$1" -package_directory="${package_directory#./}" -case "${package_directory}" in - ""|/*|../*|*/../*|*/..) - printf 'package directory must be repository-relative: %s\n' \ - "${package_directory}" >&2 - exit 2 - ;; -esac -absolute="${root}/${package_directory}" -[[ -d "${absolute}" ]] || { - printf 'package directory does not exist: %s\n' "${package_directory}" >&2 - exit 2 -} - -manifest="$(mktemp "${TMPDIR:-/tmp}/golib-source-digest.XXXXXX")" -cleanup() { - rm -f "${manifest}" -} -trap cleanup EXIT HUP INT TERM - -while IFS= read -r -d '' file; do - relative="${file#"${root}/"}" - digest="$(shasum -a 256 "${file}" | awk '{print $1}')" - printf '%s %s\n' "${digest}" "${relative}" >>"${manifest}" -done < <(find "${absolute}" -maxdepth 1 -type f -name '*.go' \ - ! -name '*_test.go' -print0 | LC_ALL=C sort -z) - -[[ -s "${manifest}" ]] || { - printf 'package has no production Go files: %s\n' "${package_directory}" >&2 - exit 1 -} -shasum -a 256 "${manifest}" | awk '{print $1}' diff --git a/.golib/scripts/patches/gremlins-module-relative-diff.patch b/.golib/scripts/patches/gremlins-module-relative-diff.patch deleted file mode 100644 index c573b4c..0000000 --- a/.golib/scripts/patches/gremlins-module-relative-diff.patch +++ /dev/null @@ -1,89 +0,0 @@ -diff --git a/cmd/unleash.go b/cmd/unleash.go ---- a/cmd/unleash.go -+++ b/cmd/unleash.go -@@ -162 +162 @@ -- fDiff, err := diff.New() -+ fDiff, err := diff.New(mod.Root) -diff --git a/internal/diff/parse.go b/internal/diff/parse.go ---- a/internal/diff/parse.go -+++ b/internal/diff/parse.go -@@ -15,2 +15,2 @@ --func New() (Diff, error) { -- return NewWithCmd(exec.Command) -+func New(root string) (Diff, error) { -+ return NewWithCmd(root, exec.Command) -@@ -24 +24 @@ --func NewWithCmd[T execCmd](cmdContext func(name string, args ...string) T) (Diff, error) { -+func NewWithCmd[T execCmd](root string, cmdContext func(name string, args ...string) T) (Diff, error) { -@@ -33 +33 @@ -- cmd := cmdContext("git", "diff", "--merge-base", diffRef) -+ cmd := cmdContext("git", "-C", root, "diff", "--relative", "--merge-base", diffRef) -diff --git a/internal/diff/diff.go b/internal/diff/diff.go ---- a/internal/diff/diff.go -+++ b/internal/diff/diff.go -@@ -38,10 +38,19 @@ -- if fragment.LinesAdded == 0 { -- continue -- } -- -- startLine := int(fragment.NewPosition + fragment.LeadingContext) -- -- changes = append(changes, Change{ -- StartLine: startLine, -- EndLine: startLine + int(fragment.LinesAdded-1), -- }) -+ lineNumber := int(fragment.NewPosition) -+ for _, line := range fragment.Lines { -+ switch line.Op { -+ case gitdiff.OpAdd: -+ last := len(changes) - 1 -+ if last >= 0 && changes[last].EndLine == lineNumber-1 { -+ changes[last].EndLine = lineNumber -+ } else { -+ changes = append(changes, Change{ -+ StartLine: lineNumber, -+ EndLine: lineNumber, -+ }) -+ } -+ lineNumber++ -+ case gitdiff.OpContext: -+ lineNumber++ -+ case gitdiff.OpDelete: -+ } -+ } -diff --git a/internal/diff/diff_test.go b/internal/diff/diff_test.go ---- a/internal/diff/diff_test.go -+++ b/internal/diff/diff_test.go -@@ -98,0 +99,11 @@ -+ { -+ NewPosition: 500, -+ LinesAdded: 2, -+ Lines: []gitdiff.Line{ -+ {Op: gitdiff.OpContext}, -+ {Op: gitdiff.OpAdd}, -+ {Op: gitdiff.OpContext}, -+ {Op: gitdiff.OpContext}, -+ {Op: gitdiff.OpAdd}, -+ }, -+ }, -@@ -109,0 +121,2 @@ -+ {StartLine: 501, EndLine: 501}, -+ {StartLine: 504, EndLine: 504}, -diff --git a/internal/diff/parse_test.go b/internal/diff/parse_test.go ---- a/internal/diff/parse_test.go -+++ b/internal/diff/parse_test.go -@@ -17 +17 @@ -- d, err := NewWithCmd(m.call) -+ d, err := NewWithCmd("/repo", m.call) -@@ -31 +31 @@ -- _, err := NewWithCmd(m.call) -+ _, err := NewWithCmd("/repo", m.call) -@@ -40 +40 @@ -- expectedArgs := []string{"diff", "--merge-base", "test"} -+ expectedArgs := []string{"-C", "/repo", "diff", "--relative", "--merge-base", "test"} -@@ -56 +56 @@ -- _, err := NewWithCmd(m.call) -+ _, err := NewWithCmd("/repo", m.call) -@@ -73 +73 @@ -- result, err := NewWithCmd(m.call) -+ result, err := NewWithCmd("/repo", m.call) diff --git a/.golib/scripts/patches/gremlins-run-all-mutants.patch b/.golib/scripts/patches/gremlins-run-all-mutants.patch deleted file mode 100644 index 7ac7799..0000000 --- a/.golib/scripts/patches/gremlins-run-all-mutants.patch +++ /dev/null @@ -1,60 +0,0 @@ -diff --git a/internal/engine/engine.go b/internal/engine/engine.go ---- a/internal/engine/engine.go -+++ b/internal/engine/engine.go -@@ -188,5 +188 @@ -- var status mutator.Status -- -- if mu.codeData.Cov.IsCovered(pos) { -- status = mutator.Runnable -- } -+ status := mutator.Runnable -diff --git a/internal/engine/executor.go b/internal/engine/executor.go ---- a/internal/engine/executor.go -+++ b/internal/engine/executor.go -@@ -197,2 +196,0 @@ -- cmd := m.execContext(ctx, "go", m.getTestArgs(pkg)...) -- cmd.Dir = m.mutant.Workdir() -@@ -200 +198,8 @@ -- cmd.Dir = rootDir -+ status := m.runTestCommand( -+ ctx, -+ m.mutant.Workdir(), -+ m.getTestArgsFor(pkg, false, false), -+ ) -+ if status != mutator.Lived { -+ return status -+ } -@@ -201,0 +207,16 @@ -+ -+ workingDir := m.mutant.Workdir() -+ if m.integrationMode { -+ workingDir = rootDir -+ } -+ -+ return m.runTestCommand(ctx, workingDir, m.getTestArgs(pkg)) -+} -+ -+func (m *mutantExecutor) runTestCommand( -+ ctx context.Context, -+ workingDir string, -+ args []string, -+) mutator.Status { -+ cmd := m.execContext(ctx, "go", args...) -+ cmd.Dir = workingDir -@@ -219,0 +241,7 @@ -+ return m.getTestArgsFor(pkg, m.integrationMode, true) -+} -+ -+func (m *mutantExecutor) getTestArgsFor( -+ pkg string, -+ integrationMode, includeBuildTags bool, -+) []string { -@@ -221 +249 @@ -- if m.buildTags != "" { -+ if includeBuildTags && m.buildTags != "" { -@@ -231 +259 @@ -- args = append(args, fmt.Sprintf("-cpu %d", m.testCPU)) -+ args = append(args, "-cpu", fmt.Sprint(m.testCPU)) -@@ -235 +263 @@ -- if m.integrationMode { -+ if integrationMode { diff --git a/.golib/scripts/patches/gremlins-shared-coverage.patch b/.golib/scripts/patches/gremlins-shared-coverage.patch deleted file mode 100644 index 86e8912..0000000 --- a/.golib/scripts/patches/gremlins-shared-coverage.patch +++ /dev/null @@ -1,51 +0,0 @@ -diff --git a/internal/coverage/coverage.go b/internal/coverage/coverage.go ---- a/internal/coverage/coverage.go -+++ b/internal/coverage/coverage.go -@@ -102,0 +103,5 @@ -+ if c.integrationMode { -+ if err := c.executeUnitBaseline(); err != nil { -+ return Result{}, fmt.Errorf("unit test baseline failed: %w", err) -+ } -+ } -@@ -143,0 +149,11 @@ -+func (c *Coverage) executeUnitBaseline() error { -+ cmd := c.cmdContext("go", "test", "-count=1", "./...") -+ if out, err := cmd.CombinedOutput(); err != nil { -+ log.Infof("\n%s\n", string(out)) -+ -+ return err -+ } -+ -+ return nil -+} -+ -@@ -144,0 +161,29 @@ -+ if sourcePath := os.Getenv("GOLIB_GREMLINS_COVERAGE_PROFILE"); sourcePath != "" { -+ elapsedText := os.Getenv("GOLIB_GREMLINS_COVERAGE_ELAPSED") -+ elapsed, err := time.ParseDuration(elapsedText) -+ if err != nil || elapsed <= 0 { -+ return 0, fmt.Errorf("invalid external coverage elapsed duration %q", elapsedText) -+ } -+ source, err := os.Open(sourcePath) -+ if err != nil { -+ return 0, fmt.Errorf("open external coverage profile: %w", err) -+ } -+ defer func() { -+ _ = source.Close() -+ }() -+ destination, err := os.Create(c.filePath()) -+ if err != nil { -+ return 0, fmt.Errorf("create coverage profile: %w", err) -+ } -+ defer func() { -+ _ = destination.Close() -+ }() -+ if _, err := io.Copy(destination, source); err != nil { -+ return 0, fmt.Errorf("copy external coverage profile: %w", err) -+ } -+ if err := destination.Close(); err != nil { -+ return 0, fmt.Errorf("close coverage profile: %w", err) -+ } -+ -+ return elapsed, nil -+ } diff --git a/.golib/scripts/release.sh b/.golib/scripts/release.sh deleted file mode 100755 index a2ed9d3..0000000 --- a/.golib/scripts/release.sh +++ /dev/null @@ -1,59 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -dry_run=0 -public=0 -while [[ $# -gt 0 ]]; do - case "$1" in - --dry-run) dry_run=1; shift ;; - --public) public=1; shift ;; - *) break ;; - esac -done -if [[ "${dry_run}" -ne 1 || $# -ne 1 ]]; then - printf 'usage: %s --dry-run [--public] \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -module="$1" -record="$(jq -ce --arg directory "${module}" \ - '.modules[] | select(.directory == $directory and .releasable == true)' \ - "${root}/modules.json")" -module_path="$(jq -r '.module_path' <<<"${record}")" -tag_prefix="$(jq -r '.tag_prefix' <<<"${record}")" -version="v$(jq -r '.version' <<<"${record}")" -tag="${tag_prefix}${version#v}" -directory="${root}/${module}" - -[[ "$(sed -n 's/^module[[:space:]]\+//p' "${directory}/go.mod")" == "${module_path}" ]] -if grep -Eq '^[[:space:]]*replace([[:space:]]|$)' "${directory}/go.mod"; then - printf 'release module contains a replace directive: %s\n' "${module}" >&2 - exit 1 -fi -if git -C "${root}" show-ref --verify --quiet "refs/tags/${tag}"; then - printf 'release tag already exists: %s\n' "${tag}" >&2 - exit 1 -fi - -task="$(mktemp -d "${TMPDIR:-/tmp}/golib-release.XXXXXX")" -# shellcheck disable=SC2329 # Invoked by the release EXIT trap. -cleanup() { - chmod -R u+w "${task}" 2>/dev/null || true - find "${task}" -depth -delete 2>/dev/null || true -} -trap cleanup EXIT HUP INT TERM - -if [[ "${public}" -eq 1 ]]; then - GOPROXY="https://proxy.golang.org,direct" GOWORK=off \ - go list -m "${module_path}@${version}" >/dev/null -else - proxy="${task}/proxy" - mkdir "${proxy}" - "${root}/.golib/scripts/build-local-proxy.sh" "${proxy}" "${version}" - GOPROXY="file://${proxy},https://proxy.golang.org,direct" \ - GONOSUMDB="github.com/faustbrian/go-*" GOWORK=off \ - go list -m "${module_path}@${version}" >/dev/null -fi - -printf 'release dry-run passed: %s %s\n' "${module_path}" "${tag}" diff --git a/.golib/scripts/repository-check.sh b/.golib/scripts/repository-check.sh deleted file mode 100755 index 231c796..0000000 --- a/.golib/scripts/repository-check.sh +++ /dev/null @@ -1,46 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -root="$(git rev-parse --show-toplevel)" -repository="github.com/faustbrian/go-lease" - -jq -e --arg repository "${repository}" ' - .repository == $repository and - (.modules | length > 0) and - all(.modules[]; - (.directory == "." or (.directory | startswith("/") | not)) and - ( - .releasable == false or - .module_path == $repository or - (.module_path | startswith($repository + "/")) - ) - ) -' "${root}/modules.json" >/dev/null - -while IFS= read -r module; do - directory="$(jq -r --arg module "${module}" \ - '.modules[] | select(.module_path == $module) | .directory' \ - "${root}/modules.json")" - [[ "$(sed -n 's/^module[[:space:]]\+//p' "${root}/${directory}/go.mod")" == "${module}" ]] - if grep -Eq '^[[:space:]]*replace([[:space:]]|$)' "${root}/${directory}/go.mod"; then - printf 'committed replace directive in %s\n' "${directory}/go.mod" >&2 - exit 1 - fi -done < <(jq -r '.modules[].module_path' "${root}/modules.json") - -if grep -REnI \ - --exclude-dir='.git' \ - --exclude-dir='.artifacts' \ - --exclude='go.sum' \ - --exclude='CHANGELOG.md' \ - --exclude='repository-check.sh' \ - 'github\.com/faustbrian/golib|/Users/[^/]+/Developer|\.\./go-' \ - "${root}"; then - printf 'monorepo or sibling-checkout reference remains\n' >&2 - exit 1 -fi - -git -C "${root}" ls-files --error-unmatch package-lock.json >/dev/null - -git diff --check -printf 'standalone repository contract passed\n' diff --git a/.golib/scripts/restore-ci-mutation-evidence.sh b/.golib/scripts/restore-ci-mutation-evidence.sh deleted file mode 100755 index 5fe812b..0000000 --- a/.golib/scripts/restore-ci-mutation-evidence.sh +++ /dev/null @@ -1,273 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -lt 1 || $# -gt 2 ]]; then - printf 'usage: %s [artifact-archive]\n' "$0" >&2 - exit 2 -fi - -root="${GOLIB_ROOT:-$(git rev-parse --show-toplevel)}" -module="$1" -archive="${2:-}" - -if ! jq -e --arg directory "${module}" \ - '.modules[] | select(.directory == $directory)' \ - "${root}/modules.json" >/dev/null; then - printf 'cannot restore mutation evidence for unknown module: %s\n' \ - "${module}" >&2 - exit 2 -fi - -task="$(mktemp -d "${TMPDIR:-/tmp}/golib-ci-evidence.XXXXXX")" -cleanup() { - find "${task}" -depth -delete 2>/dev/null || true -} -trap cleanup EXIT HUP INT TERM - -checkpoint_directory="${root}/.artifacts/${module}/mutation-checkpoints" -mkdir -p "${checkpoint_directory}" -packages="${task}/packages" -jq -r --arg directory "${module}" ' - .modules[] - | select(.directory == $directory) - | .packages[] - | select(.coverage_required == true) - | .directory -' "${root}/modules.json" | LC_ALL=C sort >"${packages}" -expected="$(wc -l <"${packages}" | tr -d ' ')" -restored=0 -require_exact_digest=0 -digest_directory="${task}/input-digests" -fallback_directory="${task}/fallback-checkpoints" -history_migrations="${root}/.golib/mutation-history-migrations.json" -current_verifier_identity="" -current_revision="" -current_gremlins_version="" - -command -v unzip >/dev/null || { - printf 'unzip is required to restore CI mutation evidence\n' >&2 - exit 1 -} - -restore_archive() { - local candidate_archive="$1" - local entries package_directory slug entry candidate destination fallback - local expected_digest migrated - if [[ ! -s "${candidate_archive}" ]] || - ! unzip -tqq "${candidate_archive}" >/dev/null; then - return 1 - fi - - entries="${task}/entries" - unzip -Z1 "${candidate_archive}" >"${entries}" - while IFS= read -r package_directory; do - [[ -n "${package_directory}" ]] || continue - slug=root - if [[ "${package_directory}" != "." ]]; then - slug="${package_directory//\//-}" - fi - destination="${checkpoint_directory}/${slug}.json" - [[ ! -s "${destination}" ]] || continue - fallback="${fallback_directory}/${slug}.json" - expected_digest="" - if [[ "${require_exact_digest}" -eq 1 ]]; then - [[ -s "${digest_directory}/${slug}" ]] || continue - expected_digest="$(<"${digest_directory}/${slug}")" - fi - entry="mutation-checkpoints/${slug}.json" - if [[ "$(grep -Fxc -- "${entry}" "${entries}")" -ne 1 ]]; then - continue - fi - candidate="$(mktemp "${task}/checkpoint.XXXXXX")" - if ! unzip -p "${candidate_archive}" "${entry}" >"${candidate}" || - ! jq -e \ - --arg module "${module}" \ - --arg package "${package_directory}" \ - ' - .schema_version == 3 and - .module == $module and - .package == $package and - (.execution_revision | type == "string") and - (.execution_revision | test("^[0-9a-f]{40}$")) and - (.gate_input_digest | type == "string") and - (.gate_input_digest | test("^[0-9a-f]{64}$")) and - (.gremlins_version | type == "string") and - (.gremlins_version | length > 0) and - ( - (.gremlins_verifier_sha256 // null) == null or - ( - (.gremlins_verifier_sha256 | type == "string") and - (.gremlins_verifier_sha256 | test("^[0-9a-f]{64}$")) - ) - ) and - (.report.files | type == "array") and - ([.report.files[].mutations[]? | select(.status != "KILLED")] | length == 0) - ' "${candidate}" >/dev/null; then - find "${candidate}" -delete 2>/dev/null || true - continue - fi - if [[ "${require_exact_digest}" -eq 0 ]]; then - mv "${candidate}" "${destination}" - restored=$((restored + 1)) - continue - fi - if jq -e \ - --arg digest "${expected_digest}" \ - --arg version "${current_gremlins_version}" \ - --arg verifier "${current_verifier_identity}" ' - .gate_input_digest == $digest and - .gremlins_version == $version and - .gremlins_verifier_sha256 == $verifier and - ( - ( - .verifier_identity_source == "executed" and - (.gremlins_binary_sha256 | type == "string") and - (.gremlins_binary_sha256 | test("^[0-9a-f]{64}$")) - ) or - ( - .verifier_identity_source == "approved-semantic-migration" and - ( - (.gremlins_binary_sha256 // null) == null or - ( - (.gremlins_binary_sha256 | type == "string") and - (.gremlins_binary_sha256 | test("^[0-9a-f]{64}$")) - ) - ) - ) - ) - ' "${candidate}" >/dev/null; then - mv "${candidate}" "${destination}" - restored=$((restored + 1)) - continue - fi - migrated="$(mktemp "${task}/migrated-checkpoint.XXXXXX")" - if [[ -s "${history_migrations}" ]] && - "${root}/.golib/scripts/internal/reuse-approved-mutation-checkpoint.sh" \ - "${history_migrations}" \ - "${candidate}" \ - "${module}" \ - "${package_directory}" \ - "${expected_digest}" \ - "${current_gremlins_version}" \ - "${current_verifier_identity}" \ - "${current_revision}" \ - "${migrated}"; then - mv "${migrated}" "${destination}" - find "${candidate}" -delete 2>/dev/null || true - restored=$((restored + 1)) - continue - fi - find "${migrated}" -delete 2>/dev/null || true - mkdir -p "${fallback_directory}" - if [[ ! -s "${fallback}" ]]; then - mv "${candidate}" "${fallback}" - else - find "${candidate}" -delete 2>/dev/null || true - fi - done <"${packages}" -} - -if [[ -z "${archive}" ]]; then - repository="${GITHUB_REPOSITORY:-}" - repository_id="${GITHUB_REPOSITORY_ID:-}" - run_id="${GITHUB_RUN_ID:-}" - if [[ -z "${GH_TOKEN:-}" || -z "${repository}" || - ! "${repository_id}" =~ ^[0-9]+$ || ! "${run_id}" =~ ^[0-9]+$ ]]; then - printf '[%s] prior CI mutation evidence is unavailable\n' "${module}" - exit 0 - fi - command -v gh >/dev/null || { - printf 'GitHub CLI is required to restore CI mutation evidence\n' >&2 - exit 1 - } - - artifact_slug="${module//\//-}" - [[ "${artifact_slug}" != "." ]] || artifact_slug=root - artifact_name="evidence-${artifact_slug}" - if ! artifacts="$( - gh api --method GET \ - "repos/${repository}/actions/artifacts" \ - -f "name=${artifact_name}" \ - -f per_page=100 - )"; then - printf '[%s] prior CI mutation evidence lookup failed; executing fresh\n' \ - "${module}" - exit 0 - fi - artifact_ids="$( - jq -r \ - --argjson repository_id "${repository_id}" \ - --argjson run_id "${run_id}" ' - [ - .artifacts[] - | select( - .expired == false and - .workflow_run.head_repository_id == $repository_id and - .workflow_run.head_branch == "main" and - .workflow_run.id != $run_id - ) - ] - | sort_by(.created_at) - | reverse - | .[].id - ' <<<"${artifacts}" - )" - if [[ -z "${artifact_ids}" ]]; then - printf '[%s] no trusted prior CI mutation evidence found\n' "${module}" - exit 0 - fi - mkdir -p "${digest_directory}" - # shellcheck disable=SC1091 - source "${root}/.golib/versions.env" - current_gremlins_version="${GREMLINS_VERSION}" - current_verifier_identity="$( - "${root}/.golib/scripts/mutation-verifier-identity.sh" - )" - current_revision="${GITHUB_SHA:-$(git -C "${root}" rev-parse HEAD)}" - require_exact_digest=1 - while IFS= read -r package_directory; do - [[ -n "${package_directory}" ]] || continue - slug=root - if [[ "${package_directory}" != "." ]]; then - slug="${package_directory//\//-}" - fi - if ! "${root}/.golib/scripts/gate-input-digest.sh" \ - mutation "${module}" "${package_directory}" \ - >"${digest_directory}/${slug}"; then - find "${digest_directory}/${slug}" -delete 2>/dev/null || true - fi - done <"${packages}" - while IFS= read -r artifact_id; do - [[ "${artifact_id}" =~ ^[0-9]+$ ]] || continue - candidate_archive="${task}/evidence-${artifact_id}.zip" - if ! gh api \ - "repos/${repository}/actions/artifacts/${artifact_id}/zip" \ - >"${candidate_archive}"; then - continue - fi - restore_archive "${candidate_archive}" || continue - [[ "${restored}" -lt "${expected}" ]] || break - done <<<"${artifact_ids}" - while IFS= read -r package_directory; do - [[ -n "${package_directory}" ]] || continue - slug=root - if [[ "${package_directory}" != "." ]]; then - slug="${package_directory//\//-}" - fi - destination="${checkpoint_directory}/${slug}.json" - fallback="${fallback_directory}/${slug}.json" - if [[ ! -s "${destination}" && -s "${fallback}" ]]; then - mv "${fallback}" "${destination}" - restored=$((restored + 1)) - fi - done <"${packages}" -else - if ! restore_archive "${archive}"; then - printf '[%s] prior CI mutation evidence archive is invalid; executing fresh\n' \ - "${module}" - exit 0 - fi -fi - -printf '[%s] restored %d prior content-addressed mutation checkpoints\n' \ - "${module}" "${restored}" diff --git a/.golib/scripts/run-gate-with-evidence.sh b/.golib/scripts/run-gate-with-evidence.sh deleted file mode 100755 index 366d64f..0000000 --- a/.golib/scripts/run-gate-with-evidence.sh +++ /dev/null @@ -1,234 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 2 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -module="$1" -gate="$2" -artifact="${root}/.artifacts/${module}/evidence" -legacy_evidence="${artifact}/${gate}.json" -legacy_log="${artifact}/${gate}.log" -lock_root="${artifact}/.locks" -lock="${lock_root}/${gate}.lock" -temporary_evidence="" -temporary_log="" -temporary_legacy_evidence="${legacy_evidence}.tmp.$$" -temporary_legacy_log="${legacy_log}.tmp.$$" -started_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" -execution_revision="$(git rev-parse HEAD)" -lock_acquired=0 - -# shellcheck disable=SC2329 # Invoked by the signal and exit trap. -cleanup() { - if [[ -n "${temporary_evidence}" ]]; then - rm -f "${temporary_evidence}" - fi - if [[ -n "${temporary_log}" ]]; then - rm -f "${temporary_log}" - fi - rm -f "${temporary_legacy_evidence}" "${temporary_legacy_log}" - if [[ "${lock_acquired}" -eq 1 ]]; then - owner="$(readlink "${lock}" 2>/dev/null || true)" - if [[ "${owner}" == "$$" ]]; then - rm -f "${lock}" - fi - fi -} - -forward_gate_output() { - local line output_open=1 - trap '' PIPE - while IFS= read -r line || [[ -n "${line}" ]]; do - if [[ "${output_open}" -eq 1 ]] && - ! printf '%s\n' "${line}" 2>/dev/null; then - output_open=0 - fi - done - return 0 -} - -emit_evidence_status() ( - trap '' PIPE - printf '[%s] %s evidence: %s\n' "$1" "$2" "$3" 2>/dev/null || true -) - -trap cleanup EXIT -trap 'exit 130' HUP INT TERM -if [[ -x "${root}/.golib/scripts/internal/migrate-runner-isolation-evidence.sh" ]]; then - "${root}/.golib/scripts/internal/migrate-runner-isolation-evidence.sh" \ - "${module}" "${gate}" -fi -mkdir -p "${artifact}" -mkdir -p "${lock_root}" -while ! ln -s "$$" "${lock}" 2>/dev/null; do - if [[ -L "${lock}" ]]; then - owner="$(readlink "${lock}" 2>/dev/null || true)" - if [[ "${owner}" =~ ^[0-9]+$ ]] && ! kill -0 "${owner}" 2>/dev/null; then - rm -f "${lock}" - continue - fi - elif [[ -d "${lock}" ]]; then - owner="$(cat "${lock}/owner" 2>/dev/null || true)" - if [[ "${owner}" =~ ^[0-9]+$ ]] && ! kill -0 "${owner}" 2>/dev/null; then - rm -f "${lock}/owner" - rmdir "${lock}" 2>/dev/null || true - continue - fi - if [[ -z "${owner}" ]] && rmdir "${lock}" 2>/dev/null; then - continue - fi - elif [[ -f "${lock}" ]]; then - owner="$(cat "${lock}" 2>/dev/null || true)" - if [[ "${owner}" =~ ^[0-9]+$ ]] && ! kill -0 "${owner}" 2>/dev/null; then - rm -f "${lock}" - continue - fi - if [[ -z "${owner}" ]]; then - rm -f "${lock}" - continue - fi - fi - sleep 0.05 -done -lock_acquired=1 - -input_digest="$("${root}/.golib/scripts/gate-input-digest.sh" "${gate}" "${module}")" -digest_artifact="${artifact}/by-input/${gate}" -evidence="${digest_artifact}/${input_digest}.json" -log="${digest_artifact}/${input_digest}.log" -temporary_evidence="${evidence}.tmp.$$" -temporary_log="${log}.tmp.$$" -mkdir -p "${digest_artifact}" -rm -f "${temporary_evidence}" "${temporary_log}" - -source_evidence="${evidence}" -source_log="${log}" -if [[ ! -f "${source_evidence}" || ! -f "${source_log}" ]]; then - source_evidence="${legacy_evidence}" - source_log="${legacy_log}" -fi -if [[ -f "${source_evidence}" && -f "${source_log}" ]]; then - recorded_log_sha256="$(jq -er '.log_sha256' "${source_evidence}" 2>/dev/null || true)" - current_log_sha256="$(shasum -a 256 "${source_log}" | awk '{print $1}')" - if jq -e \ - --arg module "${module}" \ - --arg gate "${gate}" \ - --arg input_digest "${input_digest}" \ - --arg log_sha256 "${current_log_sha256}" \ - ' - .schema_version == 1 and - .module == $module and - .gate == $gate and - (.result == "passed" or .result == "not_applicable" or .result == "advisory") and - .exit_code == 0 and - .input_digest == $input_digest and - .completed_input_digest == $input_digest and - .log_sha256 == $log_sha256 - ' "${source_evidence}" >/dev/null 2>&1 && - [[ "${recorded_log_sha256}" == "${current_log_sha256}" ]]; then - revalidated_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" - reused_result="$(jq -r '.result' "${source_evidence}")" - if grep -Fq ': not applicable by catalog policy' "${source_log}"; then - reused_result=not_applicable - elif [[ "${gate}" == "nilaway" ]] && - grep -Eq 'NilAway advisory exit status: [1-9][0-9]*$' "${source_log}"; then - reused_result=advisory - fi - jq \ - --arg revalidated_revision "${execution_revision}" \ - --arg revalidated_at "${revalidated_at}" \ - --arg result "${reused_result}" \ - ' - .result = $result | - .revalidated_revision = $revalidated_revision | - .revalidated_at = $revalidated_at | - .reuse_count = ((.reuse_count // 0) + 1) - ' "${source_evidence}" >"${temporary_evidence}" - cp "${source_log}" "${temporary_log}" - mv "${temporary_log}" "${log}" - mv "${temporary_evidence}" "${evidence}" - cp "${log}" "${temporary_legacy_log}" - cp "${evidence}" "${temporary_legacy_evidence}" - mv "${temporary_legacy_log}" "${legacy_log}" - mv "${temporary_legacy_evidence}" "${legacy_evidence}" - emit_evidence_status "${module}" "${gate}" reused - exit 0 - fi -fi - -set +e -"${root}/.golib/scripts/check-module.sh" "${module}" "${gate}" 2>&1 | - tee "${temporary_log}" | - forward_gate_output -pipeline_status=("${PIPESTATUS[@]}") -command_status=${pipeline_status[0]} -tee_status=${pipeline_status[1]} -if [[ "${command_status}" -eq 0 && "${tee_status}" -ne 0 ]]; then - command_status="${tee_status}" -fi -set -e - -completed_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" -completed_revision="$(git rev-parse HEAD)" -completed_digest="$("${root}/.golib/scripts/gate-input-digest.sh" "${gate}" "${module}")" -log_sha256="$(shasum -a 256 "${temporary_log}" | awk '{print $1}')" -result=passed -if [[ "${command_status}" -ne 0 ]]; then - result=failed -elif [[ "${completed_digest}" != "${input_digest}" ]]; then - result=invalidated - command_status=1 -elif grep -Fq ': not applicable by catalog policy' "${temporary_log}"; then - result=not_applicable -elif [[ "${gate}" == "nilaway" ]] && - grep -Eq 'NilAway advisory exit status: [1-9][0-9]*$' "${temporary_log}"; then - result=advisory -fi - -jq -n \ - --arg module "${module}" \ - --arg gate "${gate}" \ - --arg result "${result}" \ - --argjson exit_code "${command_status}" \ - --arg execution_revision "${execution_revision}" \ - --arg completed_revision "${completed_revision}" \ - --arg input_digest "${input_digest}" \ - --arg completed_digest "${completed_digest}" \ - --arg log_sha256 "${log_sha256}" \ - --arg started_at "${started_at}" \ - --arg completed_at "${completed_at}" \ - --arg go_version "$(go env GOVERSION)" \ - --arg goos "$(go env GOOS)" \ - --arg goarch "$(go env GOARCH)" \ - '{ - schema_version: 1, - module: $module, - gate: $gate, - result: $result, - exit_code: $exit_code, - execution_revision: $execution_revision, - completed_revision: $completed_revision, - input_digest: $input_digest, - completed_input_digest: $completed_digest, - log_sha256: $log_sha256, - started_at: $started_at, - completed_at: $completed_at, - environment: { - go_version: $go_version, - goos: $goos, - goarch: $goarch - } - }' >"${temporary_evidence}" - -mv "${temporary_log}" "${log}" -mv "${temporary_evidence}" "${evidence}" -cp "${log}" "${temporary_legacy_log}" -cp "${evidence}" "${temporary_legacy_evidence}" -mv "${temporary_legacy_log}" "${legacy_log}" -mv "${temporary_legacy_evidence}" "${legacy_evidence}" -emit_evidence_status "${module}" "${gate}" "${result}" -exit "${command_status}" diff --git a/.golib/scripts/run-modules.sh b/.golib/scripts/run-modules.sh deleted file mode 100755 index f22fd9d..0000000 --- a/.golib/scripts/run-modules.sh +++ /dev/null @@ -1,45 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -lt 2 ]]; then - printf 'usage: %s <--all|--modules LIST>\n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -gate="$1" -shift -case "$1" in - --all) - selection="$(jq -r '.modules[].directory' "${root}/modules.json")" - ;; - --modules) - [[ $# -eq 2 ]] || exit 2 - selection="${2//,/\\n}" - ;; - *) - printf 'unknown module selection: %s\n' "$1" >&2 - exit 2 - ;; -esac - -while IFS= read -r module; do - [[ -n "${module}" ]] || continue - ( - task="$(mktemp -d "${TMPDIR:-/tmp}/golib-services.XXXXXX")" - environment="${task}/environment" - state="${task}/state" - # shellcheck disable=SC2329 # Invoked by the subshell EXIT trap. - cleanup() { - "${root}/.golib/scripts/stop-services.sh" "${state}" || true - find "${task}" -depth -delete 2>/dev/null || true - } - trap cleanup EXIT HUP INT TERM - "${root}/.golib/scripts/start-services.sh" "${module}" "${environment}" "${state}" - set -a - # shellcheck source=/dev/null - source "${environment}" - set +a - "${root}/.golib/scripts/check-module.sh" "${module}" "${gate}" - ) -done <<<"${selection}" diff --git a/.golib/scripts/stage-ci-evidence.sh b/.golib/scripts/stage-ci-evidence.sh deleted file mode 100755 index 955435e..0000000 --- a/.golib/scripts/stage-ci-evidence.sh +++ /dev/null @@ -1,144 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 3 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="${GOLIB_ROOT:-$(git rev-parse --show-toplevel)}" -module="$1" -destination="$2" -outcome="$3" -script_directory="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -source="${root}/.artifacts" -if [[ "${module}" != "." ]]; then - source="${source}/${module}" -fi -case "${outcome}" in - success|failure|cancelled|skipped) ;; - *) - printf 'invalid CI contract outcome for %s: %s\n' \ - "${module}" "${outcome}" >&2 - exit 2 - ;; -esac -source_available=0 -if [[ -d "${source}" && ! -L "${source}" ]]; then - source="$(cd "${source}" && pwd -P)" - source_available=1 -elif [[ -e "${source}" || -L "${source}" ]]; then - printf 'invalid attributable evidence source for %s\n' "${module}" >&2 - exit 1 -fi -destination_name="$(basename "${destination}")" -destination_parent="$(cd "$(dirname "${destination}")" && pwd -P)" || { - printf 'CI evidence destination parent is unavailable: %s\n' \ - "${destination}" >&2 - exit 1 -} -[[ "${destination_name}" != "." && "${destination_name}" != ".." ]] || { - printf 'invalid CI evidence destination: %s\n' "${destination}" >&2 - exit 1 -} -destination="${destination_parent}/${destination_name}" -[[ ! -e "${destination}" && ! -L "${destination}" ]] || { - printf 'CI evidence destination already exists: %s\n' \ - "${destination}" >&2 - exit 1 -} -if [[ "${source_available}" -eq 1 ]]; then - case "${destination}" in - "${source}"|"${source}"/*) - printf 'CI evidence destination must be outside its source: %s\n' \ - "${destination}" >&2 - exit 1 - ;; - esac -fi - -# Cancellation can bypass the mutation runner's signal trap. Reclaim only -# scratch directories whose recorded owner process is demonstrably gone. -# shellcheck disable=SC1091 -. "${script_directory}/internal/mutation-scratch.sh" -if [[ "${source_available}" -eq 1 ]]; then - mutation_scratch_recover_abandoned "${source}" -fi - -mkdir -p "${destination}" -inventory="$(mktemp "${TMPDIR:-/tmp}/golib-ci-evidence.XXXXXXXX")" -stage_complete=0 -cleanup() { - local exit_code=$? - trap - EXIT HUP INT TERM - find "${inventory}" -delete 2>/dev/null || true - if [[ "${stage_complete}" -eq 0 && -d "${destination}" && - ! -L "${destination}" ]]; then - find "${destination}" -depth -delete 2>/dev/null || true - fi - exit "${exit_code}" -} -trap cleanup EXIT -trap 'exit 129' HUP -trap 'exit 130' INT -trap 'exit 143' TERM - -files=0 -if [[ "${source_available}" -eq 1 ]]; then - find "${source}" -mindepth 1 \ - \( -type d \( \ - -name 'mutation-run-*' -o \ - -name '*.go-cache-*' -o \ - -name 'go-cache' -o \ - -name '.locks' -o \ - -name '*.lock' -o \ - -name '*.tmp.*' \ - \) -prune \) -o -print0 >"${inventory}" -else - : >"${inventory}" -fi -while IFS= read -r -d '' candidate; do - relative="${candidate#"${source}/"}" - basename="${candidate##*/}" - case "${basename}" in - *.tmp.*|*.lock) - continue - ;; - esac - if [[ -L "${candidate}" ]]; then - printf 'refusing to stage symbolic link: %s\n' "${relative}" >&2 - exit 1 - fi - if [[ -d "${candidate}" ]]; then - mkdir -p "${destination}/${relative}" - continue - fi - if [[ ! -f "${candidate}" ]]; then - printf 'refusing to stage unsupported artifact: %s\n' \ - "${relative}" >&2 - exit 1 - fi - mkdir -p "$(dirname "${destination}/${relative}")" - cp -p "${candidate}" "${destination}/${relative}" - files=$((files + 1)) -done <"${inventory}" - -jq -n \ - --arg module "${module}" \ - --arg outcome "${outcome}" \ - --arg repository "${GITHUB_REPOSITORY:-}" \ - --arg run_id "${GITHUB_RUN_ID:-}" \ - --arg run_attempt "${GITHUB_RUN_ATTEMPT:-}" \ - --arg revision "${GITHUB_SHA:-}" \ - '{ - schema_version: 1, - module: $module, - outcome: $outcome, - repository: $repository, - run_id: $run_id, - run_attempt: $run_attempt, - revision: $revision - }' >"${destination}/ci-result.json" -files=$((files + 1)) -stage_complete=1 -printf '[%s] staged %d durable evidence files\n' "${module}" "${files}" diff --git a/.golib/scripts/start-services.sh b/.golib/scripts/start-services.sh deleted file mode 100755 index 44b9082..0000000 --- a/.golib/scripts/start-services.sh +++ /dev/null @@ -1,296 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 3 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -module="$1" -environment_file="$2" -state_file="$3" -slug="$(printf '%s' "${module}" | tr '/.' '--')-${RANDOM}" - -# shellcheck source=/dev/null -source "${root}/.golib/versions.env" -: >"${environment_file}" -: >"${state_file}" - -services="$(jq -r --arg directory "${module}" \ - '.modules[] | select(.directory == $directory) | .required_services[]' \ - "${root}/modules.json")" -[[ -n "${services}" ]] || exit 0 -command -v docker >/dev/null || { - printf 'Docker is required by %s for: %s\n' "${module}" "${services}" >&2 - exit 1 -} - -record() { - printf '%s\n' "$1" >>"${state_file}" -} - -record_resource() { - local kind="$1" - shift - printf '%s' "${kind}" >>"${state_file}" - printf '\t%s' "$@" >>"${state_file}" - printf '\n' >>"${state_file}" -} - -wait_for() { - local container="$1" - shift - for _ in {1..90}; do - if docker exec "${container}" "$@" >/dev/null 2>&1; then - return 0 - fi - if [[ "$(docker inspect --format '{{.State.Running}}' "${container}" 2>/dev/null || true)" != "true" ]]; then - break - fi - sleep 1 - done - docker logs "${container}" >&2 || true - printf 'service did not become healthy: %s\n' "${container}" >&2 - exit 1 -} - -published_port() { - docker port "$1" "$2/tcp" | tail -1 | sed 's/.*://' -} - -acquire_rabbitstream_lock() { - local lock owner started - lock="${TMPDIR:-/tmp}/golib-rabbitstream-fixture.lock" - started="${SECONDS}" - while ! mkdir "${lock}" 2>/dev/null; do - owner="$(cat "${lock}/owner" 2>/dev/null || true)" - if [[ "${owner}" =~ ^[0-9]+$ ]] && ! kill -0 "${owner}" 2>/dev/null; then - rm -f "${lock}/owner" - rmdir "${lock}" 2>/dev/null || true - continue - fi - if ((SECONDS - started >= 1200)); then - printf 'timed out waiting for the RabbitStream fixture lock\n' >&2 - exit 1 - fi - sleep 1 - done - printf '%s\n' "${PPID}" >"${lock}/owner" - record_resource lock "${lock}" "${PPID}" -} - -write_service_environment() { - local name="$1" - local value="$2" - printf '%s=%q\n' "${name}" "${value}" >>"${environment_file}" -} - -start_rabbitstream() { - local mode="$1" - local integration fixture_id standalone_project cluster_project tls_project - local fixture_parent tls_runtime - - command -v openssl >/dev/null || { - printf 'OpenSSL is required by %s for RabbitStream fixtures\n' "${module}" >&2 - exit 1 - } - acquire_rabbitstream_lock - integration="${root}/.golib/services/rabbitstream" - fixture_id="codex-rabbitstream-${RANDOM}-$$" - standalone_project="${fixture_id}-single" - cluster_project="${fixture_id}-cluster" - tls_project="${fixture_id}-tls" - - RABBITSTREAM_USER="rabbitstream-$(openssl rand -hex 8)" - RABBITSTREAM_PASSWORD="$(openssl rand -hex 24)" - RABBITSTREAM_ERLANG_COOKIE="$(openssl rand -hex 32)" - RABBITSTREAM_RESTRICTED_USER="restricted-$(openssl rand -hex 8)" - RABBITSTREAM_RESTRICTED_PASSWORD="$(openssl rand -hex 24)" - export RABBITSTREAM_USER RABBITSTREAM_PASSWORD RABBITSTREAM_ERLANG_COOKIE - export RABBITSTREAM_RESTRICTED_USER RABBITSTREAM_RESTRICTED_PASSWORD - - record_resource compose "${integration}" "${integration}/standalone-compose.yaml" \ - "${standalone_project}" - ( - cd "${integration}" - COMPOSE_PROJECT_NAME="${standalone_project}" ./standalone-setup.sh - ) - write_service_environment RABBITSTREAM_TEST_HOST localhost - write_service_environment RABBITSTREAM_TEST_PORT 15552 - write_service_environment RABBITSTREAM_TEST_USER "${RABBITSTREAM_USER}" - write_service_environment RABBITSTREAM_TEST_PASSWORD "${RABBITSTREAM_PASSWORD}" - write_service_environment RABBITSTREAM_TEST_RESTART_CONTAINER \ - "${standalone_project}-rabbit-1" - write_service_environment RABBITSTREAM_TEST_PROXY_API http://127.0.0.1:18474 - write_service_environment RABBITSTREAM_TEST_PROXY_NAME rabbitstream - - [[ "${mode}" == "full" ]] || return 0 - - record_resource compose "${integration}" "${integration}/compose.yaml" \ - "${cluster_project}" - ( - cd "${integration}" - COMPOSE_PROJECT_NAME="${cluster_project}" ./setup.sh - ) - write_service_environment RABBITSTREAM_CLUSTER_PORTS 15561,15562,15563 - write_service_environment RABBITSTREAM_CLUSTER_CONTAINERS \ - "15561=${cluster_project}-rabbit1-1,15562=${cluster_project}-rabbit2-1,15563=${cluster_project}-rabbit3-1" - write_service_environment RABBITSTREAM_CLUSTER_PROJECT "${cluster_project}" - write_service_environment RABBITSTREAM_ERLANG_COOKIE "${RABBITSTREAM_ERLANG_COOKIE}" - write_service_environment RABBITSTREAM_UPGRADE_IMAGE \ - "rabbitmq@sha256:397fde82bc04522d88680b57cbf5d70caae715a76c957404e52e3f0fa056b8f3" - write_service_environment RABBITSTREAM_UPGRADE_FROM_VERSION 4.3.4 - write_service_environment RABBITSTREAM_UPGRADE_TO_VERSION 4.3.5 - - fixture_parent="$(mktemp -d "${TMPDIR:-/tmp}/golib-rabbitstream.XXXXXX")" - tls_runtime="${fixture_parent}/tls" - record_resource directory "${fixture_parent}" - record_resource compose "${integration}" "${integration}/tls-compose.yaml" \ - "${tls_project}" - ( - cd "${integration}" - RABBITSTREAM_TLS_RUNTIME="${tls_runtime}" \ - COMPOSE_PROJECT_NAME="${tls_project}" ./tls-setup.sh - ) - write_service_environment RABBITSTREAM_TLS_HOST localhost - write_service_environment RABBITSTREAM_TLS_PORT 15571 - write_service_environment RABBITSTREAM_TLS_USER "${RABBITSTREAM_USER}" - write_service_environment RABBITSTREAM_TLS_PASSWORD "${RABBITSTREAM_PASSWORD}" - write_service_environment RABBITSTREAM_TLS_RUNTIME "${tls_runtime}" - write_service_environment RABBITSTREAM_RESTRICTED_USER \ - "${RABBITSTREAM_RESTRICTED_USER}" - write_service_environment RABBITSTREAM_RESTRICTED_PASSWORD \ - "${RABBITSTREAM_RESTRICTED_PASSWORD}" -} - -while IFS= read -r service; do - case "${service}" in - postgresql) - container="golib-postgres-${slug}" - postgres_version="${POSTGRES_IMAGE#postgres:}" - postgres_version="${postgres_version%-alpine}" - docker run --detach --name "${container}" -p 127.0.0.1::5432 \ - -e POSTGRES_USER=golib -e POSTGRES_PASSWORD=golib \ - -e POSTGRES_DB=golib "${POSTGRES_IMAGE}" >/dev/null - record "${container}" - wait_for "${container}" pg_isready -U golib -d golib - port="$(published_port "${container}" 5432)" - cat >>"${environment_file}" </dev/null - record "${container}" - wait_for "${container}" valkey-cli ping - port="$(published_port "${container}" 6379)" - cat >>"${environment_file}" </dev/null - record "${container}" - wait_for "${container}" redis-cli ping - port="$(published_port "${container}" 6379)" - cat >>"${environment_file}" </dev/null - record "${container}" - sleep 2 - port="$(published_port "${container}" 4222)" - printf 'NATS_URL=nats://127.0.0.1:%s\n' "${port}" >>"${environment_file}" - ;; - nsq) - container="golib-nsq-${slug}" - docker run --detach --name "${container}" -p 127.0.0.1::4150 \ - "${NSQ_IMAGE}" /nsqd --broadcast-address=127.0.0.1 >/dev/null - record "${container}" - sleep 2 - port="$(published_port "${container}" 4150)" - printf 'NSQD_TCP_ADDRESS=127.0.0.1:%s\n' "${port}" >>"${environment_file}" - ;; - rabbitmq) - container="golib-rabbitmq-${slug}" - docker run --detach --name "${container}" --hostname "${container}" \ - --user rabbitmq \ - -p 127.0.0.1::5672 \ - "${RABBITMQ_IMAGE}" >/dev/null - record "${container}" - wait_for "${container}" rabbitmq-diagnostics -q ping - port="$(published_port "${container}" 5672)" - printf 'RABBITMQ_URL=amqp://guest:guest@127.0.0.1:%s/\n' \ - "${port}" >>"${environment_file}" - ;; - rabbitstream) - start_rabbitstream full - ;; - rabbitstream-standalone) - start_rabbitstream standalone - ;; - opensearch) - # shellcheck source=/dev/null - source "${root}/.golib/services/opensearch/opensearch-images.env" - container="golib-opensearch-${slug}" - opensearch_image="${opensearch_image_repository}@${opensearch_new_digest}" - docker run --detach --name "${container}" -p 127.0.0.1::9200 \ - --cpus=1 --memory=1g --pids-limit=512 \ - --ulimit nofile=1024:1024 \ - -e discovery.type=single-node \ - -e DISABLE_SECURITY_PLUGIN=true \ - -e OPENSEARCH_JAVA_OPTS='-Xms512m -Xmx512m' \ - "${opensearch_image}" >/dev/null - record "${container}" - port="$(published_port "${container}" 9200)" - ready=0 - for _ in {1..120}; do - if curl --connect-timeout 2 --max-time 5 --fail --silent \ - "http://127.0.0.1:${port}/" >/dev/null; then - ready=1 - break - fi - if [[ "$(docker inspect --format '{{.State.Running}}' "${container}" 2>/dev/null || true)" != "true" ]]; then - break - fi - sleep 1 - done - if [[ "${ready}" -ne 1 ]]; then - docker logs "${container}" >&2 || true - printf 'service did not become healthy: %s\n' "${container}" >&2 - exit 1 - fi - cat >>"${environment_file}" <&2 - exit 1 - ;; - esac -done <<<"${services}" diff --git a/.golib/scripts/stop-services.sh b/.golib/scripts/stop-services.sh deleted file mode 100755 index a3ac576..0000000 --- a/.golib/scripts/stop-services.sh +++ /dev/null @@ -1,110 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 1 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -state_file="$1" -[[ -f "${state_file}" ]] || exit 0 -cleanup_timeout="${GOLIB_DOCKER_CLEANUP_TIMEOUT_SECONDS:-30}" -if [[ ! "${cleanup_timeout}" =~ ^[1-9][0-9]*$ ]]; then - printf 'GOLIB_DOCKER_CLEANUP_TIMEOUT_SECONDS must be a positive integer\n' >&2 - exit 2 -fi - -remove_container() { - local container="$1" - local started="${SECONDS}" - - docker rm --force "${container}" >/dev/null 2>&1 & - local docker_pid=$! - while kill -0 "${docker_pid}" >/dev/null 2>&1; do - if ((SECONDS - started >= cleanup_timeout)); then - kill -TERM "${docker_pid}" >/dev/null 2>&1 || true - sleep 0.1 - kill -KILL "${docker_pid}" >/dev/null 2>&1 || true - wait "${docker_pid}" >/dev/null 2>&1 || true - printf 'timed out removing Docker container %s after %ss\n' \ - "${container}" "${cleanup_timeout}" >&2 - return - fi - sleep 0.1 - done - wait "${docker_pid}" >/dev/null 2>&1 || true -} - -remove_compose_project() { - local directory="$1" - local file="$2" - local project="$3" - local started="${SECONDS}" - - env RABBITSTREAM_USER=cleanup RABBITSTREAM_PASSWORD=cleanup \ - RABBITSTREAM_ERLANG_COOKIE=cleanup \ - docker compose --project-directory "${directory}" -f "${file}" \ - -p "${project}" down --volumes --remove-orphans >/dev/null 2>&1 & - local docker_pid=$! - while kill -0 "${docker_pid}" >/dev/null 2>&1; do - if ((SECONDS - started >= cleanup_timeout)); then - kill -TERM "${docker_pid}" >/dev/null 2>&1 || true - sleep 0.1 - kill -KILL "${docker_pid}" >/dev/null 2>&1 || true - wait "${docker_pid}" >/dev/null 2>&1 || true - printf 'timed out removing Docker Compose project %s after %ss\n' \ - "${project}" "${cleanup_timeout}" >&2 - return - fi - sleep 0.1 - done - wait "${docker_pid}" >/dev/null 2>&1 || true -} - -remove_owned_directory() { - local directory="$1" - case "$(basename "${directory}")" in - golib-rabbitstream.*) ;; - *) - printf 'refusing to remove unexpected service directory: %s\n' \ - "${directory}" >&2 - return 1 - ;; - esac - [[ -d "${directory}" ]] || return - chmod -R u+w "${directory}" 2>/dev/null || true - find "${directory}" -depth -delete -} - -lock_path="" -lock_owner="" - -while IFS=$'\t' read -r kind first second third; do - [[ -n "${kind}" ]] || continue - case "${kind}" in - container) remove_container "${first}" ;; - compose) remove_compose_project "${first}" "${second}" "${third}" ;; - directory) remove_owned_directory "${first}" ;; - lock) - lock_path="${first}" - lock_owner="${second}" - ;; - *) remove_container "${kind}" ;; - esac -done <"${state_file}" - -if [[ -n "${lock_path}" ]]; then - case "$(basename "${lock_path}")" in - golib-rabbitstream-fixture.lock) ;; - *) - printf 'refusing to remove unexpected service lock: %s\n' \ - "${lock_path}" >&2 - exit 1 - ;; - esac - current_owner="$(cat "${lock_path}/owner" 2>/dev/null || true)" - if [[ "${current_owner}" == "${lock_owner}" ]]; then - rm -f "${lock_path}/owner" - rmdir "${lock_path}" 2>/dev/null || true - fi -fi diff --git a/.golib/scripts/test-oidc-keycloak-interoperability.sh b/.golib/scripts/test-oidc-keycloak-interoperability.sh deleted file mode 100755 index b87f798..0000000 --- a/.golib/scripts/test-oidc-keycloak-interoperability.sh +++ /dev/null @@ -1,81 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -root="$(git rev-parse --show-toplevel)" -# shellcheck source=/dev/null -source "${root}/.golib/versions.env" -: "${KEYCLOAK_IMAGE:?KEYCLOAK_IMAGE must be pinned}" -expected_version="26.3.2" -fixture="${root}/pkg/authentication/oidc/testdata/keycloak-26.3.2-realm.json" -run_directory="$(mktemp -d)" -state_file="${run_directory}/containers" -token_file="${run_directory}/id-token" -container="golib-oidc-keycloak-${$}-${RANDOM}" -: >"${state_file}" - -cleanup() { - "${root}/.golib/scripts/stop-services.sh" "${state_file}" - find "${run_directory}" -depth -delete -} -trap cleanup EXIT - -command -v docker >/dev/null || { - printf 'Docker is required for OIDC Keycloak interoperability\n' >&2 - exit 1 -} - -printf '%s\n' "${container}" >"${state_file}" -docker run --detach --name "${container}" --publish 127.0.0.1::8080 \ - --mount "type=bind,src=${fixture},dst=/opt/keycloak/data/import/realm.json,readonly" \ - --env KC_BOOTSTRAP_ADMIN_USERNAME=admin \ - --env KC_BOOTSTRAP_ADMIN_PASSWORD=interoperability-admin \ - "${KEYCLOAK_IMAGE}" start-dev --import-realm --hostname-strict=false >/dev/null -port="$(docker port "${container}" 8080/tcp | tail -1 | sed 's/.*://')" -issuer="http://127.0.0.1:${port}/realms/oidc-hardening" -ready=false -for _ in {1..120}; do - if curl --fail --silent --show-error \ - "${issuer}/.well-known/openid-configuration" >/dev/null 2>&1; then - ready=true - break - fi - if [[ "$(docker inspect --format '{{.State.Running}}' "${container}" 2>/dev/null || true)" != "true" ]]; then - break - fi - sleep 1 -done -if [[ "${ready}" != "true" ]]; then - docker logs "${container}" >&2 || true - printf 'Keycloak did not become ready\n' >&2 - exit 1 -fi - -reported_version="$(docker exec "${container}" /opt/keycloak/bin/kc.sh --version)" -if [[ "${reported_version}" != *"${expected_version}"* ]]; then - printf 'unexpected Keycloak version: %s\n' "${reported_version}" >&2 - exit 1 -fi - -token_response="$(curl --fail --silent --show-error \ - --request POST "${issuer}/protocol/openid-connect/token" \ - --data-urlencode grant_type=password \ - --data-urlencode client_id=oidc-client \ - --data-urlencode username=alice \ - --data-urlencode password=interoperability-password \ - --data-urlencode scope=openid)" -id_token="$(jq -er '.id_token | strings | select(length > 0)' <<<"${token_response}")" -umask 077 -printf '%s\n' "${id_token}" >"${token_file}" -unset token_response id_token - -OIDC_INTEROP_ISSUER="${issuer}" \ -OIDC_INTEROP_TOKEN_FILE="${token_file}" \ -OIDC_INTEROP_CLIENT_ID=oidc-client \ -GOCACHE="${run_directory}/go-cache" \ -GOWORK=off \ -go test ./... -tags=integration \ - -run '^(TestGoogleProviderMetadataSnapshot|TestRepresentativeProviderMetadataProfiles|TestKeycloakProviderIssuedIDToken)$' \ - -count=1 - -printf 'Keycloak interoperability passed: version=%s image=%s\n' \ - "${expected_version}" "${KEYCLOAK_IMAGE}" diff --git a/.golib/scripts/update-api-baseline.sh b/.golib/scripts/update-api-baseline.sh deleted file mode 100755 index 69f9085..0000000 --- a/.golib/scripts/update-api-baseline.sh +++ /dev/null @@ -1,48 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 1 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -# shellcheck disable=SC1091 # Canonical repository tool versions. -source "${root}/.golib/versions.env" -case "$1" in - /*) directory="$1" ;; - *) directory="${root}/$1" ;; -esac -if [[ ! -f "${directory}/go.mod" ]]; then - printf 'module has no go.mod: %s\n' "$1" >&2 - exit 1 -fi - -module_path="$( - cd "${directory}" - GOWORK=off go mod edit -json | jq -er '.Module.Path' -)" -baseline="${directory}/api/baseline.txt" -temporary="$(mktemp "${TMPDIR:-/tmp}/golib-api-baseline.XXXXXX")" -cleanup() { - rm -f "${temporary}" -} -trap cleanup EXIT HUP INT TERM - -run_apidiff() { - if [[ -n "${GOLIB_APIDIFF:-}" ]]; then - GOWORK=off go exec-tool "${GOLIB_APIDIFF}" "$@" - else - GOWORK=off go run \ - "golang.org/x/exp/cmd/apidiff@${APIDIFF_VERSION}" "$@" - fi -} - -mkdir -p "$(dirname "${baseline}")" -( - cd "${directory}" - run_apidiff -m -w "${temporary}" "${module_path}" -) -test -s "${temporary}" -mv "${temporary}" "${baseline}" -trap - EXIT HUP INT TERM diff --git a/.golib/scripts/verify-gate-evidence.sh b/.golib/scripts/verify-gate-evidence.sh deleted file mode 100755 index 356d3ee..0000000 --- a/.golib/scripts/verify-gate-evidence.sh +++ /dev/null @@ -1,118 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 2 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -module="$1" -gate="$2" -artifact="${root}/.artifacts/${module}/evidence" -lock_root="${artifact}/.locks" -lock="${lock_root}/${gate}.lock" -lock_acquired=0 - -cleanup() { - if [[ "${lock_acquired}" -eq 1 ]]; then - owner="$(readlink "${lock}" 2>/dev/null || true)" - if [[ "${owner}" == "$$" ]]; then - rm -f "${lock}" - fi - fi -} - -trap cleanup EXIT -trap 'exit 129' HUP -trap 'exit 130' INT -trap 'exit 143' TERM -mkdir -p "${lock_root}" -while ! ln -s "$$" "${lock}" 2>/dev/null; do - if [[ -L "${lock}" ]]; then - owner="$(readlink "${lock}" 2>/dev/null || true)" - if [[ "${owner}" =~ ^[0-9]+$ ]] && ! kill -0 "${owner}" 2>/dev/null; then - rm -f "${lock}" - continue - fi - elif [[ -d "${lock}" ]]; then - owner="$(cat "${lock}/owner" 2>/dev/null || true)" - if [[ "${owner}" =~ ^[0-9]+$ ]] && ! kill -0 "${owner}" 2>/dev/null; then - rm -f "${lock}/owner" - rmdir "${lock}" 2>/dev/null || true - continue - fi - if [[ -z "${owner}" ]] && rmdir "${lock}" 2>/dev/null; then - continue - fi - elif [[ -f "${lock}" ]]; then - owner="$(cat "${lock}" 2>/dev/null || true)" - if [[ "${owner}" =~ ^[0-9]+$ ]] && ! kill -0 "${owner}" 2>/dev/null; then - rm -f "${lock}" - continue - fi - if [[ -z "${owner}" ]]; then - rm -f "${lock}" - continue - fi - fi - sleep 0.05 -done -lock_acquired=1 - -input_digest="$("${root}/.golib/scripts/gate-input-digest.sh" "${gate}" "${module}")" -digest_artifact="${artifact}/by-input/${gate}" -evidence="${digest_artifact}/${input_digest}.json" -log="${digest_artifact}/${input_digest}.log" -legacy_evidence="${artifact}/${gate}.json" -legacy_log="${artifact}/${gate}.log" - -if [[ ! -f "${evidence}" || ! -f "${log}" ]]; then - evidence="${legacy_evidence}" - log="${legacy_log}" -fi - -[[ -f "${evidence}" ]] || { - printf '[%s] missing %s evidence\n' "${module}" "${gate}" >&2 - exit 1 -} -[[ -f "${log}" ]] || { - printf '[%s] missing %s evidence log\n' "${module}" "${gate}" >&2 - exit 1 -} - -log_sha256="$(shasum -a 256 "${log}" | awk '{print $1}')" -jq -e \ - --arg module "${module}" \ - --arg gate "${gate}" \ - --arg input_digest "${input_digest}" \ - --arg log_sha256 "${log_sha256}" \ - ' - .schema_version == 1 and - .module == $module and - .gate == $gate and - ( - .result == "passed" or - .result == "not_applicable" or - ($gate == "nilaway" and .result == "advisory") - ) and - .exit_code == 0 and - .input_digest == $input_digest and - .completed_input_digest == $input_digest and - .log_sha256 == $log_sha256 - ' "${evidence}" >/dev/null || { - printf '[%s] stale or invalid %s evidence\n' "${module}" "${gate}" >&2 - exit 1 -} - -if [[ "$(jq -r '.result' "${evidence}")" == "advisory" ]] && - ! grep -Eq 'NilAway advisory exit status: [1-9][0-9]*$' "${log}"; then - printf '[%s] stale or invalid %s evidence\n' "${module}" "${gate}" >&2 - exit 1 -fi - -if [[ "$(jq -r '.result' "${evidence}")" == "not_applicable" ]] && - ! grep -Fq ': not applicable by catalog policy' "${log}"; then - printf '[%s] stale or invalid %s evidence\n' "${module}" "${gate}" >&2 - exit 1 -fi diff --git a/.golib/scripts/with-disposable-go-cache.sh b/.golib/scripts/with-disposable-go-cache.sh deleted file mode 100755 index 35b3ba3..0000000 --- a/.golib/scripts/with-disposable-go-cache.sh +++ /dev/null @@ -1,18 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -eq 0 ]]; then - printf 'usage: %s [arguments...]\n' "$0" >&2 - exit 2 -fi - -gocache="$(mktemp -d "${TMPDIR:-/tmp}/golib-gocache.XXXXXX")" -gomodcache="$(mktemp -d "${TMPDIR:-/tmp}/golib-modcache.XXXXXX")" -cleanup() { - chmod -R u+w "${gocache}" "${gomodcache}" 2>/dev/null || true - find "${gocache}" -depth -delete 2>/dev/null || true - find "${gomodcache}" -depth -delete 2>/dev/null || true -} -trap cleanup EXIT HUP INT TERM - -GOCACHE="${gocache}" GOMODCACHE="${gomodcache}" "$@" diff --git a/.golib/versions.env b/.golib/versions.env deleted file mode 100644 index c7e8a97..0000000 --- a/.golib/versions.env +++ /dev/null @@ -1,20 +0,0 @@ -GO_VERSION=1.26.6 -GOLANGCI_LINT_VERSION=v2.12.2 -STATICCHECK_VERSION=v0.7.0 -NILAWAY_VERSION=v0.0.0-20260720194628-9fd1b8d7bac8 -GOVULNCHECK_VERSION=v1.6.0 -GREMLINS_VERSION=v0.6.0 -GREMLINS_SUM=h1:3G2ROO0I3q4bb5bxElQIUITTuEbl1iOfVYFqunGwrJI= -GREMLINS_GOMOD_SUM=h1:LLbvJR33CWsu1sgvQ4qMzU2rqkwYJK3Qy/Al59eHKjA= -GITLEAKS_VERSION=v8.30.1 -GO_LICENSES_VERSION=v2.0.1 -CYCLONEDX_VERSION=v1.10.0 -ACTIONLINT_VERSION=v1.7.12 -APIDIFF_VERSION=v0.0.0-20260718201538-764159d718ef -POSTGRES_IMAGE=postgres:18.4-alpine -VALKEY_IMAGE=valkey/valkey:9.1.0-alpine -REDIS_IMAGE=redis:8.6.4-alpine -NATS_IMAGE=nats:2.14.2-alpine -NSQ_IMAGE=nsqio/nsq:v1.3.0 -RABBITMQ_IMAGE=rabbitmq:4.3.2-management-alpine -KEYCLOAK_IMAGE=quay.io/keycloak/keycloak@sha256:98fab020a3a490aba0978f237e2a06cd0ea42bf149c6cf10f11c0aaf27728ff2 diff --git a/.golib/mutation-bootstrap/root.zip b/.verification/mutation/bootstrap/root.zip similarity index 100% rename from .golib/mutation-bootstrap/root.zip rename to .verification/mutation/bootstrap/root.zip diff --git a/.golib/mutation-history-migrations.json b/.verification/mutation/migration-ledger.json similarity index 100% rename from .golib/mutation-history-migrations.json rename to .verification/mutation/migration-ledger.json diff --git a/.golib/mutation-zero-inventory.json b/.verification/mutation/zero-inventory.json similarity index 100% rename from .golib/mutation-zero-inventory.json rename to .verification/mutation/zero-inventory.json diff --git a/CHANGELOG.md b/CHANGELOG.md index 5b374af..fed863c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,11 @@ ## Unreleased +### Changed + +- Replace copied repository tooling with the pinned `go-library-tools` v1.0.4 + contract while retaining package-owned policy and verification evidence. + ### Documentation - Replace archived monorepo links and completed execution artifacts with a diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b20fa76..5d9f2db 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -18,6 +18,11 @@ New direct dependencies and dependency updates must follow the [dependency governance policy](AGENTS.md#dependencies-and-supply-chain). Package-local update bots are forbidden; the root policy owns every module and action update. +Repository verification is provided by the released `go-library-tools` CLI. +Keep package-specific policy in `modules.json`, `packages.json`, +the repository configuration, and `verification/package.mk`; do not recreate +shared tooling in the consumer repository. + Specification-backed changes must follow the [specification governance contract](AGENTS.md#design), update the affected stable decision entries, and complete the Specification Decisions diff --git a/Makefile b/Makefile index 16b94f5..3098c3e 100644 --- a/Makefile +++ b/Makefile @@ -1,11 +1,13 @@ -SHELL := /usr/bin/env bash +GOLIB ?= golib .PHONY: check ci inventory repository-check check: - ./.golib/scripts/with-disposable-go-cache.sh ./.golib/scripts/run-modules.sh check --all + $(GOLIB) check --all -ci: repository-check check +ci: + $(GOLIB) repository check + $(GOLIB) check --all inventory repository-check: - ./.golib/scripts/repository-check.sh + $(GOLIB) repository check diff --git a/scripts/check-api-compat.sh b/scripts/check-api-compat.sh deleted file mode 100755 index 97d08c3..0000000 --- a/scripts/check-api-compat.sh +++ /dev/null @@ -1,5 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -root="$(git rev-parse --show-toplevel)" -exec "${root}/.golib/scripts/check-api-baseline.sh" . diff --git a/scripts/check-coverage.sh b/scripts/check-coverage.sh deleted file mode 100755 index b16f397..0000000 --- a/scripts/check-coverage.sh +++ /dev/null @@ -1,12 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -profile="${TMPDIR:-/tmp}/lease-coverage.out" -packages="$(go list ./... | grep -v '/leasetest$' | grep -v '/examples/' | paste -sd, -)" -go test ./... -coverpkg="$packages" -coverprofile="$profile" -total="$(go tool cover -func="$profile" | awk '/^total:/ {print $3}')" -if [[ "$total" != "100.0%" ]]; then - echo "meaningful production statement coverage: $total (required 100.0%)" >&2 - exit 1 -fi -echo "meaningful production statement coverage: $total" diff --git a/scripts/check-fuzz.sh b/scripts/check-fuzz.sh deleted file mode 100755 index 5166237..0000000 --- a/scripts/check-fuzz.sh +++ /dev/null @@ -1,7 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -duration="${1:-2s}" -go test . -run '^$' -fuzz '^FuzzKeyParsing$' -fuzztime="$duration" -go test . -run '^$' -fuzz '^FuzzPolicyBounds$' -fuzztime="$duration" -go test ./memory -run '^$' -fuzz '^FuzzLeaseStateModel$' -fuzztime="$duration" diff --git a/scripts/check-stress.sh b/scripts/check-stress.sh deleted file mode 100755 index 7f8fef6..0000000 --- a/scripts/check-stress.sh +++ /dev/null @@ -1,7 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -count="${STRESS_COUNT:-25}" -pattern='^(TestContentionElectsExactlyOneOwner|TestAcquirePropagatesBackendAndCancellation|TestHandleLifecycleFailsClosed|TestHandleRejectsConcurrentOperationsWithoutBlockingState|TestHandleFailsClosedWhenDeadlinePassesDuringOperation|TestManagedRenewalReportsUncertaintyAndStopsAdmission|TestManagedStopHonorsCallerDeadline|TestStaleOwnerCannotAffectSuccessor|TestAcquireRacingShutdownReleasesReservation|TestManagerBoundsHandlesAndReleasesOnShutdown|TestWorkerCancelsOwnershipSensitiveJobOnLoss|TestCoordinatorCancelsOwnershipSensitiveTaskOnLoss)$' - -go test -race -shuffle=on -count="$count" -run "$pattern" ./... diff --git a/verification/package.mk b/verification/package.mk new file mode 100644 index 0000000..8e3fae4 --- /dev/null +++ b/verification/package.mk @@ -0,0 +1,10 @@ +.PHONY: docs fence-mutations backend-hardening + +docs: + ./scripts/check-docs.sh + +fence-mutations: + ./scripts/check-fence-mutations.sh + +backend-hardening: + ./scripts/check-backend-faults.sh