diff --git a/.ai/GOAL.md b/.ai/GOAL.md deleted file mode 100644 index 2c36c0c..0000000 --- a/.ai/GOAL.md +++ /dev/null @@ -1,158 +0,0 @@ -# Goal: Fenced Distributed Leases - -## Objective - -Build a production-grade distributed lease package with explicit ownership, -expiry, renewal, release, and monotonically increasing fencing tokens for -Valkey and PostgreSQL. - -The package MUST provide the coordination primitive required by unique queue -jobs, scheduler overlap prevention, single-owner maintenance work, and bounded -distributed leadership without pretending that an ordinary cache lock provides -stronger guarantees than the backend can deliver. - -## Product Principles - -- A lease is time-bounded ownership, not an indefinitely held mutex. -- Every successful acquisition returns an opaque owner identity and fencing - token that protected resources can use to reject stale owners. -- Expiry, renewal, loss, release, and backend uncertainty are distinct states. -- No process may assume ownership after its lease deadline or uncertain renewal. -- Backend guarantees and clock assumptions are explicit and testable. -- Context cancellation and shutdown never imply successful remote release. - -## Core Model - -- Typed, bounded, namespaced lease keys. -- Immutable acquisition policy: TTL, wait, retry, jitter, renewal, and failure - behavior. -- Lease handle exposing owner, fencing token, acquired time, deadline, state, - renew, validate, and release. -- `TryAcquire`, bounded `Acquire`, explicit `Renew`, and idempotent `Release`. -- Optional managed renewal with explicit goroutine ownership and loss channel. -- Stable errors for contention, timeout, cancellation, lost lease, stale owner, - backend unavailable, invalid state, and ambiguous outcome. -- Test clock and deterministic retry source without production global state. - -## Correctness Semantics - -- Fencing tokens MUST be monotonically increasing for a key within documented - backend continuity guarantees. -- Renewal MUST compare owner identity and current token atomically. -- Release MUST compare ownership atomically and never delete a successor lease. -- A stale handle MUST never renew or release a newer owner's lease. -- TTL and safety margin account for network delay, pauses, scheduling, and - backend clock behavior. -- Acquisition fairness is explicitly documented; no unsupported fairness claim. -- Multi-key atomic leases are out of scope unless a proven backend transaction - model and deadlock policy are added later. - -## Valkey Adapter - -- Native `valkey-go` implementation using atomic server-side scripts/functions. -- Cluster-safe key layout, script loading, `NOSCRIPT`, failover, reconnect, - timeout, ACL, TLS, and rolling-version behavior. -- Backend/server time SHOULD anchor expiry semantics where practical. -- Define fencing continuity after failover, restore, flush, or data loss. -- No Redlock claims or multi-independent-master algorithm by default. - -## PostgreSQL Adapter - -- Native `pgx` implementation with transactional acquisition and renewal. -- Durable lease row and monotonic fencing sequence semantics. -- Indexed schema, cleanup, contention, isolation, deadlock, failover, and - connection-loss behavior. -- Migrations owned through `migrations`. -- PostgreSQL advisory locks MAY be evaluated separately but MUST NOT be confused - with durable TTL leases or reused across pooled sessions unsafely. - -## Integration - -- `queue` middleware for unique jobs and non-overlapping handlers. -- `scheduler` adapter for `onOneServer` and `withoutOverlapping` semantics. -- `idempotency` MAY consume lease/fencing primitives where its stronger - operation state machine remains intact. -- `service` lifecycle integration for managed renewal and shutdown. -- Optional `log` and `telemetry` observations with hashed bounded keys. -- Protected-resource examples MUST demonstrate fencing checks; acquiring a - lease alone is not sufficient safety documentation. -- Valkey MAY expose opaque active-ownership coordinates to a protected writer - that atomically compares the exact lease owner and token with a - reconstructible cache write. - -## Security And Resource Bounds - -- Cryptographically random owner identities with injectable test source. -- Bounded key, owner, waiters, retry attempts, renewal goroutines, observations, - cleanup batches, and backend operations. -- Keys and owner identities MUST not leak through default logs or metric labels. -- Threat-model stale writers, split brain, clock anomalies, replay, token - overflow, key collision, backend rollback, restore, and malicious contention. -- Callbacks MUST not execute while internal locks are held. - -## Non-Goals - -- No distributed transaction, consensus system, membership service, election - platform, semaphore, idempotency state machine, or queue. -- No guarantee that expired work stopped; fencing is required for protected - resources when stale work is dangerous. -- No hidden infinite waiting or retry. -- No in-memory adapter presented as distributed. -- No Redis/Valkey compatibility through one ambiguous adapter. - -## Package Shape - -- Root: keys, policies, handles, states, errors, retry, observations. -- `memory`: deterministic process-local reference and tests only, clearly scoped. -- `valkey` and `postgres`: native distributed adapters. -- `leasequeue`, `leasescheduler`, and `leaseservice`: integrations. -- `leasetest`: conformance, clocks, fault injection, and fencing assertions. - -## Testing And Quality Standard - -Meaningful 100% production statement coverage is mandatory. Required evidence: - -- state-machine and model-based tests for acquisition through final release -- cross-backend lease and fencing conformance -- race/stress tests for contention, renewal, loss, cancellation, and shutdown -- Valkey and PostgreSQL failover, restart, timeout, partition, and fault tests -- stale-owner and successor-protection tests at every operation boundary -- clock skew/jump, process pause, token overflow, and retry fuzzing -- mutation testing of ownership comparisons and stale-owner rejection -- benchmarks for contention, renewal load, latency, allocations, and cleanup - -## Documentation Deliverables - -- Five-minute Valkey and PostgreSQL quickstarts. -- Formal state machine, fencing model, backend guarantees, and API reference. -- Guides for unique jobs, schedulers, protected writes, renewal, loss handling, - shutdown, Kubernetes, failover, and migrations. -- Laravel lock/unique-job migration guide, threat model, operations runbook, - performance, FAQ, troubleshooting, examples, and changelog. -- Every exported API and user-facing scenario MUST be documented. - -## Automation And Release - -Use the latest stable Go release as the minimum at implementation time. Pin all -tools and dependencies. GitHub Actions MUST run formatting, vet, Staticcheck, -strict golangci-lint, advisory NilAway, tests, meaningful 100% coverage, race, -fuzz smoke, mutation checks, Valkey/PostgreSQL matrices, vulnerability scans, -benchmarks, docs, API compatibility, and releases. All blocking commands MUST -be locally reproducible through documented `make` targets. - -## Execution Plan - -1. Specify states, owner identity, fencing, timing, errors, and conformance. -2. Implement deterministic reference behavior and native Valkey adapter. -3. Implement PostgreSQL adapter and migration contract. -4. Add queue, scheduler, service, logging, and telemetry integrations. -5. Complete failover, stale-owner, race, mutation, and performance hardening. -6. Publish complete operational documentation and release v1. - -## Acceptance Criteria - -- Stale owners cannot renew, release, or overwrite protected successor work. -- Every backend satisfies the documented fencing and lease state machine. -- Lease uncertainty and loss always stop ownership-dependent admission. -- Resource use, retries, wait, renewal, and shutdown are bounded. -- Meaningful 100% coverage and every required CI gate pass. diff --git a/.ai/GOAL_HARDEN.md b/.ai/GOAL_HARDEN.md deleted file mode 100644 index 2ec879f..0000000 --- a/.ai/GOAL_HARDEN.md +++ /dev/null @@ -1,67 +0,0 @@ -# Hardening Goal: Fenced Distributed Leases - -## Objective - -Prove that `lease` prevents stale-owner corruption and successor deletion -under contention, pauses, clock anomalies, backend failure, failover, restore, -network uncertainty, cancellation, and rolling deployments. - -## Required Audits - -### State And Fencing Audit - -- Model every acquisition, contention, renewal, expiry, loss, validation, - release, and successor transition. -- Mutation-test owner and token comparisons capable of accepting stale work. -- Prove fencing tokens remain monotonic within documented continuity bounds. -- Verify stale handles cannot affect a successor after every failure point. - -### Timing And Concurrency Audit - -- Inject clock rollback/jump, scheduler delay, process pause, stop-the-world-like - delay, network latency, frozen time, and late responses. -- Race and stress-test acquisition, managed renewal, explicit renewal, release, - cancellation, loss notification, and shutdown. -- Prove callbacks and observations cannot deadlock lease state transitions. -- Enforce wait, retry, jitter, goroutine, and renewal safety margins. - -### Backend Failure Audit - -- Exercise Valkey failover, restart, `NOSCRIPT`, ACL/TLS rotation, disconnect, - restore, flush, and ambiguous write outcomes. -- Exercise PostgreSQL failover, deadlocks, transaction abort, pool churn, - isolation, sequence behavior, restore, and cleanup races. -- Document exactly when fencing continuity can reset and how operators detect it. -- Test rolling clients with compatible and incompatible schema/script versions. - -### Security And Integration Audit - -- Threat-model split brain, stale writer, replay, owner spoofing, key collision, - denial of service, token overflow, and sensitive key leakage. -- Prove queue and scheduler integrations stop ownership-sensitive work on loss. -- Prove protected-write examples reject stale fencing tokens transactionally. -- Verify errors, logs, metrics, traces, and inspection output redact identifiers. - -## Required Deliverables - -- Formal state machine, fencing proof matrix, and backend continuity contract. -- Threat model, failure matrix, resource budgets, and hardening findings. -- Valkey/PostgreSQL failover, restore, fault, and rolling-upgrade evidence. -- Race, fuzz, mutation, stale-owner, and benchmark reports. -- Updated API, operations, security, migration, FAQ, and troubleshooting docs. - -## Release Blockers - -- Any stale owner that can renew, release, or commit a protected write. -- Any successor lease deleted by an earlier handle. -- Any undocumented fencing reset, ambiguous ownership treated as valid, race, - deadlock, panic, leaked renewal goroutine, or unbounded wait/retry. -- Missing meaningful 100% coverage, mutation evidence, or green blocking CI. - -## Completion Criteria - -- State, fencing, timing, backend, restore, and integration suites pass. -- Every ambiguity fails ownership closed and is observable. -- Race, fuzz, vulnerability, compatibility, and performance gates pass. -- NilAway runs visibly as advisory without blocking findings. -- No release blocker remains and the changelog is current. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1f8df68..3277d6a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -9,261 +9,30 @@ on: workflow_dispatch: inputs: release_dry_run: - description: Run the stable v1 dry-run for every releasable module + description: Run the stable release rehearsal required: false default: false type: boolean permissions: - actions: read contents: read - -concurrency: - group: go-lease-${{ github.event.pull_request.number || github.ref }} - cancel-in-progress: ${{ github.event_name == 'pull_request' }} + security-events: write jobs: - prepare: - name: Select modules - runs-on: ubuntu-24.04 - timeout-minutes: 15 - outputs: - matrix: ${{ steps.selection.outputs.matrix }} - steps: - - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 - - name: Select repository modules - id: selection - env: - RELEASE_DRY_RUN: ${{ inputs.release_dry_run }} - run: | - set -euo pipefail - filter='.' - if [[ "${RELEASE_DRY_RUN}" == true ]]; then - filter='map(select(.releasable == true))' - fi - matrix="$( - jq -c \ - --arg filter "${filter}" ' - .modules - | if $filter == "." then . else map(select(.releasable == true)) end - | map({ - directory, - artifact: ( - if .directory == "." then "root" - else (.directory | gsub("/"; "-")) - end - ) - }) - ' modules.json - )" - [[ "$(jq 'length' <<<"${matrix}")" -gt 0 ]] - echo "matrix=${matrix}" >>"${GITHUB_OUTPUT}" - - repository-contract: - name: Repository contract - runs-on: ubuntu-24.04 - timeout-minutes: 15 - steps: - - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 - with: - fetch-depth: 0 - - run: ./.golib/scripts/repository-check.sh - - quality: - name: Quality / ${{ matrix.directory }} - needs: prepare - runs-on: ubuntu-24.04 - timeout-minutes: 360 - strategy: - fail-fast: false - max-parallel: 8 - matrix: - include: ${{ fromJSON(needs.prepare.outputs.matrix) }} - steps: - - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 - with: - fetch-depth: 0 - - uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6 - with: - go-version-file: .go-version - cache: false - - name: Set up pinned ripgrep - env: - RIPGREP_VERSION: 15.2.0 - RIPGREP_SHA256: 33e15bcf1624b25cdd2a55813a47a2f95dbe126268203e76aa6a585d1e7b149c - run: | - set -euo pipefail - archive="${RUNNER_TEMP}/ripgrep.tar.gz" - root="${RUNNER_TEMP}/ripgrep" - curl --fail --silent --show-error --location \ - --retry 5 --retry-delay 2 --retry-all-errors \ - "https://github.com/BurntSushi/ripgrep/releases/download/${RIPGREP_VERSION}/ripgrep-${RIPGREP_VERSION}-x86_64-unknown-linux-musl.tar.gz" \ - --output "${archive}" - printf '%s %s\n' "${RIPGREP_SHA256}" "${archive}" | sha256sum --check - - mkdir -p "${root}" - tar --extract --gzip --file "${archive}" --directory "${root}" \ - --strip-components 1 - echo "${root}" >>"${GITHUB_PATH}" - "${root}/rg" --version | grep -Eq "^ripgrep ${RIPGREP_VERSION}( |$)" - - name: Set up Node - if: github.repository == 'faustbrian/go-ecma-regexp' || github.repository == 'faustbrian/go-queue-control-plane' - uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6 - with: - node-version: '24.4.1' - package-manager-cache: false - - name: Set up Deno - if: github.repository == 'faustbrian/go-ecma-regexp' - uses: denoland/setup-deno@22d081ff2d3a40755e97629de92e3bcbfa7cf2ed # v2.0.5 - with: - deno-version: '2.9.4' - cache: false - - name: Set up CLI shell runtime - if: github.repository == 'faustbrian/go-cli' - env: - ZSH_DEB_SHA256: bd5cc8dd3a01a6db38c0a815d75202c356a9c7f378674ba7bed9bc86dcba8af0 - run: | - set -euo pipefail - archive="${RUNNER_TEMP}/zsh.deb" - root="${RUNNER_TEMP}/zsh-runtime" - curl --fail --silent --show-error --location \ - --retry 5 --retry-delay 2 --retry-all-errors \ - 'https://archive.ubuntu.com/ubuntu/pool/main/z/zsh/zsh_5.9-6ubuntu2_amd64.deb' \ - --output "${archive}" - printf '%s %s\n' "${ZSH_DEB_SHA256}" "${archive}" | sha256sum --check - - mkdir -p "${root}" - dpkg-deb --extract "${archive}" "${root}" - echo "${root}/bin" >>"${GITHUB_PATH}" - "${root}/bin/zsh" --version | grep -Eq '^zsh 5\.9 ' - - name: Set up unpublished module proxy - if: vars.GOLIB_BOOTSTRAP_PROXY_URL != '' - env: - PROXY_URL: ${{ vars.GOLIB_BOOTSTRAP_PROXY_URL }} - PROXY_SHA256: ${{ vars.GOLIB_BOOTSTRAP_PROXY_SHA256 }} - run: | - set -euo pipefail - [[ "${PROXY_SHA256}" =~ ^[0-9a-f]{64}$ ]] - archive="${RUNNER_TEMP}/golib-proxy.tar.gz" - proxy="${RUNNER_TEMP}/golib-proxy" - curl --fail --silent --show-error --location \ - --retry 5 --retry-delay 2 --retry-all-errors \ - "${PROXY_URL}" --output "${archive}" - printf '%s %s\n' "${PROXY_SHA256}" "${archive}" | sha256sum --check - - mkdir -p "${proxy}" - tar --extract --gzip --file "${archive}" --directory "${proxy}" - echo "GOPROXY=file://${proxy},https://proxy.golang.org,direct" >>"${GITHUB_ENV}" - echo 'GONOSUMDB=github.com/faustbrian/go-*' >>"${GITHUB_ENV}" - - name: Restore content-addressed mutation evidence - if: inputs.release_dry_run != true - env: - GH_TOKEN: ${{ github.token }} - GITHUB_REPOSITORY_ID: ${{ github.repository_id }} - run: | - set -euo pipefail - seed='.golib/mutation-bootstrap/${{ matrix.artifact }}.zip' - if [[ -s "${seed}" ]]; then - ./.golib/scripts/restore-ci-mutation-evidence.sh \ - '${{ matrix.directory }}' "${seed}" - else - ./.golib/scripts/restore-ci-mutation-evidence.sh \ - '${{ matrix.directory }}' - fi - - name: Run strict module contract - id: strict_contract - if: inputs.release_dry_run != true - run: ./.golib/scripts/with-disposable-go-cache.sh ./.golib/scripts/run-modules.sh check --modules '${{ matrix.directory }}' - - name: Run release dry-run - id: release_dry_run - if: inputs.release_dry_run == true - env: - GOLIB_VERIFICATION_SNAPSHOT: '1' - run: | - set -euo pipefail - output='.artifacts/release-dry-run.log' - if [[ '${{ matrix.directory }}' != '.' ]]; then - output='.artifacts/${{ matrix.directory }}/release-dry-run.log' - fi - mkdir -p "$(dirname "${output}")" - ./.golib/scripts/with-disposable-go-cache.sh \ - ./.golib/scripts/run-modules.sh release-dry-run \ - --modules '${{ matrix.directory }}' 2>&1 | tee "${output}" - - name: Stage attributable evidence - if: always() - env: - CONTRACT_OUTCOME: ${{ inputs.release_dry_run == true && steps.release_dry_run.outcome || steps.strict_contract.outcome }} - run: >- - ./.golib/scripts/stage-ci-evidence.sh '${{ matrix.directory }}' - '${{ format('{0}/golib-evidence-{1}', runner.temp, matrix.artifact) }}' - "${CONTRACT_OUTCOME}" - - name: Upload attributable evidence - if: always() - uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 - with: - name: evidence-${{ matrix.artifact }} - path: ${{ format('{0}/golib-evidence-{1}', runner.temp, matrix.artifact) }} - if-no-files-found: error - include-hidden-files: true - retention-days: 30 - - codeql: - name: CodeQL - runs-on: ubuntu-24.04 - timeout-minutes: 120 - permissions: - contents: read - packages: read - security-events: write - steps: - - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 - with: - fetch-depth: 0 - - uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6 - with: - go-version-file: .go-version - cache: false - - name: Set up unpublished module proxy - if: vars.GOLIB_BOOTSTRAP_PROXY_URL != '' - env: - PROXY_URL: ${{ vars.GOLIB_BOOTSTRAP_PROXY_URL }} - PROXY_SHA256: ${{ vars.GOLIB_BOOTSTRAP_PROXY_SHA256 }} - run: | - set -euo pipefail - [[ "${PROXY_SHA256}" =~ ^[0-9a-f]{64}$ ]] - archive="${RUNNER_TEMP}/golib-proxy.tar.gz" - proxy="${RUNNER_TEMP}/golib-proxy" - curl --fail --silent --show-error --location \ - --retry 5 --retry-delay 2 --retry-all-errors \ - "${PROXY_URL}" --output "${archive}" - printf '%s %s\n' "${PROXY_SHA256}" "${archive}" | sha256sum --check - - mkdir -p "${proxy}" - tar --extract --gzip --file "${archive}" --directory "${proxy}" - echo "GOPROXY=file://${proxy},https://proxy.golang.org,direct" >>"${GITHUB_ENV}" - echo 'GONOSUMDB=github.com/faustbrian/go-*' >>"${GITHUB_ENV}" - - uses: github/codeql-action/init@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4 - with: - languages: go - queries: security-extended,security-and-quality - - run: ./.golib/scripts/with-disposable-go-cache.sh ./.golib/scripts/codeql-build.sh - - uses: github/codeql-action/analyze@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4 + ci: + uses: faustbrian/go-library-tools/.github/workflows/library-ci.yml@6028deef86539aa26fcd776e932f9f50ab4d8ad9 # v1.0.4 + with: + tooling_sha: 6028deef86539aa26fcd776e932f9f50ab4d8ad9 + release_dry_run: ${{ inputs.release_dry_run || false }} required: name: Required if: always() - needs: [prepare, repository-contract, quality, codeql] + needs: ci runs-on: ubuntu-24.04 timeout-minutes: 5 steps: - - name: Require every repository result + - name: Require shared workflow env: - PREPARE_RESULT: ${{ needs.prepare.result }} - CONTRACT_RESULT: ${{ needs.repository-contract.result }} - QUALITY_RESULT: ${{ needs.quality.result }} - CODEQL_RESULT: ${{ needs.codeql.result }} - run: | - set -euo pipefail - for result in \ - "${PREPARE_RESULT}" \ - "${CONTRACT_RESULT}" \ - "${QUALITY_RESULT}" \ - "${CODEQL_RESULT}"; do - [[ "${result}" == success ]] - done + CI_RESULT: ${{ needs.ci.result }} + run: test "${CI_RESULT}" = success diff --git a/.golib.yaml b/.golib.yaml new file mode 100644 index 0000000..013585d --- /dev/null +++ b/.golib.yaml @@ -0,0 +1,65 @@ +schema_version: 1 +tool_version: v1.0.4 +tool_checksums_sha256: 6c1ee70e3fa7aa333d12420dc78d7740f4de78717d1979eeb8a5f70a2a124ba0 +manifest: + modules: modules.json + packages: packages.json +evidence: + root: .verification +mutation: + root: .verification/mutation + imports: + - module: . + archive: .verification/mutation/bootstrap/root.zip + ledger: .verification/mutation/migration-ledger.json +api: + baselines: + - module: . + mode: apidiff + path: api/baseline.txt +operations: +- module: . + gate: test + steps: + - type: go-test + packages: ["./..."] + run: '^(TestContentionElectsExactlyOneOwner|TestAcquirePropagatesBackendAndCancellation|TestHandleLifecycleFailsClosed|TestHandleRejectsConcurrentOperationsWithoutBlockingState|TestHandleFailsClosedWhenDeadlinePassesDuringOperation|TestManagedRenewalReportsUncertaintyAndStopsAdmission|TestManagedStopHonorsCallerDeadline|TestStaleOwnerCannotAffectSuccessor|TestAcquireRacingShutdownReleasesReservation|TestManagerBoundsHandlesAndReleasesOnShutdown|TestWorkerCancelsOwnershipSensitiveJobOnLoss|TestCoordinatorCancelsOwnershipSensitiveTaskOnLoss)$' + count: 25 + timeout: 20m + - type: make + makefile: verification/package.mk + target: fence-mutations + timeout: 20m +- module: . + gate: docs + steps: + - type: make + makefile: verification/package.mk + target: docs + timeout: 20m +- module: . + gate: fuzz + steps: + - type: go-test + packages: ["."] + fuzz: '^FuzzKeyParsing$' + budget: 10000x + timeout: 2m + - type: go-test + packages: ["."] + fuzz: '^FuzzPolicyBounds$' + budget: 10000x + timeout: 2m + - type: go-test + packages: ["./memory"] + fuzz: '^FuzzLeaseStateModel$' + budget: 10000x + timeout: 2m +- module: . + gate: benchmark + steps: + - type: go-test + packages: ["./..."] + benchmark: "." + budget: 100ms + timeout: 20m diff --git a/.golib/documentation-tools.env b/.golib/documentation-tools.env deleted file mode 100644 index f58285a..0000000 --- a/.golib/documentation-tools.env +++ /dev/null @@ -1,6 +0,0 @@ -CSPELL_VERSION=10.0.0 -LYCHEE_VERSION=0.24.2 -LYCHEE_SHA256_DARWIN_ARM64=c9d3740ea2d891854d37116c9fba840f37b6e7c89d330e7db84ac333631c4977 -LYCHEE_SHA256_DARWIN_AMD64=887503a9cff667d322b8d0892b40bf49976eb9507af8483220a3706cdad55978 -LYCHEE_SHA256_LINUX_ARM64=91a7bd65685da41b90ccb9bc867a3d649a7818042dae04ff405e55a25bddee4c -LYCHEE_SHA256_LINUX_AMD64=1f4e0ef7f6554a6ed33dd7ac144fb2e1bbed98598e7af973042fc5cd43951c9a diff --git a/.golib/package.mk b/.golib/package.mk deleted file mode 100644 index 0bbb36e..0000000 --- a/.golib/package.mk +++ /dev/null @@ -1,75 +0,0 @@ -GO ?= go -GOLANGCI_LINT ?= golangci-lint -ACTIONLINT ?= $(GO) run github.com/rhysd/actionlint/cmd/actionlint@v1.7.12 -FUZZ_TIME ?= 2s -BENCH_TIME ?= 100ms -GREMLINS_VERSION ?= v0.6.0 -NILAWAY_VERSION ?= v0.0.0-20260710181136-2378218750e4 -STATICCHECK_VERSION ?= v0.8.0-rc.1 - -.PHONY: api-compat backend-hardening benchmark check coverage docs format \ - format-check fuzz integration lint mutation nilaway race staticcheck stress \ - test vet vuln workflows - -format: - gofmt -w . - -format-check: - test -z "$$(gofmt -l .)" - -test: - $(GO) test ./... - -race: - $(GO) test -race ./... - -stress: - ./scripts/check-stress.sh - -coverage: - ./scripts/check-coverage.sh - -vet: - $(GO) vet ./... - -staticcheck: - $(GO) run honnef.co/go/tools/cmd/staticcheck@$(STATICCHECK_VERSION) ./... - -lint: - $(GOLANGCI_LINT) run --timeout=5m - -nilaway: - -$(GO) run go.uber.org/nilaway/cmd/nilaway@$(NILAWAY_VERSION) \ - -include-pkgs='github.com/faustbrian/go-lease' ./... - -fuzz: - ./scripts/check-fuzz.sh "$(FUZZ_TIME)" - -mutation: - $(GO) run github.com/go-gremlins/gremlins/cmd/gremlins@$(GREMLINS_VERSION) \ - unleash --workers 2 --timeout-coefficient 10 \ - --threshold-efficacy 100 --threshold-mcover 100 ./memory - ./scripts/check-fence-mutations.sh - -benchmark: - $(GO) test ./... -run '^$$' -bench . -benchmem -benchtime="$(BENCH_TIME)" - -integration: - $(GO) test -race -count=1 -timeout=15m ./postgres ./valkey - -backend-hardening: - ./scripts/check-backend-faults.sh - -docs: - ./scripts/check-docs.sh - -api-compat: - ./scripts/check-api-compat.sh - -vuln: - $(GO) run golang.org/x/vuln/cmd/govulncheck@v1.6.0 ./... - -workflows: - $(ACTIONLINT) .github/workflows/*.yml - -check: format-check vet test race stress coverage fuzz benchmark docs api-compat diff --git a/.golib/scripts/audit-goals.sh b/.golib/scripts/audit-goals.sh deleted file mode 100755 index 4f6b24e..0000000 --- a/.golib/scripts/audit-goals.sh +++ /dev/null @@ -1,109 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 1 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -module="$1" -artifact="${root}/.artifacts/${module}" -output="${artifact}/goal-traceability.json" -temporary="${output}.tmp.$$" -gate_records="$(mktemp "${TMPDIR:-/tmp}/golib-goal-gates.XXXXXX")" -cleanup() { - rm -f "${temporary}" "${gate_records}" -} -trap cleanup EXIT HUP INT TERM - -module_record="$( - jq -e -c --arg directory "${module}" \ - '.modules[] | select(.directory == $directory)' \ - "${root}/modules.json" -)" || { - printf 'module is absent from modules.json: %s\n' "${module}" >&2 - exit 2 -} -goal_count="$(jq '.goal_evidence | length' <<<"${module_record}")" -mkdir -p "${artifact}" - -if [[ "${goal_count}" -gt 0 ]]; then - while IFS= read -r gate; do - [[ -n "${gate}" ]] || continue - if ! jq -e --arg gate "${gate}" ' - [ - .goal_evidence[] - | select(.implementation_status != "future-not-started") - | .verification_gates[] - ] - | index($gate) != null - ' <<<"${module_record}" >/dev/null; then - continue - fi - "${root}/.golib/scripts/verify-gate-evidence.sh" "${module}" "${gate}" - input_digest="$( - "${root}/.golib/scripts/gate-input-digest.sh" "${gate}" "${module}" - )" - evidence="${artifact}/evidence/by-input/${gate}/${input_digest}.json" - if [[ ! -f "${evidence}" ]]; then - evidence="${artifact}/evidence/${gate}.json" - fi - jq -c '{ - gate, - result, - input_digest, - execution_revision, - completed_at, - revalidated_revision, - revalidated_at - }' "${evidence}" >>"${gate_records}" - done <"${root}/.golib/scripts/check-gates.txt" -fi - -verified_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" -revision="$(git rev-parse HEAD)" -manifest_sha256="$(shasum -a 256 "${root}/modules.json" | awk '{print $1}')" -jq -n \ - --argjson module "${module_record}" \ - --slurpfile gates "${gate_records}" \ - --arg verified_at "${verified_at}" \ - --arg revision "${revision}" \ - --arg manifest_sha256 "${manifest_sha256}" \ - '{ - schema_version: 1, - module: $module.directory, - module_path: $module.module_path, - requirements_manifest_sha256: $manifest_sha256, - implementation_status: $module.goal_status, - verification_status: ( - if ($module.goal_evidence | length) == 0 - then "not-applicable" - elif ([ - $module.goal_evidence[] - | select(.implementation_status != "future-not-started") - ] | length) == 0 - then "deferred" - else "verified" - end - ), - verified_revision: $revision, - verified_at: $verified_at, - goals: [ - $module.goal_evidence[] | - . + { - verification_status: ( - if .implementation_status == "future-not-started" - then "deferred" - else "verified" - end - ) - } - ], - gate_evidence: $gates - }' >"${temporary}" -mv "${temporary}" "${output}" -printf '[%s] goal traceability: %s (%s goals)\n' \ - "${module}" \ - "$(jq -r '.verification_status' "${output}")" \ - "${goal_count}" diff --git a/.golib/scripts/build-golib-gremlins.sh b/.golib/scripts/build-golib-gremlins.sh deleted file mode 100755 index 37179a3..0000000 --- a/.golib/scripts/build-golib-gremlins.sh +++ /dev/null @@ -1,69 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -root="$(git rev-parse --show-toplevel)" - -# Resolved from the verified repository root rather than the caller's cwd. -# shellcheck disable=SC1091 -source "${root}/.golib/versions.env" - -semantic_patch="${root}/.golib/scripts/patches/gremlins-run-all-mutants.patch" -coverage_patch="${root}/.golib/scripts/patches/gremlins-shared-coverage.patch" -diff_patch="${root}/.golib/scripts/patches/gremlins-module-relative-diff.patch" -verifier_identity="$("${root}/.golib/scripts/mutation-verifier-identity.sh")" -platform_identity="$(go env GOOS GOARCH | paste -sd- -)" -artifact="${root}/.artifacts/tooling/gremlins-${verifier_identity}-${platform_identity}" -binary="${artifact}/golib-gremlins" -lock="${artifact}.lock" - -if [[ -x "${binary}" ]]; then - printf '%s\n' "${binary}" - exit 0 -fi - -mkdir -p "$(dirname "${artifact}")" -attempts=0 -while ! mkdir "${lock}" 2>/dev/null; do - if [[ -x "${binary}" ]]; then - printf '%s\n' "${binary}" - exit 0 - fi - attempts=$((attempts + 1)) - if [[ "${attempts}" -ge 120 ]]; then - printf 'timed out waiting for golib-gremlins build lock: %s\n' \ - "${lock}" >&2 - exit 1 - fi - sleep 1 -done -temporary="$(mktemp -d "${TMPDIR:-/tmp}/golib-gremlins.XXXXXX")" -cleanup() { - rm -rf "${temporary}" "${lock}" -} -trap cleanup EXIT HUP INT TERM - -download="$(GOWORK=off go mod download -json \ - "github.com/go-gremlins/gremlins@${GREMLINS_VERSION}")" -source_directory="$(jq -er '.Dir' <<<"${download}")" -jq -e \ - --arg sum "${GREMLINS_SUM}" \ - --arg gomod_sum "${GREMLINS_GOMOD_SUM}" ' - .Sum == $sum and .GoModSum == $gomod_sum - ' <<<"${download}" >/dev/null - -cp -R "${source_directory}" "${temporary}/source" -chmod -R u+w "${temporary}/source" -patch --batch --forward -d "${temporary}/source" -p1 <"${semantic_patch}" >&2 -patch --batch --forward -d "${temporary}/source" -p1 <"${coverage_patch}" >&2 -patch --batch --forward -d "${temporary}/source" -p1 <"${diff_patch}" >&2 -( - cd "${temporary}/source" - GOWORK=off go build -trimpath -buildvcs=false \ - -o "${temporary}/golib-gremlins" ./cmd/gremlins -) - -rm -rf "${artifact}" -mkdir "${artifact}" -mv "${temporary}/golib-gremlins" "${binary}" -chmod 0755 "${binary}" -printf '%s\n' "${binary}" diff --git a/.golib/scripts/build-local-proxy.sh b/.golib/scripts/build-local-proxy.sh deleted file mode 100755 index 40a1b34..0000000 --- a/.golib/scripts/build-local-proxy.sh +++ /dev/null @@ -1,180 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -lt 1 || $# -gt 3 ]]; then - printf 'usage: %s [version] [module-directories]\n' \ - "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -output="$1" -version="${2:-v1.0.0}" -selected="${3:-}" - -if [[ ! "${version}" =~ ^v(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)$ ]]; then - printf 'local proxy version must be canonical semantic version: %s\n' \ - "${version}" >&2 - exit 2 -fi -if [[ -e "${output}" && ! -d "${output}" ]]; then - printf 'local proxy output is not a directory: %s\n' "${output}" >&2 - exit 2 -fi - -mkdir -p "${output}" -if find "${output}" -mindepth 1 -print -quit | grep -q .; then - printf 'local proxy output must be empty: %s\n' "${output}" >&2 - exit 2 -fi - -temporary="$(mktemp -d "${TMPDIR:-/tmp}/golib-local-proxy.XXXXXX")" -mkdir -p "${temporary}/proxy" -cleanup() { - rm -rf "${temporary}" -} -trap cleanup EXIT HUP INT TERM - -rewrite_owned_dependencies() { - GOLIB_PROXY_VERSION="${version}" perl -pi -e \ - 's#(github\.com/faustbrian/go-[a-z0-9/-]+) v0\.0\.0(?:-[0-9]{14}-[0-9a-f]{12})?#$1 $ENV{GOLIB_PROXY_VERSION}#g' \ - "$1" -} - -selection_file="${temporary}/selected-modules" -jq -r --arg selected "${selected}" ' - . as $catalog - | def closure($directories): - ([ - $catalog.modules[] - | select(.directory as $directory | $directories | index($directory)) - | .owned_dependencies[] - ] | unique) as $dependencies - | ([ - $catalog.modules[] - | select(.module_path as $path | $dependencies | index($path)) - | .directory - ] + $directories | unique) as $next - | if $next == $directories then $next else closure($next) end; - ( - if $selected == "" or ($selected | split(",") | index(".") != null) - then [.modules[] | select(.releasable == true) | .directory] - else - ($selected | split(",")) as $selected_directories - | ([ - .modules[] - | select( - .directory as $directory - | any( - $selected_directories[]; - . as $selected_directory - | $directory == $selected_directory or - ($directory | startswith($selected_directory + "/")) - ) - ) - | .directory - ]) as $expanded - | closure($expanded) - end - ) as $directories - | .modules[] - | select( - .releasable == true - and (.directory as $directory | $directories | index($directory)) - ) - | [.module_path, .directory] - | @tsv -' "${root}/modules.json" >"${selection_file}" -if [[ ! -s "${selection_file}" ]]; then - if [[ -n "${selected}" ]] && jq -e --arg selected "${selected}" ' - . as $catalog - | ($selected | split(",")) as $directories - | all( - $directories[]; - . as $directory - | any($catalog.modules[]; .directory == $directory) - ) - ' "${root}/modules.json" >/dev/null; then - exit 0 - fi - printf 'local proxy module selection is empty: %s\n' "${selected}" >&2 - exit 1 -fi - -while IFS=$'\t' read -r module_path module_directory; do - [[ -n "${module_path}" && -n "${module_directory}" ]] || continue - if [[ "${module_path}" =~ [A-Z] ]]; then - printf 'local proxy does not support unescaped uppercase module paths: %s\n' \ - "${module_path}" >&2 - exit 1 - fi - - proxy_directory="${temporary}/proxy/${module_path}/@v" - archive_directory="${temporary}/archive" - archive_root="${archive_directory}/${module_path}@${version}" - zip_file="${proxy_directory}/${version}.zip" - nested_modules="${temporary}/nested-modules" - archive_files="${temporary}/archive-files" - archive_tar="${temporary}/archive.tar" - mkdir -p "${proxy_directory}" "${archive_root}" - jq -r --arg current "${module_directory}" ' - .modules[] - | .directory - | select( - ($current == "." and . != ".") or - ($current != "." and startswith($current + "/")) - ) - ' "${root}/modules.json" >"${nested_modules}" - - cp "${root}/${module_directory}/go.mod" \ - "${proxy_directory}/${version}.mod" - rewrite_owned_dependencies "${proxy_directory}/${version}.mod" - printf '{"Version":"%s","Time":"2000-01-01T00:00:00Z"}\n' \ - "${version}" >"${proxy_directory}/${version}.info" - printf '%s\n' "${version}" >"${proxy_directory}/list" - - : >"${archive_files}" - while IFS= read -r -d '' source; do - if [[ ! -e "${root}/${source}" ]]; then - continue - fi - relative="${source#"${module_directory}/"}" - if [[ "${relative}" == ".golib" || "${relative}" == ".golib/"* ]]; then - continue - fi - nested=0 - while IFS= read -r nested_module; do - [[ -n "${nested_module}" ]] || continue - if [[ "${source}" == "${nested_module}" || - "${source}" == "${nested_module}/"* ]]; then - nested=1 - break - fi - done <"${nested_modules}" - [[ "${nested}" -eq 0 ]] || continue - if [[ -L "${root}/${source}" ]]; then - continue - fi - printf '%s\0' "${relative}" >>"${archive_files}" - done < <( - git -C "${root}" ls-files -z --cached -- "${module_directory}" - ) - ( - cd "${root}/${module_directory}" - tar --null -cf "${archive_tar}" -T "${archive_files}" - ) - tar -xf "${archive_tar}" -C "${archive_root}" - rm -f "${archive_tar}" - rewrite_owned_dependencies "${archive_root}/go.mod" - - find "${archive_directory}" -exec touch -t 200001010000 {} + - ( - cd "${archive_directory}" - LC_ALL=C find "${module_path}@${version}" -type f -print | - LC_ALL=C sort | - zip -q -X "${zip_file}" -@ - ) - rm -rf "${archive_directory}" -done <"${selection_file}" - -cp -R "${temporary}/proxy/." "${output}/" diff --git a/.golib/scripts/check-api-baseline.sh b/.golib/scripts/check-api-baseline.sh deleted file mode 100755 index 9eccb14..0000000 --- a/.golib/scripts/check-api-baseline.sh +++ /dev/null @@ -1,66 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 1 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -# shellcheck disable=SC1091 # Canonical repository tool versions. -source "${root}/.golib/versions.env" -case "$1" in - /*) directory="$1" ;; - *) directory="${root}/$1" ;; -esac -if [[ ! -f "${directory}/go.mod" ]]; then - printf 'module has no go.mod: %s\n' "$1" >&2 - exit 1 -fi - -baseline="${directory}/api/baseline.txt" -if [[ ! -s "${baseline}" ]]; then - printf 'missing API baseline: %s\n' "${baseline}" >&2 - exit 1 -fi -module_path="$( - cd "${directory}" - GOWORK=off go mod edit -json | jq -er '.Module.Path' -)" -current="$(mktemp "${TMPDIR:-/tmp}/golib-api-current.XXXXXX")" -report="$(mktemp "${TMPDIR:-/tmp}/golib-api-report.XXXXXX")" -cleanup() { - rm -f "${current}" "${report}" -} -trap cleanup EXIT HUP INT TERM - -run_apidiff() { - if [[ -n "${GOLIB_APIDIFF:-}" ]]; then - GOWORK=off go exec-tool "${GOLIB_APIDIFF}" "$@" - else - GOWORK=off go run \ - "golang.org/x/exp/cmd/apidiff@${APIDIFF_VERSION}" "$@" - fi -} - -( - cd "${directory}" - run_apidiff -m -w "${current}" "${module_path}" -) -set +e -( - cd "${directory}" - run_apidiff -m -incompatible "${baseline}" "${current}" >"${report}" -) -status=$? -set -e -if [[ -s "${report}" ]]; then - printf 'incompatible exported API changes in %s:\n' \ - "${module_path}" >&2 - cat "${report}" >&2 - exit 1 -fi -if [[ "${status}" -ne 0 ]]; then - printf 'API compatibility tool failed for %s\n' "${module_path}" >&2 - exit "${status}" -fi diff --git a/.golib/scripts/check-coverage.sh b/.golib/scripts/check-coverage.sh deleted file mode 100755 index 02ed3d2..0000000 --- a/.golib/scripts/check-coverage.sh +++ /dev/null @@ -1,133 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 1 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -module="$1" -directory="${root}/${module}" -artifact="${root}/.artifacts/${module}" -profile="${artifact}/coverage.out" -packages="${artifact}/coverage-packages.txt" -report="${artifact}/coverage-report.txt" -identity="${artifact}/coverage-profile.json" -mkdir -p "${artifact}" -rm -f "${identity}" - -cd "${directory}" -jq -r --arg directory "${module}" ' - .modules[] - | select(.directory == $directory) - | .packages[] - | select(.coverage_required == true) - | .import_path -' "${root}/modules.json" | sort -u >"${packages}" -[[ -s "${packages}" ]] || { - printf 'coverage expected packages are empty for %s\n' "${module}" >&2 - exit 1 -} - -tags="$(jq -r --arg directory "${module}" \ - '.modules[] | select(.directory == $directory) | .test_tags | join(",")' \ - "${root}/modules.json")" -test_arguments=( - ./... - -count=1 - -timeout=20m - -covermode=atomic - -coverpkg=./... - -coverprofile="${profile}" -) -if [[ -n "${tags}" ]]; then - test_arguments=(-tags="${tags}" "${test_arguments[@]}") -fi -if [[ "${module}" == "pkg/xsd" ]]; then - xsts_work="${artifact}/xsts" - mkdir -p "${xsts_work}" - XSTS_ROOT="$(./scripts/prepare-xsts.sh "${xsts_work}")" - export XSTS_ROOT -fi -input_digest="$( - "${root}/.golib/scripts/gate-input-digest.sh" coverage "${module}" -)" -started="$(date +%s)" -GOWORK=off go test "${test_arguments[@]}" -finished="$(date +%s)" -elapsed=$((finished - started)) -if [[ "${elapsed}" -lt 1 ]]; then - elapsed=1 -fi -[[ -s "${profile}" ]] || { - printf 'coverage profile is missing for %s\n' "${module}" >&2 - exit 1 -} - -set +e -awk ' - FNR == NR { expected[$1] = 1; next } - FNR == 1 { next } - { - split($1, location, ":") - package = location[1] - sub("/[^/]+$", "", package) - block = $1 - blockPackage[block] = package - blockStatements[block] = $(NF - 1) + 0 - if (($NF + 0) > 0) blockCovered[block] = 1 - } - END { - failed = 0 - for (block in blockStatements) { - package = blockPackage[block] - total[package] += blockStatements[block] - if (blockCovered[block]) covered[package] += blockStatements[block] - } - for (package in expected) { - if (!(package in total) || total[package] == 0) { - printf "%s missing executable coverage evidence\n", package - failed = 1 - continue - } - printf "%s %d/%d statements\n", package, covered[package], total[package] - if (covered[package] != total[package]) failed = 1 - } - exit failed - } -' "${packages}" "${profile}" | sort >"${report}" -coverage_status="${PIPESTATUS[0]}" -set -e - -cat "${report}" -if [[ "${coverage_status}" -eq 0 ]]; then - printf 'all production packages have exact 100%% statement coverage\n' -else - printf 'one or more production packages are below exact 100%% coverage\n' >&2 - exit 1 -fi - -profile_sha256="$(shasum -a 256 "${profile}" | awk '{print $1}')" -completed_input_digest="$( - "${root}/.golib/scripts/gate-input-digest.sh" coverage "${module}" -)" -if [[ "${completed_input_digest}" != "${input_digest}" ]]; then - printf 'coverage inputs changed during execution for %s\n' "${module}" >&2 - exit 1 -fi -identity_tmp="$(mktemp "${identity}.tmp.XXXXXX")" -jq -n \ - --arg input_digest "${input_digest}" \ - --arg test_tags "${tags}" \ - --arg profile_sha256 "${profile_sha256}" \ - --arg elapsed "${elapsed}s" ' - { - schema_version: 1, - input_digest: $input_digest, - test_tags: $test_tags, - profile_sha256: $profile_sha256, - elapsed: $elapsed - } - ' >"${identity_tmp}" -mv "${identity_tmp}" "${identity}" diff --git a/.golib/scripts/check-documentation.sh b/.golib/scripts/check-documentation.sh deleted file mode 100755 index 4eb2e05..0000000 --- a/.golib/scripts/check-documentation.sh +++ /dev/null @@ -1,107 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -root="$(git rev-parse --show-toplevel)" -# shellcheck source=/dev/null -source "${root}/.golib/documentation-tools.env" - -tool_directory="$(mktemp -d "${TMPDIR:-/tmp}/golib-documentation-tools.XXXXXX")" - -cleanup() { - chmod -R u+w "${tool_directory}" 2>/dev/null || true - find "${tool_directory}" -depth -delete 2>/dev/null || true -} -trap cleanup EXIT HUP INT TERM - -cd "${root}" -test -s README.md - -configured_cspell_version="$(jq -r '.devDependencies.cspell // empty' package.json)" -if [[ "${configured_cspell_version}" != "${CSPELL_VERSION}" ]]; then - printf 'cspell version mismatch: package.json=%s versions.env=%s\n' \ - "${configured_cspell_version}" "${CSPELL_VERSION}" >&2 - exit 1 -fi - -cspell="${GOLIB_CSPELL:-}" -if [[ -z "${cspell}" ]]; then - command -v npm >/dev/null || { - printf 'npm is required for the documentation spelling gate\n' >&2 - exit 1 - } - node_directory="${tool_directory}/node" - npm_cache="${tool_directory}/npm-cache" - mkdir -p "${node_directory}" "${npm_cache}" - cp package.json package-lock.json "${node_directory}/" - ( - cd "${node_directory}" - NPM_CONFIG_CACHE="${npm_cache}" npm ci \ - --ignore-scripts \ - --no-audit \ - --no-fund \ - --silent - ) - cspell="${node_directory}/node_modules/.bin/cspell" -fi -[[ -x "${cspell}" ]] || { - printf 'cspell executable is unavailable: %s\n' "${cspell}" >&2 - exit 1 -} -"${cspell}" lint \ - --config cspell.json \ - --no-config-search \ - --validate-directives \ - --no-progress \ - --no-summary \ - README.md 'docs/**/*.md' - -lychee="${GOLIB_LYCHEE:-}" -if [[ -z "${lychee}" ]]; then - case "$(uname -s)/$(uname -m)" in - Darwin/arm64) - target="aarch64-apple-darwin" - checksum="${LYCHEE_SHA256_DARWIN_ARM64}" - ;; - Darwin/x86_64) - target="x86_64-apple-darwin" - checksum="${LYCHEE_SHA256_DARWIN_AMD64}" - ;; - Linux/aarch64|Linux/arm64) - target="aarch64-unknown-linux-gnu" - checksum="${LYCHEE_SHA256_LINUX_ARM64}" - ;; - Linux/x86_64) - target="x86_64-unknown-linux-gnu" - checksum="${LYCHEE_SHA256_LINUX_AMD64}" - ;; - *) - printf 'unsupported lychee platform: %s/%s\n' "$(uname -s)" "$(uname -m)" >&2 - exit 1 - ;; - esac - archive="lychee-${target}.tar.gz" - archive_path="${tool_directory}/${archive}" - curl --fail --silent --show-error --location \ - "https://github.com/lycheeverse/lychee/releases/download/lychee-v${LYCHEE_VERSION}/${archive}" \ - --output "${archive_path}" - printf '%s %s\n' "${checksum}" "${archive_path}" | shasum -a 256 -c - - tar -xzf "${archive_path}" -C "${tool_directory}" - lychee="${tool_directory}/lychee-${target}/lychee" -fi -[[ -x "${lychee}" ]] || { - printf 'lychee executable is unavailable: %s\n' "${lychee}" >&2 - exit 1 -} -# golib-unpublished-pkg-go-dev: public-proxy checks own publication readiness. -"${lychee}" \ - --cache=false \ - --exclude '^https://pkg\.go\.dev/(badge/)?github\.com/faustbrian/go-' \ - --exclude '^https://doi\.org/10\.1145/190314\.190317$' \ - --exclude '^https://service\.unece\.org/trade/' \ - --exclude-private \ - --exclude-loopback \ - --exclude '^https://www\.iso\.org/standard/' \ - --max-concurrency 16 \ - --max-retries 3 \ - --no-progress \ - README.md 'docs/**/*.md' diff --git a/.golib/scripts/check-fuzz.sh b/.golib/scripts/check-fuzz.sh deleted file mode 100755 index 6d31e48..0000000 --- a/.golib/scripts/check-fuzz.sh +++ /dev/null @@ -1,61 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 1 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -module="$1" -fuzz_budget="${GOLIB_FUZZ_SMOKE_BUDGET:-10000x}" -if [[ "${module}" = /* ]]; then - directory="${module}" -else - directory="${root}/${module}" -fi -if [[ ! -f "${directory}/go.mod" ]]; then - printf 'module has no go.mod: %s\n' "${module}" >&2 - exit 1 -fi -targets="$(mktemp)" -trap 'rm -f "${targets}"' EXIT - -cd "${directory}" -while IFS= read -r -d '' file; do - parent="$(dirname "${file#./}")" - nested=0 - while [[ "${parent}" != "." ]]; do - if [[ -f "${parent}/go.mod" ]]; then - nested=1 - break - fi - parent="$(dirname "${parent}")" - done - [[ "${nested}" -eq 0 ]] || continue - names="$( - sed -nE \ - 's/^func (Fuzz[A-Za-z0-9_]+)\([A-Za-z_][A-Za-z0-9_]* \*testing\.F\).*/\1/p' \ - "${file}" - )" - while IFS= read -r target; do - [[ -n "${target}" ]] || continue - printf '%s %s\n' "${file}" "${target}" >>"${targets}" - done <<<"${names}" -done < <(find . -type f -name '*_test.go' -not -path './vendor/*' -print0) -sort -u -o "${targets}" "${targets}" - -count=0 -while read -r file target; do - [[ -n "${file:-}" ]] || continue - package="./$(dirname "${file#./}")" - [[ "${package}" != "./." ]] || package=. - GOWORK=off go test "${package}" -run '^$' -fuzz "^${target}$" \ - -fuzztime="${fuzz_budget}" -parallel=2 - count=$((count + 1)) -done <"${targets}" -if [[ "${count}" -eq 0 ]]; then - printf 'no fuzz targets were executed for %s\n' "${module}" >&2 - exit 1 -fi -printf 'executed %s registered fuzz targets\n' "${count}" diff --git a/.golib/scripts/check-gates.txt b/.golib/scripts/check-gates.txt deleted file mode 100644 index b4c532d..0000000 --- a/.golib/scripts/check-gates.txt +++ /dev/null @@ -1,21 +0,0 @@ -format-check -tidy-check -safety -vet -test -race -coverage -lint -staticcheck -vulnerability -secrets -licenses -sbom -fuzz -mutation -nilaway -docs -api -conformance -interoperability -benchmark diff --git a/.golib/scripts/check-go-safety.go b/.golib/scripts/check-go-safety.go deleted file mode 100644 index a41bec3..0000000 --- a/.golib/scripts/check-go-safety.go +++ /dev/null @@ -1,87 +0,0 @@ -package main - -import ( - "fmt" - "go/parser" - "go/token" - "io/fs" - "os" - "path/filepath" - "sort" - "strconv" - "strings" -) - -func main() { - if len(os.Args) != 2 { - fmt.Fprintln(os.Stderr, "usage: check-go-safety ") - os.Exit(2) - } - violations, err := scan(os.Args[1]) - if err != nil { - fmt.Fprintln(os.Stderr, err) - os.Exit(1) - } - for _, violation := range violations { - fmt.Fprintln(os.Stderr, violation) - } - if len(violations) != 0 { - os.Exit(1) - } -} - -func scan(directory string) ([]string, error) { - violations := make([]string, 0) - err := filepath.WalkDir(directory, func(path string, entry fs.DirEntry, walkErr error) error { - if walkErr != nil { - return walkErr - } - if entry.IsDir() { - if path != directory && excludedDirectory(entry.Name()) { - return filepath.SkipDir - } - return nil - } - if !strings.HasSuffix(path, ".go") || strings.HasSuffix(path, "_test.go") { - return nil - } - - fileSet := token.NewFileSet() - file, err := parser.ParseFile(fileSet, path, nil, parser.ParseComments) - if err != nil { - return fmt.Errorf("parse %s: %w", path, err) - } - for _, imported := range file.Imports { - name, err := strconv.Unquote(imported.Path.Value) - if err != nil { - return fmt.Errorf("parse import in %s: %w", path, err) - } - if name == "unsafe" || name == "C" { - violations = append(violations, fmt.Sprintf( - "%s:%d: forbidden import %q", - path, - fileSet.Position(imported.Pos()).Line, - name, - )) - } - } - for _, group := range file.Comments { - for _, comment := range group.List { - if strings.HasPrefix(strings.TrimSpace(comment.Text), "//go:linkname") { - violations = append(violations, fmt.Sprintf( - "%s:%d: forbidden go:linkname directive", - path, - fileSet.Position(comment.Pos()).Line, - )) - } - } - } - return nil - }) - sort.Strings(violations) - return violations, err -} - -func excludedDirectory(name string) bool { - return name == "vendor" || name == "testdata" || strings.HasPrefix(name, ".") -} diff --git a/.golib/scripts/check-go-safety.sh b/.golib/scripts/check-go-safety.sh deleted file mode 100755 index 840e13b..0000000 --- a/.golib/scripts/check-go-safety.sh +++ /dev/null @@ -1,14 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 1 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -module="$1" -directory="${root}/${module}" - -go run "${root}/.golib/scripts/check-go-safety.go" "${directory}" -printf 'standalone safety policy passed for %s\n' "${module}" diff --git a/.golib/scripts/check-module.sh b/.golib/scripts/check-module.sh deleted file mode 100755 index 2c521e3..0000000 --- a/.golib/scripts/check-module.sh +++ /dev/null @@ -1,516 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 2 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -module="$1" -gate="$2" -directory="${root}/${module}" -local_proxy_owned=0 -local_modcache_owned=0 -isolated_modfiles_owned=0 -GOLIB_GO_TOOL_PATH="" - -cleanup() { - if [[ "${local_proxy_owned}" -eq 1 ]]; then - rm -rf "${GOLIB_LOCAL_PROXY}" - fi - if [[ "${local_modcache_owned}" -eq 1 ]]; then - chmod -R u+w "${GOLIB_LOCAL_MODCACHE}" - rm -rf "${GOLIB_LOCAL_MODCACHE}" - fi - if [[ "${isolated_modfiles_owned}" -eq 1 ]]; then - rm -rf "${GOLIB_ISOLATED_MODFILES_DIRECTORY}" - fi -} -trap cleanup EXIT HUP INT TERM - -if ! jq -e --arg directory "${module}" \ - '.modules[] | select(.directory == $directory)' \ - "${root}/modules.json" >/dev/null; then - printf 'module is absent from modules.json: %s\n' "${module}" >&2 - exit 1 -fi - -if [[ ! -f "${directory}/go.mod" ]]; then - printf 'module has no go.mod: %s\n' "${module}" >&2 - exit 1 -fi - -enable_local_proxy() { - local no_sum_db upstream upstream_flags - - if [[ -z "${GOLIB_REAL_GO:-}" ]]; then - GOLIB_REAL_GO="$(command -v go)" - fi - export GOLIB_REAL_GO - upstream_flags="$( - printf '%s' "${GOLIB_UPSTREAM_GOFLAGS:-$("${GOLIB_REAL_GO}" env GOFLAGS)}" - )" - export GOLIB_UPSTREAM_GOFLAGS="${upstream_flags}" - - if [[ -z "${GOLIB_LOCAL_PROXY:-}" ]]; then - GOLIB_LOCAL_PROXY="$(mktemp -d "${TMPDIR:-/tmp}/golib-proxy.XXXXXX")" - local_proxy_owned=1 - "${root}/.golib/scripts/build-local-proxy.sh" \ - "${GOLIB_LOCAL_PROXY}" v1.0.0 "${module}" - fi - export GOLIB_LOCAL_PROXY - - upstream="${GOLIB_UPSTREAM_GOPROXY:-$(go env GOPROXY)}" - no_sum_db="$(go env GONOSUMDB)" - export GOPROXY="file://${GOLIB_LOCAL_PROXY},${upstream}" - export GONOSUMDB="github.com/faustbrian/go-*${no_sum_db:+,${no_sum_db}}" - if [[ -z "${GOLIB_LOCAL_MODCACHE:-}" ]]; then - GOLIB_LOCAL_MODCACHE="$( - mktemp -d "${TMPDIR:-/tmp}/golib-modcache.XXXXXX" - )" - local_modcache_owned=1 - fi - export GOLIB_LOCAL_MODCACHE - export GOMODCACHE="${GOLIB_LOCAL_MODCACHE}" - if [[ -z "${GOLIB_ISOLATED_MODFILES_DIRECTORY:-}" ]]; then - GOLIB_ISOLATED_MODFILES_DIRECTORY="$( - mktemp -d "${TMPDIR:-/tmp}/golib-modfiles.XXXXXX" - )" - isolated_modfiles_owned=1 - fi - export GOLIB_ISOLATED_MODFILES_DIRECTORY - mkdir -p "${GOLIB_ISOLATED_MODFILES_DIRECTORY}/bin" - ln -sf "${root}/.golib/scripts/internal/isolated-go.sh" \ - "${GOLIB_ISOLATED_MODFILES_DIRECTORY}/bin/go" - case ":${PATH}:" in - *":${GOLIB_ISOLATED_MODFILES_DIRECTORY}/bin:"*) ;; - *) - PATH="${GOLIB_ISOLATED_MODFILES_DIRECTORY}/bin:${PATH}" - export PATH - ;; - esac - export GOFLAGS="${upstream_flags}" -} - -isolated() { - enable_local_proxy - GOWORK=off "$@" -} - -install_go_tool() { - local package="$1" - local executable="$2" - local tool_directory - - enable_local_proxy - tool_directory="${GOLIB_ISOLATED_MODFILES_DIRECTORY}/tools" - mkdir -p "${tool_directory}" - GOBIN="${tool_directory}" GOWORK=off \ - GOFLAGS="${GOLIB_UPSTREAM_GOFLAGS}" \ - "${GOLIB_REAL_GO}" install "${package}" - GOLIB_GO_TOOL_PATH="${tool_directory}/${executable}" -} - -run_go_tool() { - local package="$1" - local executable="$2" - shift 2 - - install_go_tool "${package}" "${executable}" - isolated go exec-tool "${GOLIB_GO_TOOL_PATH}" "$@" -} - -refresh_owned_sums() { - [[ -f go.sum ]] || return 0 - local temporary - temporary="$(mktemp "${TMPDIR:-/tmp}/golib-go-sum.XXXXXX")" - awk '$1 !~ /^github\.com\/faustbrian\/golib\// { print }' \ - go.sum >"${temporary}" - if cmp -s go.sum "${temporary}"; then - rm -f "${temporary}" - else - mv "${temporary}" go.sum - fi -} - -applicable() { - jq -e --arg directory "${module}" --arg gate "$1" \ - '.modules[] | select(.directory == $directory) | .gates[$gate] == true' \ - "${root}/modules.json" >/dev/null -} - -package_makefile="Makefile" -if [[ "${module}" == "." && -f "${root}/.golib/package.mk" ]]; then - package_makefile="${root}/.golib/package.mk" -fi - -package_make() { - make -f "${package_makefile}" "$@" -} - -make_has_target() { - [[ -f "${package_makefile}" ]] && - grep -Eq "^$1([[:space:]]+[^:]*)?:" "${package_makefile}" -} - -find_make_target() { - local target - for target in "$@"; do - if make_has_target "${target}"; then - printf '%s\n' "${target}" - return 0 - fi - done - return 1 -} - -skip_not_applicable() { - printf '[%s] %s: not applicable by catalog policy\n' "${module}" "$1" -} - -test_tags() { - jq -r --arg directory "${module}" \ - '.modules[] | select(.directory == $directory) | .test_tags | join(",")' \ - "${root}/modules.json" -} - -interoperability_declared() { - jq -e --arg directory "${module}" \ - '.modules[] | select(.directory == $directory) | - .interoperability_tools | length > 0' \ - "${root}/modules.json" >/dev/null -} - -run_benchmark() { - local output temporary status target - enable_local_proxy - output="${root}/.artifacts/${module}/benchmark.txt" - temporary="${output}.tmp.$$" - mkdir -p "$(dirname "${output}")" - rm -f "${output}" "${temporary}" - - set +e - if target="$(find_make_target benchmark performance)"; then - package_make GOWORK=off "${target}" 2>&1 | tee "${temporary}" - status=${PIPESTATUS[0]} - else - GOWORK=off go test ./... -run '^$' -bench . -benchmem 2>&1 | - tee "${temporary}" - status=${PIPESTATUS[0]} - fi - set -e - - if [[ "${status}" -ne 0 ]]; then - rm -f "${temporary}" - return "${status}" - fi - if ! grep -Eq '^Benchmark[^[:space:]]*(-[0-9]+)?[[:space:]]+' \ - "${temporary}"; then - printf '[%s] benchmark gate produced no Go benchmark results\n' \ - "${module}" >&2 - rm -f "${temporary}" - return 1 - fi - mv "${temporary}" "${output}" -} - -run_make_evidence() { - local selected="$1" - local target="$2" - local output temporary status - output="${root}/.artifacts/${module}/${selected}.txt" - temporary="${output}.tmp.$$" - mkdir -p "$(dirname "${output}")" - rm -f "${output}" "${temporary}" - - set +e - package_make "${target}" 2>&1 | tee "${temporary}" - status=${PIPESTATUS[0]} - set -e - - if [[ "${status}" -ne 0 ]]; then - rm -f "${temporary}" - return "${status}" - fi - if [[ ! -s "${temporary}" ]]; then - printf '[%s] %s gate produced no attributable output\n' \ - "${module}" "${selected}" >&2 - rm -f "${temporary}" - return 1 - fi - mv "${temporary}" "${output}" -} - -go_test() { - local tags - tags="$(test_tags)" - if [[ -n "${tags}" ]]; then - isolated go test -tags="${tags}" "$@" - else - isolated go test "$@" - fi -} - -run_gate() { - local selected="$1" - printf '\n[%s] %s\n' "${module}" "${selected}" - case "${selected}" in - format) - if target="$(find_make_target format)"; then - package_make GOWORK=off "${target}" - else - local -a format_files=() - while IFS= read -r package_directory; do - while IFS= read -r -d '' source_file; do - format_files+=("${source_file}") - done < <( - find "${package_directory}" -maxdepth 1 -type f \ - -name '*.go' -print0 - ) - done < <( - jq -r --arg directory "${module}" \ - '.modules[] | select(.directory == $directory) | - .packages[].directory' "${root}/modules.json" - ) - if [[ "${#format_files[@]}" -gt 0 ]]; then - gofmt -w "${format_files[@]}" - fi - fi - ;; - format-check) - if target="$(find_make_target format-check)"; then - package_make GOWORK=off "${target}" - else - local -a format_files=() - while IFS= read -r package_directory; do - while IFS= read -r -d '' source_file; do - format_files+=("${source_file}") - done < <( - find "${package_directory}" -maxdepth 1 -type f \ - -name '*.go' -print0 - ) - done < <( - jq -r --arg directory "${module}" \ - '.modules[] | select(.directory == $directory) | - .packages[].directory' "${root}/modules.json" - ) - unformatted="" - if [[ "${#format_files[@]}" -gt 0 ]]; then - unformatted="$(gofmt -l "${format_files[@]}")" - fi - [[ -z "${unformatted}" ]] || { - printf 'unformatted Go files:\n%s\n' "${unformatted}" >&2 - exit 1 - } - fi - ;; - tidy-check) - enable_local_proxy - GOWORK=off GOFLAGS="${GOLIB_UPSTREAM_GOFLAGS}" go mod tidy -diff - ;; - tidy) - enable_local_proxy - refresh_owned_sums - GOWORK=off GOFLAGS="${GOLIB_UPSTREAM_GOFLAGS}" \ - "${GOLIB_REAL_GO}" mod tidy - ;; - test) - applicable tests || { skip_not_applicable tests; return; } - packages="$(isolated go list ./...)" - [[ -n "${packages}" ]] || { - printf '[%s] no Go packages were executed\n' "${module}" >&2 - exit 1 - } - go_test ./... -count=1 -timeout=20m - ;; - workspace-test) - applicable tests || { skip_not_applicable tests; return; } - go test ./... -count=1 -timeout=20m - ;; - race) - applicable race || { skip_not_applicable race; return; } - go_test -race ./... -count=1 -timeout=20m - ;; - coverage) - applicable coverage || { skip_not_applicable coverage; return; } - enable_local_proxy - "${root}/.golib/scripts/check-coverage.sh" "${module}" - ;; - mutation) - applicable mutation || { skip_not_applicable mutation; return; } - enable_local_proxy - "${root}/.golib/scripts/check-mutation.sh" "${module}" - ;; - fuzz) - applicable fuzz || { skip_not_applicable fuzz; return; } - enable_local_proxy - if target="$(find_make_target fuzz fuzz-smoke)"; then - package_make \ - FUZZ_TIME="${GOLIB_FUZZ_SMOKE_BUDGET:-10000x}" \ - FUZZTIME="${GOLIB_FUZZ_SMOKE_BUDGET:-10000x}" \ - "${target}" - else - "${root}/.golib/scripts/check-fuzz.sh" "${module}" - fi - ;; - safety) - "${root}/.golib/scripts/check-go-safety.sh" "${module}" - ;; - vet) - applicable lint || { skip_not_applicable lint; return; } - isolated go vet ./... - ;; - lint) - applicable lint || { skip_not_applicable lint; return; } - enable_local_proxy - GOLANGCI_LINT_CACHE="${GOLIB_ISOLATED_MODFILES_DIRECTORY}/golangci-lint-cache" - export GOLANGCI_LINT_CACHE - mkdir -p "${GOLANGCI_LINT_CACHE}" - run_go_tool \ - "github.com/golangci/golangci-lint/v2/cmd/golangci-lint@${GOLANGCI_LINT_VERSION}" \ - golangci-lint \ - run --allow-parallel-runners --timeout=10m ./... - ;; - staticcheck) - applicable lint || { skip_not_applicable lint; return; } - run_go_tool \ - "honnef.co/go/tools/cmd/staticcheck@${STATICCHECK_VERSION}" \ - staticcheck ./... - ;; - nilaway) - applicable lint || { skip_not_applicable lint; return; } - set +e - run_go_tool \ - "go.uber.org/nilaway/cmd/nilaway@${NILAWAY_VERSION}" \ - nilaway \ - -include-pkgs="$(go mod edit -json | jq -r '.Module.Path')" ./... - status=$? - set -e - printf '[%s] NilAway advisory exit status: %s\n' "${module}" "${status}" - ;; - vulnerability) - applicable security || { skip_not_applicable security; return; } - run_go_tool \ - "golang.org/x/vuln/cmd/govulncheck@${GOVULNCHECK_VERSION}" \ - govulncheck ./... - ;; - secrets) - applicable security || { skip_not_applicable security; return; } - run_go_tool \ - "github.com/zricethezav/gitleaks/v8@${GITLEAKS_VERSION}" \ - gitleaks \ - dir . --config "${root}/.gitleaks.toml" --no-banner --redact - ;; - licenses) - applicable security || { skip_not_applicable security; return; } - test -s "${root}/LICENSE" - module_path="$(go mod edit -json | jq -er '.Module.Path')" - if [[ "${module_path}" != "github.com/faustbrian/go-lease" && - "${module_path}" != github.com/faustbrian/go-lease/* ]]; then - printf '[%s] refusing to ignore non-owned module license: %s\n' \ - "${module}" "${module_path}" >&2 - exit 1 - fi - run_go_tool \ - "github.com/google/go-licenses/v2@${GO_LICENSES_VERSION}" \ - go-licenses \ - check ./... \ - --ignore "github.com/faustbrian/go-lease" - ;; - sbom) - applicable security || { skip_not_applicable security; return; } - mkdir -p "${root}/.artifacts/${module}" - run_go_tool \ - "github.com/CycloneDX/cyclonedx-gomod/cmd/cyclonedx-gomod@${CYCLONEDX_VERSION}" \ - cyclonedx-gomod \ - mod -json -licenses -type library -noserial -notimestamp \ - -output "${root}/.artifacts/${module}/sbom.json" . - test -s "${root}/.artifacts/${module}/sbom.json" - ;; - docs) - applicable documentation || { skip_not_applicable documentation; return; } - enable_local_proxy - if [[ "${module}" == "." ]]; then - GOWORK=off "${root}/.golib/scripts/check-documentation.sh" - fi - if target="$(find_make_target docs documentation)"; then - package_make "${target}" - elif [[ "${module}" != "." ]]; then - GOWORK=off go test ./... -run '^Example' -count=1 - fi - ;; - api) - applicable api_compatibility || { skip_not_applicable api_compatibility; return; } - install_go_tool \ - "golang.org/x/exp/cmd/apidiff@${APIDIFF_VERSION}" \ - apidiff - export GOLIB_APIDIFF="${GOLIB_GO_TOOL_PATH}" - if target="$(find_make_target api-compat api-check api compatibility)"; then - package_make "${target}" - elif [[ -x "./scripts/check-api.sh" ]]; then - GOWORK=off ./scripts/check-api.sh - else - "${root}/.golib/scripts/check-api-baseline.sh" "${module}" - fi - ;; - api-update) - applicable api_compatibility || { skip_not_applicable api_compatibility; return; } - install_go_tool \ - "golang.org/x/exp/cmd/apidiff@${APIDIFF_VERSION}" \ - apidiff - export GOLIB_APIDIFF="${GOLIB_GO_TOOL_PATH}" - "${root}/.golib/scripts/update-api-baseline.sh" "${module}" - ;; - conformance) - applicable conformance || { skip_not_applicable conformance; return; } - enable_local_proxy - if target="$(find_make_target conformance specification)"; then - run_make_evidence conformance "${target}" - else - printf '[%s] conformance is declared but has no command\n' \ - "${module}" >&2 - exit 1 - fi - ;; - interoperability) - enable_local_proxy - if target="$(find_make_target interoperability integration conformance)"; then - run_make_evidence interoperability "${target}" - elif interoperability_declared; then - printf '[%s] interoperability is declared but has no command\n' \ - "${module}" >&2 - exit 1 - else - skip_not_applicable interoperability - fi - ;; - benchmark) - applicable benchmarks || { skip_not_applicable benchmarks; return; } - run_benchmark - ;; - release-dry-run) - "${root}/.golib/scripts/release.sh" --dry-run "${module}" - ;; - release-public) - "${root}/.golib/scripts/release.sh" --dry-run --public "${module}" - ;; - check) - while IFS= read -r required_gate; do - [[ -n "${required_gate}" ]] || continue - run_gate "${required_gate}" - done <"${root}/.golib/scripts/check-gates.txt" - ;; - *) - printf 'unknown gate: %s\n' "${selected}" >&2 - exit 2 - ;; - esac -} - -set -a -# shellcheck disable=SC1091 # Repository-pinned tool versions. -source "${root}/.golib/versions.env" -set +a -cd "${directory}" -run_gate "${gate}" diff --git a/.golib/scripts/check-mutation.sh b/.golib/scripts/check-mutation.sh deleted file mode 100755 index 2329a37..0000000 --- a/.golib/scripts/check-mutation.sh +++ /dev/null @@ -1,10 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 1 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -exec "${root}/.golib/scripts/internal/run-mutation.sh" enforce "$1" diff --git a/.golib/scripts/codeql-build.sh b/.golib/scripts/codeql-build.sh deleted file mode 100755 index 8bc3afb..0000000 --- a/.golib/scripts/codeql-build.sh +++ /dev/null @@ -1,65 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -root="$(git rev-parse --show-toplevel)" -task="$(mktemp -d "${TMPDIR:-/tmp}/golib-codeql-build.XXXXXX")" -cleanup() { - find "${task}" -depth -delete 2>/dev/null || true -} -trap cleanup EXIT HUP INT TERM - -root_version="$(jq -er ' - .modules[] | select(.directory == ".") | .version -' "${root}/modules.json")" -self_proxy="${task}/self-proxy" -"${root}/.golib/scripts/build-local-proxy.sh" \ - "${self_proxy}" "v${root_version}" "." -upstream_proxy="${GOPROXY:-$(go env GOPROXY)}" -export GOPROXY="file://${self_proxy},${upstream_proxy}" -current_no_sum_db="$(go env GONOSUMDB)" -export GONOSUMDB="github.com/faustbrian/go-*${current_no_sum_db:+,${current_no_sum_db}}" - -while IFS= read -r module; do - [[ -n "${module}" ]] || continue - module_root="${root}" - if [[ "${module}" != "." ]]; then - module_root="${root}/${module}" - fi - while IFS= read -r package; do - [[ -n "${package}" ]] || continue - package_tags="$( - jq -r \ - --arg module "${module}" \ - --arg package "${package}" ' - .modules[] - | select(.directory == $module) - | .packages[] - | select(.import_path == $package) - | .build_tags[]? - ' "${root}/modules.json" - )" - slug="$(printf '%s' "${package}" | tr '/.' '--')" - if [[ "${module}" == "benchmarks/platform" && -n "${package_tags}" ]]; then - package_tags=benchmark_disabled - fi - if [[ -z "${package_tags}" ]]; then - (cd "${module_root}" && GOWORK=off go build -o "${task}/${slug}" "${package}") - continue - fi - variant=0 - while IFS= read -r tag; do - [[ -n "${tag}" ]] || continue - (cd "${module_root}" && GOWORK=off go build \ - -tags="${tag}" -o "${task}/${slug}-${variant}" "${package}") - variant=$((variant + 1)) - done <<<"${package_tags}" - done < <( - jq -r --arg module "${module}" ' - .modules[] - | select(.directory == $module) - | .packages[] - | select(.build_required == true) - | .import_path - ' "${root}/modules.json" - ) -done < <(jq -r '.modules[].directory' "${root}/modules.json") diff --git a/.golib/scripts/create-verification-snapshot.sh b/.golib/scripts/create-verification-snapshot.sh deleted file mode 100755 index cceb53d..0000000 --- a/.golib/scripts/create-verification-snapshot.sh +++ /dev/null @@ -1,59 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 2 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -source_repository="$(cd "$1" && pwd)" -snapshot_directory="$2" -if [[ -e "${snapshot_directory}" ]]; then - printf 'snapshot destination already exists: %s\n' \ - "${snapshot_directory}" >&2 - exit 1 -fi - -git -C "${source_repository}" rev-parse --verify HEAD >/dev/null -git clone --shared --no-checkout --quiet \ - "${source_repository}" "${snapshot_directory}" -git -C "${snapshot_directory}" config --local core.fsmonitor false -git -C "${snapshot_directory}" checkout --detach --quiet \ - "$(git -C "${source_repository}" rev-parse HEAD)" - -patch="$(mktemp "${TMPDIR:-/tmp}/golib-snapshot-patch.XXXXXX")" -cleanup() { - rm -f "${patch}" -} -trap cleanup EXIT HUP INT TERM -git -C "${source_repository}" diff --binary --full-index HEAD -- >"${patch}" -if [[ -s "${patch}" ]]; then - git -C "${snapshot_directory}" apply --binary "${patch}" -fi -git -C "${snapshot_directory}" ls-files --deleted -z | - git -C "${snapshot_directory}" update-index --remove -z --stdin - -# A staged addition remains tracked even when a machine-wide ignore rule -# matches its name. Mirror that index classification so verification input -# discovery sees the same files in the snapshot and source repository. -while IFS= read -r -d '' path; do - git -C "${snapshot_directory}" add -N -f -- "${path}" -done < <( - git -C "${source_repository}" \ - diff --cached --name-only --diff-filter=A -z -) - -while IFS= read -r -d '' path; do - mkdir -p "${snapshot_directory}/$(dirname "${path}")" - cp -pP \ - "${source_repository}/${path}" \ - "${snapshot_directory}/${path}" -done < <( - git -C "${source_repository}" \ - ls-files --others --exclude-standard -z -) - -mkdir -p "${source_repository}/.artifacts" -ln -s "${source_repository}/.artifacts" \ - "${snapshot_directory}/.artifacts" -printf '%s\n' '/.artifacts' >>"${snapshot_directory}/.git/info/exclude" diff --git a/.golib/scripts/discover-mutation.sh b/.golib/scripts/discover-mutation.sh deleted file mode 100755 index d84a4ff..0000000 --- a/.golib/scripts/discover-mutation.sh +++ /dev/null @@ -1,10 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 1 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -exec "${root}/.golib/scripts/internal/run-mutation.sh" discover "$1" diff --git a/.golib/scripts/filter-releasable-modules.sh b/.golib/scripts/filter-releasable-modules.sh deleted file mode 100755 index 589eca5..0000000 --- a/.golib/scripts/filter-releasable-modules.sh +++ /dev/null @@ -1,12 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -root="${GOLIB_ROOT:-$(git rev-parse --show-toplevel)}" -selection="$(cat)" - -jq -r --arg selection "${selection}" ' - ($selection | split("\n") | map(select(length > 0)))[] as $directory - | .modules[] - | select(.directory == $directory and .releasable == true) - | .directory -' "${root}/modules.json" diff --git a/.golib/scripts/gate-input-digest.sh b/.golib/scripts/gate-input-digest.sh deleted file mode 100755 index 84a2d91..0000000 --- a/.golib/scripts/gate-input-digest.sh +++ /dev/null @@ -1,1035 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -lt 2 || $# -gt 3 ]]; then - printf 'usage: %s [package-directory]\n' "$0" >&2 - exit 2 -fi - -root="${GOLIB_ROOT:-$(git rev-parse --show-toplevel)}" -gate="$1" -module="$2" -package="${3:-}" -input_policy="${GOLIB_GATE_INPUT_POLICY:-current}" -case "${input_policy}" in - current|legacy-api-baseline|legacy-runner-isolation|legacy-runner-isolation-post) ;; - *) - printf 'unsupported gate input policy: %s\n' "${input_policy}" >&2 - exit 2 - ;; -esac -if ! jq -e --arg directory "${module}" \ - '.modules[] | select(.directory == $directory)' \ - "${root}/modules.json" >/dev/null; then - printf 'module is absent from modules.json: %s\n' "${module}" >&2 - exit 2 -fi - -manifest="$(mktemp "${TMPDIR:-/tmp}/golib-gate-inputs.XXXXXX")" -directories="$(mktemp "${TMPDIR:-/tmp}/golib-gate-directories.XXXXXX")" -input_files="$(mktemp "${TMPDIR:-/tmp}/golib-gate-files.XXXXXX")" -package_data="${manifest}.packages" -relevant_package_data="${manifest}.relevant-packages" -existing_files="${manifest}.existing" -file_hashes="${manifest}.hashes" -nested_directories="${manifest}.nested" -bounded_output="${manifest}.bounded-output" -owned_module_paths="${manifest}.owned-modules" -digest_modfile="" -cleanup() { - rm -f \ - "${manifest}" "${directories}" "${input_files}" "${package_data}" \ - "${relevant_package_data}" \ - "${existing_files}" "${file_hashes}" "${nested_directories}" \ - "${bounded_output}" "${owned_module_paths}" - if [[ -n "${digest_modfile}" ]]; then - rm -f "${digest_modfile}" "${digest_modfile%.mod}.sum" - fi -} -trap cleanup EXIT HUP INT TERM - -append_value() { - printf 'value %s=%s\n' "$1" "$2" >>"${manifest}" -} - -append_version_value() { - local name="$1" - local value - if ! value="$({ - # shellcheck source=/dev/null - source "${root}/.golib/versions.env" - set +u - printf '%s' "${!name}" - })" || [[ -z "${value}" ]]; then - printf 'required version value is missing: %s\n' "${name}" >&2 - exit 1 - fi - append_value "version:${name}" "${value}" -} - -append_required_service_versions() { - local service variable - while IFS= read -r service; do - [[ -n "${service}" ]] || continue - case "${service}" in - postgresql) variable=POSTGRES_IMAGE ;; - valkey) variable=VALKEY_IMAGE ;; - redis) variable=REDIS_IMAGE ;; - nats) variable=NATS_IMAGE ;; - nsq) variable=NSQ_IMAGE ;; - rabbitmq) variable=RABBITMQ_IMAGE ;; - rabbitstream-standalone) - append_file "${root}/.golib/services/rabbitstream/enabled_plugins" - append_file "${root}/.golib/services/rabbitstream/standalone-compose.yaml" - append_file "${root}/.golib/services/rabbitstream/standalone-setup.sh" - append_file "${root}/.golib/services/rabbitstream/standalone.conf" - continue - ;; - rabbitstream) - append_file "${root}/.golib/services/rabbitstream/compose.yaml" - append_file "${root}/.golib/services/rabbitstream/enabled_plugins" - append_file "${root}/.golib/services/rabbitstream/rabbit1.conf" - append_file "${root}/.golib/services/rabbitstream/rabbit2.conf" - append_file "${root}/.golib/services/rabbitstream/rabbit3.conf" - append_file "${root}/.golib/services/rabbitstream/setup.sh" - append_file "${root}/.golib/services/rabbitstream/standalone-compose.yaml" - append_file "${root}/.golib/services/rabbitstream/standalone-setup.sh" - append_file "${root}/.golib/services/rabbitstream/standalone.conf" - append_file "${root}/.golib/services/rabbitstream/tls-compose.yaml" - append_file "${root}/.golib/services/rabbitstream/tls-rabbitmq.conf" - append_file "${root}/.golib/services/rabbitstream/tls-setup.sh" - continue - ;; - opensearch) - append_file "${root}/.golib/services/opensearch/opensearch-images.env" - continue - ;; - *) - printf 'unsupported required service %s for %s\n' \ - "${service}" "${module}" >&2 - exit 1 - ;; - esac - append_version_value "${variable}" - done < <(jq -r --arg directory "${module}" ' - .modules[] - | select(.directory == $directory) - | (.required_services // [])[] - ' "${root}/modules.json" | LC_ALL=C sort -u) -} - -append_gate_tool_versions() { - case "${gate}" in - lint) append_version_value GOLANGCI_LINT_VERSION ;; - staticcheck) append_version_value STATICCHECK_VERSION ;; - nilaway) append_version_value NILAWAY_VERSION ;; - vulnerability) append_version_value GOVULNCHECK_VERSION ;; - secrets) append_version_value GITLEAKS_VERSION ;; - licenses) append_version_value GO_LICENSES_VERSION ;; - sbom) append_version_value CYCLONEDX_VERSION ;; - api|api-update) append_version_value APIDIFF_VERSION ;; - workflow) append_version_value ACTIONLINT_VERSION ;; - # Interoperability scripts may use package-specific pinned tools that - # are not yet represented individually in the module catalog. - interoperability) append_file "${root}/.golib/versions.env" ;; - esac -} - -append_file() { - local file="$1" - local relative digest - [[ -f "${file}" ]] || { - printf 'gate input is missing: %s\n' "${file}" >&2 - exit 1 - } - relative="${file#"${root}/"}" - digest="$(shasum -a 256 "${file}" | awk '{print $1}')" - printf 'file %s %s\n' "${digest}" "${relative}" >>"${manifest}" -} - -append_mutation_module_manifest() { - local file="$1" - local relative digest - if [[ ! -f "${owned_module_paths}" ]]; then - jq -r '.modules[].module_path' "${root}/modules.json" | - LC_ALL=C sort -u >"${owned_module_paths}" - fi - relative="${file#"${root}/"}" - digest="$({ - GOLIB_OWNED_MODULE_PATHS="${owned_module_paths}" perl -pe ' - BEGIN { - open my $paths, "<", $ENV{GOLIB_OWNED_MODULE_PATHS} - or die "open owned module paths: $!"; - chomp(@owned = <$paths>); - } - for my $owned (@owned) { - s/(\Q$owned\E)([ \t]+)v[^\s]+/$1$2v0.0.0/g; - } - ' "${file}" - } | shasum -a 256 | awk '{print $1}')" - printf 'file %s %s\n' "${digest}" "${relative}" >>"${manifest}" -} - -append_repository_files() { - local file - : >"${existing_files}" - while IFS= read -r file; do - [[ -n "${file}" ]] || continue - if [[ -f "${root}/${file}" ]]; then - printf '%s\n' "${file}" >>"${existing_files}" - else - append_value missing-file "${file}" - fi - done - [[ -s "${existing_files}" ]] || return - git -C "${root}" hash-object --stdin-paths \ - <"${existing_files}" >"${file_hashes}" - paste "${file_hashes}" "${existing_files}" | - awk -F '\t' '{ printf "file %s %s\n", $1, $2 }' >>"${manifest}" -} - -append_module_files() { - local directory="$1" - local include_api_baseline=0 - local include_documentation=0 - local include_secret_policy=0 - local include_tests=0 - if [[ "${input_policy}" == "legacy-api-baseline" ]]; then - include_api_baseline=1 - fi - if [[ "${directory}" == "${module}" ]]; then - include_tests=1 - fi - case "${gate}" in - api|api-update) - include_api_baseline=1 - ;; - docs) - include_documentation=1 - ;; - secrets) - include_documentation=1 - include_secret_policy=1 - ;; - esac - : >"${nested_directories}" - jq -r --arg directory "${directory}" ' - .modules[] - | .directory - | select( - $directory != "." and - . != $directory and - startswith($directory + "/") - ) - ' "${root}/modules.json" >"${nested_directories}" - git -C "${root}" ls-files -co --exclude-standard -- "${directory}" | - awk \ - -v include_api_baseline="${include_api_baseline}" \ - -v include_documentation="${include_documentation}" \ - -v include_secret_policy="${include_secret_policy}" \ - -v include_tests="${include_tests}" \ - -v module_directory="${directory}" ' - FILENAME != "-" { - nested[++count] = $0 - next - } - { - relative = tolower($0) - module_prefix = module_directory == "." ? "" : tolower(module_directory) "/" - if (module_prefix != "" && substr(relative, 1, length(module_prefix)) == module_prefix) { - relative = substr(relative, length(module_prefix) + 1) - } - is_markdown = relative ~ /\.(md|markdown)$/ - in_documentation = relative ~ /^(docs|\.ai)\// - in_test_data = relative ~ /(^|\/)(testdata|fixtures|corpus)\// - is_named_documentation = relative ~ /(^|\/)(readme|changelog|contributing|security|code_of_conduct|support)\.(md|markdown)$/ - is_generated_documentation = relative == "llms.txt" || relative == "llms-full.txt" - is_api_baseline = relative == "api/baseline.txt" - is_repository_catalog = relative == "modules.json" || relative == "packages.json" - is_secret_policy = relative == ".gitleaks.toml" - is_test_source = relative ~ /_test\.go$/ - skip_documentation = !include_documentation && (is_generated_documentation || (is_markdown && (in_documentation || (!in_test_data && is_named_documentation)))) - skip_secret_policy = !include_secret_policy && is_secret_policy - if (is_repository_catalog) { - next - } - if (!include_api_baseline && is_api_baseline) { - next - } - if (skip_secret_policy) { - next - } - if (!include_tests && is_test_source) { - next - } - for (position = 1; position <= count; position++) { - prefix = nested[position] "/" - if ($0 == nested[position] || substr($0, 1, length(prefix)) == prefix) { - next - } - } - if (skip_documentation) { - next - } - print - } - ' "${nested_directories}" - | - while IFS= read -r file; do - if [[ -e "${root}/${file}" || -L "${root}/${file}" ]]; then - printf '%s\n' "${file}" - fi - done >>"${input_files}" -} - -append_tool_inputs() { - append_version_value GREMLINS_VERSION - append_required_service_versions - append_file "${root}/.golib/scripts/internal/mutation-command.sh" - append_file "${root}/.golib/scripts/internal/mutation-coverage.sh" - append_file "${root}/.golib/scripts/patches/gremlins-run-all-mutants.patch" - append_file "${root}/.golib/scripts/patches/gremlins-shared-coverage.patch" - append_file "${root}/.golib/scripts/start-services.sh" -} - -append_environment() { - append_value go-version "$(go env GOVERSION)" - append_value goos "$(go env GOOS)" - append_value goarch "$(go env GOARCH)" - append_value cgo-enabled "$(go env CGO_ENABLED)" -} - -append_assurance_environment() { - if [[ -z "${GOLIB_ASSURANCE_GO_VERSION:-}" && - -z "${GOLIB_ASSURANCE_GOOS:-}" && - -z "${GOLIB_ASSURANCE_GOARCH:-}" && - -z "${GOLIB_ASSURANCE_CGO_ENABLED:-}" && - -z "${GOLIB_ASSURANCE_KERNEL:-}" && - -z "${GOLIB_ASSURANCE_NODE:-}" ]]; then - return 1 - fi - if [[ -z "${GOLIB_ASSURANCE_GO_VERSION:-}" || - -z "${GOLIB_ASSURANCE_GOOS:-}" || - -z "${GOLIB_ASSURANCE_GOARCH:-}" || - -z "${GOLIB_ASSURANCE_CGO_ENABLED:-}" || - -z "${GOLIB_ASSURANCE_KERNEL:-}" || - -z "${GOLIB_ASSURANCE_NODE:-}" ]]; then - printf 'operational-assurance environment override is incomplete\n' >&2 - exit 2 - fi - local value - for value in \ - "${GOLIB_ASSURANCE_GO_VERSION}" \ - "${GOLIB_ASSURANCE_GOOS}" \ - "${GOLIB_ASSURANCE_GOARCH}" \ - "${GOLIB_ASSURANCE_CGO_ENABLED}" \ - "${GOLIB_ASSURANCE_KERNEL}" \ - "${GOLIB_ASSURANCE_NODE}"; do - if [[ "${value}" == *$'\n'* || "${value}" == *$'\r'* ]]; then - printf 'operational-assurance environment override contains control characters\n' >&2 - exit 2 - fi - done - if [[ "${GOLIB_ASSURANCE_CGO_ENABLED}" != "0" && - "${GOLIB_ASSURANCE_CGO_ENABLED}" != "1" ]]; then - printf 'operational-assurance cgo override must be 0 or 1\n' >&2 - exit 2 - fi - append_value go-version "${GOLIB_ASSURANCE_GO_VERSION}" - append_value goos "${GOLIB_ASSURANCE_GOOS}" - append_value goarch "${GOLIB_ASSURANCE_GOARCH}" - append_value cgo-enabled "${GOLIB_ASSURANCE_CGO_ENABLED}" - append_value kernel "${GOLIB_ASSURANCE_KERNEL}" - append_value node "${GOLIB_ASSURANCE_NODE}" -} - -bounded_command_output() { - local timeout_seconds="$1" - shift - local command_pid status timer_pid - : >"${bounded_output}" - "$@" >"${bounded_output}" 2>/dev/null & - command_pid=$! - ( - sleep "${timeout_seconds}" - kill -TERM "${command_pid}" 2>/dev/null || true - sleep 1 - kill -KILL "${command_pid}" 2>/dev/null || true - ) & - timer_pid=$! - status=0 - wait "${command_pid}" || status=$? - kill -TERM "${timer_pid}" 2>/dev/null || true - wait "${timer_pid}" 2>/dev/null || true - if [[ "${status}" -eq 0 ]]; then - cat "${bounded_output}" - fi - return "${status}" -} - -append_legacy_docker_environment() { - local docker_timeout docker_value - if ! jq -e --arg directory "${module}" ' - .modules[] - | select(.directory == $directory) - | select((.required_services // []) | length > 0) - ' "${root}/modules.json" >/dev/null; then - append_value docker not-required - return - fi - if [[ -n "${GOLIB_LEGACY_DOCKER_VALUE:-}" ]]; then - append_value docker "${GOLIB_LEGACY_DOCKER_VALUE}" - return - fi - if ! command -v docker >/dev/null 2>&1; then - append_value docker missing - return - fi - docker_timeout="${GOLIB_DOCKER_VERSION_TIMEOUT_SECONDS:-5}" - if [[ ! "${docker_timeout}" =~ ^[1-9][0-9]*$ ]]; then - printf 'invalid Docker version timeout: %s\n' "${docker_timeout}" >&2 - exit 1 - fi - docker_value="$({ - bounded_command_output \ - "${docker_timeout}" \ - docker version --format '{{.Server.Version}}' || - printf unavailable - })" - append_value docker "${docker_value}" -} - -append_verification_environment() { - if [[ "${gate}" == "operational-assurance" ]] && append_assurance_environment; then - return - fi - append_environment - append_value kernel "$(uname -srm)" - # Pinned service images define the runtime contract. Live daemon - # availability is orchestration state and must not invalidate evidence. - # The legacy policy reproduces the old identity solely for verified, - # one-time evidence migration. - if [[ "${input_policy}" == "legacy-api-baseline" ]]; then - append_legacy_docker_environment - fi - if command -v node >/dev/null 2>&1; then - append_value node "$(node --version)" - else - append_value node missing - fi -} - -normalize_irrelevant_formatter_dispatch() { - awk -v selected_gate="${gate}" ' - BEGIN { quote = sprintf("%c", 39) } - selected_gate != "format" && selected_gate != "format-check" && - $0 == " format)" { - print " format)" - print " find . -name " quote "*.go" quote " -not -path " quote "./.tools/*" quote " -print0 | xargs -0 gofmt -w" - print " ;;" - print " format-check)" - print " unformatted=\"$(find . -name " quote "*.go" quote " -not -path " quote "./.tools/*" quote " -print0 | xargs -0 gofmt -l)\"" - print " [[ -z \"${unformatted}\" ]] || {" - print " printf " quote "unformatted Go files:\\n%s\\n" quote " \"${unformatted}\" >&2" - print " exit 1" - print " }" - print " ;;" - skip_formatter_dispatch = 1 - next - } - skip_formatter_dispatch && $0 == " tidy-check)" { - skip_formatter_dispatch = 0 - print - next - } - skip_formatter_dispatch { next } - { print } - ' -} - -append_verification_tool_files() { - local check_module_digest legacy_runner_digest - local paths=( - scripts/start-services.sh - ) - if [[ "${gate}" == "operational-assurance" && "${input_policy}" == "current" ]]; then - paths+=( - scripts/create-verification-snapshot.sh - scripts/run-modules.sh - scripts/stop-services.sh - ) - elif [[ "${input_policy}" == "legacy-runner-isolation" || - "${input_policy}" == "legacy-runner-isolation-post" ]]; then - paths+=( - scripts/create-verification-snapshot.sh - scripts/stop-services.sh - ) - # These are the exact Git blob identities immediately before and after - # process isolation, matching append_repository_files. They identify - # retained evidence without requiring that history to remain available. - legacy_runner_digest='cf841512fc1e48c8c7708259c878028f06a8726f' - if [[ "${input_policy}" == "legacy-runner-isolation-post" ]]; then - legacy_runner_digest='d30bc5a6f7e52b2080a3fe13200dfb2963a1415a' - fi - printf 'file %s %s\n' \ - "${legacy_runner_digest}" \ - 'scripts/run-modules.sh' >>"${manifest}" - else - # Snapshot creation, module selection, and post-gate cleanup do not - # alter a single gate's command or inputs. Their behavior belongs to - # aggregate-run evidence rather than every package checkpoint. - append_value verification-orchestration-contract v1 - fi - append_gate_tool_versions - append_required_service_versions - case "${gate}" in - format-check|workspace-test|safety|benchmark|release-public) ;; - *) - paths+=( - scripts/build-local-proxy.sh - scripts/internal/isolated-go.sh - ) - ;; - esac - case "${gate}" in - coverage) - paths+=(scripts/check-coverage.sh) - ;; - fuzz) - paths+=(scripts/check-fuzz.sh) - ;; - safety) - paths+=(scripts/check-go-safety.sh) - ;; - api) - paths+=(scripts/check-api-baseline.sh) - ;; - api-update) - paths+=(scripts/update-api-baseline.sh) - ;; - licenses) - paths+=(LICENSE) - ;; - release-dry-run|release-public) - paths+=( - scripts/filter-releasable-modules.sh - scripts/release.sh - ) - ;; - esac - git -C "${root}" ls-files -co --exclude-standard -- \ - "${paths[@]}" >>"${input_files}" - - # Runner concurrency changes whether a linter can start, not its findings. - # Keep successful gate evidence bound to the analyzer contract rather than - # invalidating every package when isolated runners are made parallel-safe. - if [[ -f "${root}/.golib/scripts/check-module.sh" ]]; then - if [[ "${gate}" == "docs" ]]; then - check_module_digest="$( - normalize_irrelevant_formatter_dispatch \ - <"${root}/.golib/scripts/check-module.sh" | - sed 's/ --allow-parallel-runners//g' | - git hash-object --stdin - )" - else - # Root-only documentation dispatch cannot alter another gate's - # executable contract or invalidate its retained evidence. - check_module_digest="$( - awk ' - $0 == " if [[ \"${module}\" == \".\" ]]; then" { - print " if target=\"$(find_make_target docs documentation)\"; then" - skip_root_documentation = 1 - next - } - skip_root_documentation && $0 == " elif target=\"$(find_make_target docs documentation)\"; then" { - skip_root_documentation = 0 - next - } - skip_root_documentation { next } - { print } - ' "${root}/.golib/scripts/check-module.sh" | - normalize_irrelevant_formatter_dispatch | - sed 's/ --allow-parallel-runners//g' | - git hash-object --stdin - )" - fi - printf 'file %s %s\n' \ - "${check_module_digest}" \ - 'scripts/check-module.sh' >>"${manifest}" - else - append_value missing-file scripts/check-module.sh - fi -} - -verification_module_policy() { - local directory="$1" - jq -S -c --arg directory "${directory}" ' - .modules[] - | select(.directory == $directory) - | { - directory, - module_path, - owned_dependencies, - required_services, - test_tags, - interoperability_tools, - gates - } - ' "${root}/modules.json" -} - -verification_digest() { - local directory file - local repository_paths=() - append_value gate "${gate}" - append_value module "${module}" - append_verification_environment - - printf '%s\n' "${module}" >"${directories}" - jq -r --arg directory "${module}" ' - . as $catalog - | def closure($seen): - ([ - $catalog.modules[] - | select(.module_path as $path | $seen | index($path)) - | .owned_dependencies[] - ] | unique) as $dependencies - | ($seen + $dependencies | unique) as $next - | if $next == $seen then $next else closure($next) end; - (.modules[] | select(.directory == $directory).owned_dependencies) as $owned - | closure($owned) as $paths - | .modules[] - | select(.module_path as $path | $paths | index($path)) - | .directory - ' "${root}/modules.json" >>"${directories}" - - while IFS= read -r directory; do - [[ -n "${directory}" ]] || continue - append_value "module-policy:${directory}" "$( - verification_module_policy "${directory}" - )" - append_value "package-policy:${directory}" "$( - jq -S -c --arg directory "${directory}" \ - '[.packages[] | select(.module_directory == $directory)]' \ - "${root}/packages.json" - )" - append_module_files "${directory}" - done < <(LC_ALL=C sort -u "${directories}") - append_verification_tool_files - case "${gate}" in - secrets) - repository_paths+=(.gitleaks.toml) - ;; - benchmark|workspace-test) - repository_paths+=(go.work) - ;; - esac - if [[ "${#repository_paths[@]}" -gt 0 ]]; then - git -C "${root}" ls-files -co --exclude-standard -- \ - "${repository_paths[@]}" >>"${input_files}" - fi - - LC_ALL=C sort -u "${input_files}" | append_repository_files -} - -mutation_digest() { - local package_directory package_input_digest - append_value gate mutation - append_value module "${module}" - append_file "${root}/.golib/scripts/check-mutation.sh" - append_file "${root}/.golib/scripts/internal/configure-mutation-workers.sh" - append_file "${root}/.golib/scripts/internal/run-mutation.sh" - append_file "${root}/.golib/scripts/internal/mutation-scratch.sh" - append_file "${root}/.golib/scripts/internal/reuse-mutation-coverage.sh" - while IFS= read -r package_directory; do - [[ -n "${package_directory}" ]] || continue - package_input_digest="$( - "${root}/.golib/scripts/gate-input-digest.sh" \ - mutation "${module}" "${package_directory}" - )" - append_value "package:${package_directory}" "${package_input_digest}" - done < <( - jq -r --arg directory "${module}" ' - .modules[] - | select(.directory == $directory) - | .packages[] - | select(.coverage_required == true) - | .directory - ' "${root}/modules.json" | LC_ALL=C sort - ) -} - -legacy_digest() { - local directory - append_value gate mutation - append_value module "${module}" - append_environment - append_value module-policy "$( - jq -S -c --arg directory "${module}" \ - '.modules[] | select(.directory == $directory)' \ - "${root}/modules.json" - )" - append_value package-policy "$( - jq -S -c --arg directory "${module}" \ - '[.packages[] | select(.module_directory == $directory)]' \ - "${root}/packages.json" - )" - append_value zero-mutant-policy "$( - jq -S -c --arg directory "${module}" \ - '[.packages[] | select(.module_directory == $directory)]' \ - "${root}/.golib/mutation-zero-inventory.json" - )" - - append_file "${root}/.golib/versions.env" - append_file "${root}/.golib/scripts/build-golib-gremlins.sh" - append_file "${root}/.golib/scripts/internal/mutation-command.sh" - append_file "${root}/.golib/scripts/patches/gremlins-run-all-mutants.patch" - append_file "${root}/.golib/scripts/patches/gremlins-shared-coverage.patch" - append_file "${root}/.golib/scripts/start-services.sh" - - printf '%s\n' "${module}" >"${directories}" - jq -r --arg directory "${module}" ' - . as $catalog - | def closure($seen): - ([ - $catalog.modules[] - | select(.module_path as $path | $seen | index($path)) - | .owned_dependencies[] - ] | unique) as $dependencies - | ($seen + $dependencies | unique) as $next - | if $next == $seen then $next else closure($next) end; - (.modules[] | select(.directory == $directory).owned_dependencies) as $owned - | closure($owned) as $paths - | .modules[] - | select(.module_path as $path | $paths | index($path)) - | .directory - ' "${root}/modules.json" >>"${directories}" - - while IFS= read -r directory; do - [[ -n "${directory}" ]] || continue - while IFS= read -r -d '' file; do - append_file "${file}" - done < <( - find "${root}/${directory}" -type f \ - ! -path '*/.git/*' \ - ! -path '*/.artifacts/*' \ - ! -path '*/.tools/*' \ - ! -name '*.coverprofile' \ - ! -name 'coverage.out' \ - -print0 | LC_ALL=C sort -z - ) - done < <(LC_ALL=C sort -u "${directories}") -} - -package_digest() { - local data_name digest_go digest_go_flags digest_workspace flag owned_count - local module_path module_root package_directory resolution tags - local target_import_path - module_root="${root}/${module}" - if ! jq -e --arg directory "${module}" --arg package "${package}" ' - .modules[] - | select(.directory == $directory) - | .packages[] - | select(.directory == $package and .coverage_required == true) - ' "${root}/modules.json" >/dev/null; then - printf 'mutation package is absent from catalog: %s %s\n' \ - "${module}" "${package}" >&2 - exit 2 - fi - - append_value gate mutation - append_value module "${module}" - append_value package "${package}" - append_environment - append_value module-policy "$( - jq -S -c --arg directory "${module}" ' - .modules[] - | select(.directory == $directory) - | { - directory, - module_path, - go_version, - owned_dependencies, - required_services, - test_tags, - mutation: .gates.mutation - } - ' "${root}/modules.json" - )" - append_value package-policy "$( - jq -S -c --arg directory "${module}" --arg package "${package}" ' - .modules[] - | select(.directory == $directory) - | .packages[] - | select(.directory == $package) - | { - module_directory, - directory, - name, - import_path, - kind, - production, - executable, - coverage_required - } - ' "${root}/modules.json" - )" - append_value zero-mutant-policy "$( - jq -S -c --arg directory "${module}" --arg package "${package}" ' - [.packages[] | select( - .module_directory == $directory and - .package_directory == $package - )] - ' "${root}/.golib/mutation-zero-inventory.json" - )" - if [[ "${module}" == "pkg/ecma-regexp" ]]; then - append_value mutation-test-environment "TEST262_ROOT=unset" - fi - # Evidence orchestration does not affect which mutants execute or which - # tests observe them. Campaign semantics are captured by append_tool_inputs. - append_tool_inputs - - module_path="$(jq -r --arg directory "${module}" ' - .modules[] - | select(.directory == $directory) - | .module_path - ' "${root}/modules.json")" - tags="$(jq -r --arg directory "${module}" ' - .modules[] - | select(.directory == $directory) - | .test_tags - | join(",") - ' "${root}/modules.json")" - resolution="${GOLIB_MUTATION_DIGEST_RESOLUTION:-stable}" - if [[ "${resolution}" == "caller" ]]; then - ( - cd "${module_root}" - if [[ -n "${tags}" ]]; then - go list -deps -test -json -tags="${tags}" ./... - else - go list -deps -test -json ./... - fi - ) >"${package_data}" - elif [[ "${resolution}" != "stable" && - "${resolution}" != "observer-v1" && - "${resolution}" != "legacy-stable" ]]; then - printf 'unknown mutation digest resolution: %s\n' \ - "${resolution}" >&2 - exit 2 - else - digest_go="${GOLIB_REAL_GO:-$(command -v go)}" - digest_workspace=off - digest_go_flags="" - for flag in ${GOLIB_UPSTREAM_GOFLAGS:-${GOFLAGS:-}}; do - case "${flag}" in - -mod=*|-modfile=*) ;; - *) - digest_go_flags="$( - printf '%s%s' \ - "${digest_go_flags:+${digest_go_flags} }" "${flag}" - )" - ;; - esac - done - owned_count="$(jq -r --arg directory "${module}" ' - .modules[] | select(.directory == $directory) | .owned_dependencies | length - ' "${root}/modules.json")" - if [[ "${owned_count}" -gt 0 ]]; then - digest_modfile="${manifest}.mutation.mod" - cp "${module_root}/go.mod" "${digest_modfile}" - if [[ -f "${module_root}/go.sum" ]]; then - cp "${module_root}/go.sum" "${digest_modfile%.mod}.sum" - fi - while IFS=$'\t' read -r owned_path owned_directory; do - [[ -n "${owned_path}" && -n "${owned_directory}" ]] || continue - GOWORK=off "${digest_go}" mod edit -modfile="${digest_modfile}" \ - -replace="${owned_path}=${root}/${owned_directory}" - done < <(jq -r --arg directory "${module}" ' - . as $catalog - | def closure($seen): - ([ $catalog.modules[] - | select(.module_path as $path | $seen | index($path)) - | .owned_dependencies[] ] | unique) as $dependencies - | ($seen + $dependencies | unique) as $next - | if $next == $seen then $next else closure($next) end; - (.modules[] | select(.directory == $directory).owned_dependencies) as $owned - | closure($owned) as $paths - | .modules[] - | select(.module_path as $path | $paths | index($path)) - | [.module_path, .directory] - | @tsv - ' "${root}/modules.json") - digest_go_flags="$( - printf '%s%s' "${digest_go_flags:+${digest_go_flags} }" \ - "-modfile=${digest_modfile} -mod=mod" - )" - fi - ( - cd "${module_root}" - if [[ -n "${tags}" ]]; then - GOWORK="${digest_workspace}" GOFLAGS="${digest_go_flags}" \ - "${digest_go}" list -deps -test -json \ - -tags="${tags}" ./... - else - GOWORK="${digest_workspace}" GOFLAGS="${digest_go_flags}" \ - "${digest_go}" list -deps -test -json ./... - fi - ) >"${package_data}" - fi - - if [[ "${resolution}" == "legacy-stable" ]]; then - jq -s \ - --arg root "${root}/" \ - --arg module_path "${module_path}" ' - .[] - | select((.Dir // "") | startswith($root)) - | . + { - GolibMutationObserver: - ((.Module.Path // "") == $module_path) - } - ' "${package_data}" >"${relevant_package_data}" - elif [[ "${resolution}" == "observer-v1" ]]; then - target_import_path="${module_path}" - if [[ "${package}" != "." ]]; then - target_import_path="${module_path}/${package}" - fi - jq -s \ - --arg root "${root}/" \ - --arg target "${target_import_path}" ' - def canonical_import: - (.ImportPath // "" | sub(" \\[.*$"; "")); - [.[] | select( - canonical_import == $target or - (.ForTest // "") == $target - )] as $observers - | ([$observers[].Dir] | unique) as $observer_directories - | ([ - $observers[] - | canonical_import, (.Deps // [])[] - ] | unique) as $relevant_imports - | .[] - | select((.Dir // "") | startswith($root)) - | select( - canonical_import as $import - | ($relevant_imports | index($import)) != null - ) - | .Dir as $directory - | . + { - GolibMutationObserver: - (($observer_directories | index($directory)) != null) - } - ' "${package_data}" >"${relevant_package_data}" - else - target_import_path="${module_path}" - if [[ "${package}" != "." ]]; then - target_import_path="${module_path}/${package}" - fi - jq -s \ - --arg root "${root}/" \ - --arg target "${target_import_path}" ' - def canonical_import: - (.ImportPath // "" | sub(" \\[.*$"; "")); - [.[] | select( - (canonical_import == $target and - (.ForTest // "") == "") or - (.ForTest // "") == $target - )] as $observers - | ([$observers[].Dir] | unique) as $observer_directories - | ([ - $observers[] - | canonical_import, (.Deps // [])[] - ] | unique) as $relevant_imports - | .[] - | select((.Dir // "") | startswith($root)) - | select( - (.ForTest // "") == "" or - (.ForTest // "") == $target - ) - | select( - canonical_import as $import - | ($relevant_imports | index($import)) != null - ) - | .Dir as $directory - | . + { - GolibMutationObserver: - (($observer_directories | index($directory)) != null) - } - ' "${package_data}" >"${relevant_package_data}" - fi - - jq -r --arg root "${root}/" --arg module_path "${module_path}" ' - select(.Dir | startswith($root)) - | .Dir as $directory - | ( - [ - .GoFiles[]?, - .CgoFiles[]?, - .CFiles[]?, - .CXXFiles[]?, - .MFiles[]?, - .HFiles[]?, - .FFiles[]?, - .SFiles[]?, - .SwigFiles[]?, - .SwigCXXFiles[]?, - .SysoFiles[]?, - .EmbedFiles[]? - ] + - ( - if .GolibMutationObserver == true - then [ - .TestGoFiles[]?, - .XTestGoFiles[]?, - .TestEmbedFiles[]?, - .XTestEmbedFiles[]? - ] - else [] - end - ) - )[] - | if startswith("/") then . else "\($directory)/\(.)" end - | select(startswith($root)) - ' "${relevant_package_data}" >>"${input_files}" - - while IFS= read -r file; do - [[ -n "${file}" ]] || continue - append_mutation_module_manifest "${file}" - done < <(jq -r --arg root "${root}/" ' - select( - (.Module.GoMod // "") == $root or - ((.Module.GoMod // "") | startswith($root)) - ) - | .Module.GoMod - ' "${relevant_package_data}" | LC_ALL=C sort -u) - - while IFS= read -r package_directory; do - [[ -n "${package_directory}" ]] || continue - for data_name in corpus fixtures testdata; do - if [[ -d "${package_directory}/${data_name}" ]]; then - find "${package_directory}/${data_name}" -type f \ - -print >>"${input_files}" - fi - done - done < <( - jq -r --arg root "${root}/" ' - select(.Dir | startswith($root)) - | .Dir - ' "${relevant_package_data}" | LC_ALL=C sort -u - ) - - while IFS= read -r file; do - [[ -n "${file}" ]] || continue - append_file "${file}" - done < <(LC_ALL=C sort -u "${input_files}") -} - -if [[ "${gate}" == "mutation" && -z "${package}" ]]; then - mutation_digest -elif [[ "${gate}" == "mutation-legacy" ]]; then - legacy_digest -elif [[ "${gate}" == "mutation" ]]; then - package_digest -else - verification_digest -fi - -LC_ALL=C sort "${manifest}" | shasum -a 256 | awk '{print $1}' diff --git a/.golib/scripts/internal/configure-mutation-workers.sh b/.golib/scripts/internal/configure-mutation-workers.sh deleted file mode 100755 index 0e86726..0000000 --- a/.golib/scripts/internal/configure-mutation-workers.sh +++ /dev/null @@ -1,20 +0,0 @@ -#!/usr/bin/env bash - -configure_mutation_workers() { - local workers="$1" - local index - - if [[ ! "${workers}" =~ ^[1-9][0-9]*$ ]]; then - printf 'mutation workers must be a positive integer\n' >&2 - return 2 - fi - for index in "${!mutation_arguments[@]}"; do - if [[ "${mutation_arguments[${index}]}" == "--workers" ]]; then - mutation_arguments[index + 1]="${workers}" - return - fi - done - - printf 'mutation command is missing --workers\n' >&2 - return 2 -} diff --git a/.golib/scripts/internal/isolated-go.sh b/.golib/scripts/internal/isolated-go.sh deleted file mode 100755 index d84188f..0000000 --- a/.golib/scripts/internal/isolated-go.sh +++ /dev/null @@ -1,144 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -real_go="${GOLIB_REAL_GO:-}" -cache_root="${GOLIB_ISOLATED_MODFILES_DIRECTORY:-}" -if [[ -z "${real_go}" || -z "${cache_root}" ]]; then - printf 'isolated Go requires GOLIB_REAL_GO and ' >&2 - printf 'GOLIB_ISOLATED_MODFILES_DIRECTORY\n' >&2 - exit 2 -fi - -module_root="${PWD}" -while [[ ! -f "${module_root}/go.mod" ]]; do - parent="$(dirname "${module_root}")" - if [[ "${parent}" == "${module_root}" ]]; then - exec "${real_go}" "$@" - fi - module_root="${parent}" -done - -clean_flags="" -for flag in ${GOFLAGS:-}; do - case "${flag}" in - -mod=*|-modfile=*) ;; - *) clean_flags="${clean_flags:+${clean_flags} }${flag}" ;; - esac -done - -identity="$( - { - printf '%s\n' "${module_root}" - cksum "${module_root}/go.mod" - if [[ -f "${module_root}/go.sum" ]]; then - cksum "${module_root}/go.sum" - fi - } | cksum | awk '{print $1 "-" $2}' -)" -state="${cache_root}/${identity}" -modfile="${state}/isolated.mod" -sumfile="${state}/isolated.sum" -ready="${state}/ready" - -if [[ ! -f "${ready}" ]]; then - lock="${state}.lock" - if mkdir "${lock}" 2>/dev/null; then - mkdir -p "${state}" - cp "${module_root}/go.mod" "${modfile}" - if [[ -f "${module_root}/go.sum" ]]; then - awk '$1 !~ /^github\.com\/faustbrian\/golib\// { print }' \ - "${module_root}/go.sum" >"${sumfile}" - else - : >"${sumfile}" - fi - preparation_flags="${clean_flags:+${clean_flags} }-modfile=${modfile}" - if ! GOWORK=off GOFLAGS="${preparation_flags}" \ - "${real_go}" mod download all; then - rm -rf "${state}" "${lock}" - exit 1 - fi - : >"${ready}" - rmdir "${lock}" - else - while [[ ! -f "${ready}" ]]; do - sleep 0.05 - done - fi -fi - -if [[ "${1:-}" == "mod" && "${2:-}" == "tidy" && - " $* " == *" -diff "* ]]; then - tidy_arguments=() - for argument in "$@"; do - [[ "${argument}" == "-diff" ]] || tidy_arguments+=("${argument}") - done - tidy_flags="${clean_flags:+${clean_flags} }-modfile=${modfile}" - GOWORK=off GOFLAGS="${tidy_flags}" \ - "${real_go}" "${tidy_arguments[@]}" - - status=0 - if ! diff -u "${module_root}/go.mod" "${modfile}"; then - status=1 - fi - source_sum="${state}/source-external.sum" - tidy_sum="${state}/tidy-external.sum" - if [[ -f "${module_root}/go.sum" ]]; then - awk '$1 !~ /^github\.com\/faustbrian\/golib\// { print }' \ - "${module_root}/go.sum" >"${source_sum}" - else - : >"${source_sum}" - fi - awk '$1 !~ /^github\.com\/faustbrian\/golib\// { print }' \ - "${sumfile}" >"${tidy_sum}" - if ! diff -u "${source_sum}" "${tidy_sum}"; then - status=1 - fi - exit "${status}" -fi - -case "${1:-}" in - exec-tool) - shift - command_flags="${clean_flags:+${clean_flags} }-modfile=${modfile} -mod=readonly" - GOLIB_ISOLATED_MODFILE="${modfile}" \ - GOWORK=off GOFLAGS="${command_flags}" \ - PATH="$(dirname "${real_go}"):${PATH}" exec "$@" - ;; - run|install) - command="$1" - shift - if [[ " $* " == *"@"* ]]; then - GOWORK=off GOFLAGS="${clean_flags}" \ - "${real_go}" "${command}" "$@" - else - GOLIB_ISOLATED_MODFILE="${modfile}" \ - GOWORK=off GOFLAGS="${clean_flags}" \ - "${real_go}" "${command}" \ - "-modfile=${modfile}" -mod=readonly "$@" - fi - ;; - get) - command_flags="${clean_flags:+${clean_flags} }-modfile=${modfile} -mod=mod" - GOWORK=off GOFLAGS="${command_flags}" "${real_go}" "$@" - ;; - build|clean|fix|fmt|generate|list|test|vet) - command="$1" - shift - GOLIB_ISOLATED_MODFILE="${modfile}" \ - GOWORK=off GOFLAGS="${clean_flags}" \ - "${real_go}" "${command}" \ - "-modfile=${modfile}" -mod=readonly "$@" - ;; - doc) - command_flags="${clean_flags:+${clean_flags} }-modfile=${modfile} -mod=readonly" - GOLIB_ISOLATED_MODFILE="${modfile}" \ - GOWORK=off GOFLAGS="${command_flags}" "${real_go}" "$@" - ;; - mod) - command_flags="${clean_flags:+${clean_flags} }-modfile=${modfile}" - GOWORK=off GOFLAGS="${command_flags}" "${real_go}" "$@" - ;; - *) - GOWORK=off GOFLAGS="${clean_flags}" "${real_go}" "$@" - ;; -esac diff --git a/.golib/scripts/internal/migrate-api-baseline-evidence.sh b/.golib/scripts/internal/migrate-api-baseline-evidence.sh deleted file mode 100755 index 081483a..0000000 --- a/.golib/scripts/internal/migrate-api-baseline-evidence.sh +++ /dev/null @@ -1,165 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 1 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -module="$1" -artifact="${root}/.artifacts/${module}/evidence" -validated_revision="$(git rev-parse HEAD)" -validated_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" -lock="" -lock_acquired=0 -migration_lock="${artifact}/.locks/api-baseline-evidence-migration.lock" -migration_lock_acquired=0 -temporary_evidence="" -temporary_log="" -temporary_legacy_evidence="" -temporary_legacy_log="" - -cleanup_gate() { - rm -f \ - "${temporary_evidence}" "${temporary_log}" \ - "${temporary_legacy_evidence}" "${temporary_legacy_log}" - if [[ "${lock_acquired}" -eq 1 ]]; then - rm -f "${lock}/owner" - rmdir "${lock}" 2>/dev/null || true - fi - lock_acquired=0 - temporary_evidence="" - temporary_log="" - temporary_legacy_evidence="" - temporary_legacy_log="" -} - -cleanup() { - cleanup_gate - if [[ "${migration_lock_acquired}" -eq 1 ]]; then - rm -f "${migration_lock}/owner" - rmdir "${migration_lock}" 2>/dev/null || true - fi -} -trap cleanup EXIT HUP INT TERM - -[[ -f "${root}/${module}/api/baseline.txt" ]] || { - printf 'module has no API baseline: %s\n' "${module}" >&2 - exit 1 -} - -mkdir -p "${artifact}/.locks" -while ! mkdir "${migration_lock}" 2>/dev/null; do - if [[ -f "${migration_lock}/owner" ]]; then - owner="$(cat "${migration_lock}/owner" 2>/dev/null || true)" - if [[ "${owner}" =~ ^[0-9]+$ ]] && ! kill -0 "${owner}" 2>/dev/null; then - rm -f "${migration_lock}/owner" - rmdir "${migration_lock}" 2>/dev/null || true - continue - fi - fi - sleep 0.05 -done -printf '%s\n' "$$" >"${migration_lock}/owner" -migration_lock_acquired=1 - -while IFS= read -r gate; do - [[ -n "${gate}" ]] || continue - source_evidence="${artifact}/${gate}.json" - source_log="${artifact}/${gate}.log" - [[ -f "${source_evidence}" && -f "${source_log}" ]] || continue - - lock_root="${artifact}/.locks" - lock="${lock_root}/${gate}.lock" - mkdir -p "${lock_root}" - while ! mkdir "${lock}" 2>/dev/null; do - if [[ -f "${lock}/owner" ]]; then - owner="$(cat "${lock}/owner" 2>/dev/null || true)" - if [[ "${owner}" =~ ^[0-9]+$ ]] && ! kill -0 "${owner}" 2>/dev/null; then - rm -f "${lock}/owner" - rmdir "${lock}" 2>/dev/null || true - continue - fi - fi - sleep 0.05 - done - printf '%s\n' "$$" >"${lock}/owner" - lock_acquired=1 - - current_digest="$( - "${root}/.golib/scripts/gate-input-digest.sh" "${gate}" "${module}" - )" - source_digest="$(jq -r '.input_digest // empty' "${source_evidence}")" - if [[ "${source_digest}" == "${current_digest}" ]]; then - cleanup_gate - continue - fi - - log_sha256="$(shasum -a 256 "${source_log}" | awk '{print $1}')" - jq -e \ - --arg module "${module}" \ - --arg gate "${gate}" \ - --arg source_digest "${source_digest}" \ - --arg log_sha256 "${log_sha256}" ' - .schema_version == 1 and - .module == $module and - .gate == $gate and - (.result == "passed" or .result == "not_applicable" or - ($gate == "nilaway" and .result == "advisory")) and - .exit_code == 0 and - ($source_digest | length) > 0 and - .input_digest == $source_digest and - .completed_input_digest == $source_digest and - .log_sha256 == $log_sha256 - ' "${source_evidence}" >/dev/null || { - cleanup_gate - continue - } - - legacy_digest="$( - GOLIB_GATE_INPUT_POLICY=legacy-api-baseline \ - "${root}/.golib/scripts/gate-input-digest.sh" "${gate}" "${module}" - )" - if [[ "${source_digest}" != "${legacy_digest}" ]]; then - cleanup_gate - continue - fi - - destination="${artifact}/by-input/${gate}" - destination_evidence="${destination}/${current_digest}.json" - destination_log="${destination}/${current_digest}.log" - temporary_evidence="${destination_evidence}.tmp.$$" - temporary_log="${destination_log}.tmp.$$" - temporary_legacy_evidence="${source_evidence}.tmp.$$" - temporary_legacy_log="${source_log}.tmp.$$" - mkdir -p "${destination}" - jq \ - --arg current_digest "${current_digest}" \ - --arg source_digest "${source_digest}" \ - --arg validated_revision "${validated_revision}" \ - --arg validated_at "${validated_at}" ' - .identity_lineage = (( - (.identity_lineage // []) + - [.input_digest] + - [(.identity_migration.previous_gate_input_digest // "")] - ) | map(select(length > 0)) | unique) - | .input_digest = $current_digest - | .completed_input_digest = $current_digest - | .revalidated_revision = $validated_revision - | .revalidated_at = $validated_at - | .identity_migration = { - reason: "non-semantic-gate-input-scope-narrowing", - previous_gate_input_digest: $source_digest - } - ' "${source_evidence}" >"${temporary_evidence}" - cp "${source_log}" "${temporary_log}" - mv "${temporary_log}" "${destination_log}" - mv "${temporary_evidence}" "${destination_evidence}" - cp "${destination_log}" "${temporary_legacy_log}" - cp "${destination_evidence}" "${temporary_legacy_evidence}" - mv "${temporary_legacy_log}" "${source_log}" - mv "${temporary_legacy_evidence}" "${source_evidence}" - cleanup_gate - printf '[%s] migrated content-identical %s evidence\n' "${module}" "${gate}" -done <"${root}/.golib/scripts/check-gates.txt" diff --git a/.golib/scripts/internal/migrate-runner-isolation-evidence.sh b/.golib/scripts/internal/migrate-runner-isolation-evidence.sh deleted file mode 100755 index f883567..0000000 --- a/.golib/scripts/internal/migrate-runner-isolation-evidence.sh +++ /dev/null @@ -1,168 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -lt 1 || $# -gt 2 ]]; then - printf 'usage: %s [gate]\n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -module="$1" -selected_gate="${2:-}" -artifact="${root}/.artifacts/${module}/evidence" -validated_revision="$(git rev-parse HEAD)" -validated_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" - -if [[ -n "${selected_gate}" ]]; then - gates="${selected_gate}" -else - gates="$(cat "${root}/.golib/scripts/check-gates.txt")" -fi - -while IFS= read -r gate; do - [[ -n "${gate}" ]] || continue - # Aggregate assurance intentionally fingerprints runner implementation. - # Process-isolation changes therefore invalidate rather than migrate it. - [[ "${gate}" != "operational-assurance" ]] || continue - current_digest="$( - "${root}/.golib/scripts/gate-input-digest.sh" "${gate}" "${module}" - )" - destination="${artifact}/by-input/${gate}" - destination_evidence="${destination}/${current_digest}.json" - destination_log="${destination}/${current_digest}.log" - if [[ -f "${destination_evidence}" && -f "${destination_log}" ]]; then - continue - fi - source_evidence="" - source_log="" - source_digest="" - for legacy_policy in \ - legacy-runner-isolation-post \ - legacy-runner-isolation; do - legacy_digest="$( - GOLIB_GATE_INPUT_POLICY="${legacy_policy}" \ - "${root}/.golib/scripts/gate-input-digest.sh" "${gate}" "${module}" - )" - [[ "${current_digest}" != "${legacy_digest}" ]] || continue - candidate_evidence="${artifact}/by-input/${gate}/${legacy_digest}.json" - candidate_log="${artifact}/by-input/${gate}/${legacy_digest}.log" - if [[ ! -f "${candidate_evidence}" || ! -f "${candidate_log}" ]]; then - candidate_evidence="${artifact}/${gate}.json" - candidate_log="${artifact}/${gate}.log" - fi - [[ -f "${candidate_evidence}" && -f "${candidate_log}" ]] || continue - candidate_digest="$(jq -r '.input_digest // empty' "${candidate_evidence}")" - [[ "${candidate_digest}" == "${legacy_digest}" ]] || continue - source_evidence="${candidate_evidence}" - source_log="${candidate_log}" - source_digest="${candidate_digest}" - break - done - [[ -n "${source_evidence}" ]] || continue - log_sha256="$(shasum -a 256 "${source_log}" | awk '{print $1}')" - jq -e \ - --arg module "${module}" \ - --arg gate "${gate}" \ - --arg source_digest "${source_digest}" \ - --arg log_sha256 "${log_sha256}" ' - .schema_version == 1 and - .module == $module and - .gate == $gate and - ( - .result == "passed" or - .result == "not_applicable" or - ($gate == "nilaway" and .result == "advisory") - ) and - .exit_code == 0 and - .input_digest == $source_digest and - .completed_input_digest == $source_digest and - .log_sha256 == $log_sha256 - ' "${source_evidence}" >/dev/null || continue - if [[ "$(jq -r '.result' "${source_evidence}")" == "advisory" ]] && - ! grep -Eq 'NilAway advisory exit status: [1-9][0-9]*$' "${source_log}"; then - continue - fi - if [[ "$(jq -r '.result' "${source_evidence}")" == "not_applicable" ]] && - ! grep -Fq ': not applicable by catalog policy' "${source_log}"; then - continue - fi - - lock_root="${artifact}/.locks" - lock="${lock_root}/${gate}.lock" - mkdir -p "${destination}" "${lock_root}" - while ! ln -s "$$" "${lock}" 2>/dev/null; do - if [[ -L "${lock}" ]]; then - owner="$(readlink "${lock}" 2>/dev/null || true)" - if [[ "${owner}" =~ ^[0-9]+$ ]] && ! kill -0 "${owner}" 2>/dev/null; then - rm -f "${lock}" - continue - fi - elif [[ -d "${lock}" ]]; then - owner="$(cat "${lock}/owner" 2>/dev/null || true)" - if [[ "${owner}" =~ ^[0-9]+$ ]] && ! kill -0 "${owner}" 2>/dev/null; then - rm -f "${lock}/owner" - rmdir "${lock}" 2>/dev/null || true - continue - fi - if [[ -z "${owner}" ]] && rmdir "${lock}" 2>/dev/null; then - continue - fi - elif [[ -f "${lock}" ]]; then - owner="$(cat "${lock}" 2>/dev/null || true)" - if [[ "${owner}" =~ ^[0-9]+$ ]] && ! kill -0 "${owner}" 2>/dev/null; then - rm -f "${lock}" - continue - fi - if [[ -z "${owner}" ]]; then - rm -f "${lock}" - continue - fi - fi - sleep 0.05 - done - - temporary_evidence="${destination_evidence}.tmp.$$" - temporary_log="${destination_log}.tmp.$$" - temporary_pointer_evidence="${artifact}/${gate}.json.tmp.$$" - temporary_pointer_log="${artifact}/${gate}.log.tmp.$$" - cleanup_gate() { - rm -f \ - "${temporary_evidence}" "${temporary_log}" \ - "${temporary_pointer_evidence}" "${temporary_pointer_log}" - if [[ "$(readlink "${lock}" 2>/dev/null || true)" == "$$" ]]; then - rm -f "${lock}" - fi - } - trap cleanup_gate EXIT - trap 'exit 130' HUP INT TERM - - jq \ - --arg current_digest "${current_digest}" \ - --arg source_digest "${source_digest}" \ - --arg validated_revision "${validated_revision}" \ - --arg validated_at "${validated_at}" ' - .identity_lineage = (( - (.identity_lineage // []) + - [.input_digest] + - [(.identity_migration.previous_gate_input_digest // "")] - ) | map(select(length > 0)) | unique) - | .input_digest = $current_digest - | .completed_input_digest = $current_digest - | .revalidated_revision = $validated_revision - | .revalidated_at = $validated_at - | .identity_migration = { - reason: "non-semantic-runner-isolation-scope-narrowing", - previous_gate_input_digest: $source_digest - } - ' "${source_evidence}" >"${temporary_evidence}" - cp "${source_log}" "${temporary_log}" - mv "${temporary_log}" "${destination_log}" - mv "${temporary_evidence}" "${destination_evidence}" - cp "${destination_log}" "${temporary_pointer_log}" - cp "${destination_evidence}" "${temporary_pointer_evidence}" - mv "${temporary_pointer_log}" "${artifact}/${gate}.log" - mv "${temporary_pointer_evidence}" "${artifact}/${gate}.json" - cleanup_gate - trap - EXIT HUP INT TERM - printf '[%s] migrated content-identical %s evidence\n' "${module}" "${gate}" -done <<<"${gates}" diff --git a/.golib/scripts/internal/mutation-command.sh b/.golib/scripts/internal/mutation-command.sh deleted file mode 100755 index 0068cb2..0000000 --- a/.golib/scripts/internal/mutation-command.sh +++ /dev/null @@ -1,27 +0,0 @@ -#!/usr/bin/env bash - -build_mutation_arguments() { - local target="$1" - local output="$2" - local tags="$3" - local discover_only="$4" - - mutation_arguments=( - unleash "${target}" - --integration --coverpkg "${target}" - --exclude-files '^.+/' - --workers 4 --test-cpu 1 --timeout-coefficient 10 - --threshold-efficacy 100 --threshold-mcover 100 - --arithmetic-base --conditionals-boundary --conditionals-negation - --invert-assignments --invert-bitwise --invert-bwassign - --increment-decrement --invert-logical --invert-loopctrl - --invert-negatives --remove-self-assignments - --output-statuses lctvsr --output "${output}" - ) - if [[ -n "${tags}" ]]; then - mutation_arguments+=(--tags "${tags}") - fi - if [[ "${discover_only}" -eq 1 ]]; then - mutation_arguments+=(--dry-run) - fi -} diff --git a/.golib/scripts/internal/mutation-coverage.sh b/.golib/scripts/internal/mutation-coverage.sh deleted file mode 100755 index ab02715..0000000 --- a/.golib/scripts/internal/mutation-coverage.sh +++ /dev/null @@ -1,31 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 2 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -profile="$1" -tags="$2" -started="$(date +%s)" -arguments=( - test - -count=1 - -timeout=20m - -cover - -coverpkg=./... - "-coverprofile=${profile}" -) -if [[ -n "${tags}" ]]; then - arguments+=("-tags=${tags}") -fi -arguments+=(./...) - -go "${arguments[@]}" >&2 -finished="$(date +%s)" -elapsed=$((finished - started)) -if [[ "${elapsed}" -lt 1 ]]; then - elapsed=1 -fi -printf '%ss\n' "${elapsed}" diff --git a/.golib/scripts/internal/mutation-scratch.sh b/.golib/scripts/internal/mutation-scratch.sh deleted file mode 100644 index e2dce95..0000000 --- a/.golib/scripts/internal/mutation-scratch.sh +++ /dev/null @@ -1,169 +0,0 @@ -#!/usr/bin/env bash - -mutation_scratch_process_start() { - LC_ALL=C ps -o lstart= -p "$1" 2>/dev/null | awk '{$1=$1; print}' -} - -mutation_scratch_owner_is_abandoned() { - local candidate="$1" - local expected_host="$2" - local owner_file owner_version owner_host owner_pid owner_start owner_run_id - local current_start - owner_file="${candidate}/.mutation-owner" - [[ -f "${owner_file}" && ! -L "${owner_file}" ]] || return 1 - IFS=$'\t' read -r owner_version owner_host owner_pid owner_start \ - owner_run_id <"${owner_file}" || return 1 - [[ "${owner_version}" == "1" && - "${owner_host}" == "${expected_host}" && - "${owner_pid}" =~ ^[1-9][0-9]*$ && - "${owner_start}" != "" && - "${owner_run_id}" == "$(basename "${candidate}")" ]] || return 1 - current_start="$(mutation_scratch_process_start "${owner_pid}")" - if [[ -n "${current_start}" ]]; then - [[ "${current_start}" != "${owner_start}" ]] - return - fi - if kill -0 "${owner_pid}" 2>/dev/null; then - return 1 - fi - return 0 -} - -mutation_scratch_recover_abandoned() { - local recovery_artifact="$1" - local recovery_host candidate claim - [[ -d "${recovery_artifact}" && ! -L "${recovery_artifact}" ]] || return 0 - recovery_host="$(hostname)" - while IFS= read -r candidate; do - case "${candidate}" in - "${recovery_artifact}"/mutation-run-*) ;; - *) continue ;; - esac - [[ "$(dirname "${candidate}")" == "${recovery_artifact}" && - -d "${candidate}" && ! -L "${candidate}" ]] || continue - mutation_scratch_owner_is_abandoned \ - "${candidate}" "${recovery_host}" || continue - claim="${candidate}/.mutation-recovery-claim" - mkdir "${claim}" 2>/dev/null || continue - if mutation_scratch_owner_is_abandoned \ - "${candidate}" "${recovery_host}"; then - find "${candidate}" -depth -delete - else - rmdir "${claim}" - fi - done < <( - find "${recovery_artifact}" -mindepth 1 -maxdepth 1 -type d \ - -name 'mutation-run-*' -print - ) -} - -mutation_scratch_remove_owned_run() { - local owner_file owner_version owner_host owner_pid owner_start owner_run_id - [[ -n "${artifact:-}" && -n "${run_directory:-}" ]] || return 0 - case "${run_directory}" in - "${artifact}"/mutation-run-*) ;; - *) - printf 'refusing to remove unexpected mutation run: %s\n' \ - "${run_directory}" >&2 - return 1 - ;; - esac - [[ "$(dirname "${run_directory}")" == "${artifact}" && - -d "${run_directory}" && ! -L "${run_directory}" ]] || return 0 - owner_file="${run_directory}/.mutation-owner" - if [[ -e "${owner_file}" || -L "${owner_file}" ]]; then - IFS=$'\t' read -r owner_version owner_host owner_pid owner_start \ - owner_run_id <"${owner_file}" || { - printf 'refusing to remove mutation run without a valid owner: %s\n' \ - "${run_directory}" >&2 - return 1 - } - if [[ "${owner_version}" != "1" || - "${owner_host}" != "${mutation_owner_host}" || - "${owner_pid}" != "$$" || - "${owner_start}" != "${mutation_owner_start}" || - "${owner_run_id}" != "$(basename "${run_directory}")" ]]; then - printf 'refusing to remove mutation run with a different owner: %s\n' \ - "${run_directory}" >&2 - return 1 - fi - elif [[ "${mutation_owner_run_id:-}" != \ - "$(basename "${run_directory}")" ]]; then - printf 'refusing to remove unmarked mutation run: %s\n' \ - "${run_directory}" >&2 - return 1 - fi - find "${run_directory}" -depth -delete - run_directory="" -} - -mutation_scratch_package_cache() { - local slug="$1" - case "${slug}" in - ""|"."|".."|*/*) - printf 'invalid mutation package cache slug: %s\n' "${slug}" >&2 - return 1 - ;; - esac - active_build_cache="$( - mktemp -d "${run_directory}/${slug}.go-cache-XXXXXXXX" - )" -} - -mutation_scratch_cleanup_package_cache() { - [[ -n "${active_build_cache:-}" ]] || return 0 - case "${active_build_cache}" in - "${run_directory}"/*.go-cache-*) ;; - *) - printf 'refusing to remove unexpected mutation cache: %s\n' \ - "${active_build_cache}" >&2 - return 1 - ;; - esac - [[ "$(dirname "${active_build_cache}")" == "${run_directory}" && - -d "${active_build_cache}" && - ! -L "${active_build_cache}" ]] || return 0 - find "${active_build_cache}" -depth -delete - active_build_cache="" -} - -mutation_scratch_on_exit() { - local status=$? - trap - EXIT HUP INT TERM - if ! mutation_scratch_remove_owned_run && [[ "${status}" -eq 0 ]]; then - status=1 - fi - exit "${status}" -} - -mutation_scratch_on_signal() { - exit "$1" -} - -mutation_scratch_install_traps() { - trap mutation_scratch_on_exit EXIT - trap 'mutation_scratch_on_signal 129' HUP - trap 'mutation_scratch_on_signal 130' INT - trap 'mutation_scratch_on_signal 143' TERM -} - -mutation_scratch_initialize() { - local owner_tmp - artifact="$1" - mkdir -p "${artifact}" - mutation_scratch_recover_abandoned "${artifact}" - run_directory="$(mktemp -d "${artifact}/mutation-run-XXXXXXXX")" - mutation_owner_run_id="$(basename "${run_directory}")" - mutation_scratch_install_traps - mutation_owner_host="$(hostname)" - mutation_owner_start="$(mutation_scratch_process_start "$$")" - [[ -n "${mutation_owner_start}" ]] || { - printf 'cannot identify mutation scratch owner process\n' >&2 - return 1 - } - owner_tmp="$(mktemp "${run_directory}/.mutation-owner.XXXXXXXX")" - printf '1\t%s\t%s\t%s\t%s\n' \ - "${mutation_owner_host}" "$$" "${mutation_owner_start}" \ - "$(basename "${run_directory}")" >"${owner_tmp}" - mv "${owner_tmp}" "${run_directory}/.mutation-owner" -} diff --git a/.golib/scripts/internal/optional-mutation-digest.sh b/.golib/scripts/internal/optional-mutation-digest.sh deleted file mode 100755 index 6e71fa3..0000000 --- a/.golib/scripts/internal/optional-mutation-digest.sh +++ /dev/null @@ -1,23 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 3 ]]; then - printf 'usage: %s \n' \ - "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -resolution="$1" -module="$2" -package="$3" -digest="" -if ! digest="$( - GOLIB_MUTATION_DIGEST_RESOLUTION="${resolution}" \ - "${root}/.golib/scripts/gate-input-digest.sh" \ - mutation "${module}" "${package}" 2>/dev/null -)"; then - exit 0 -fi - -printf '%s\n' "${digest}" diff --git a/.golib/scripts/internal/reuse-approved-mutation-checkpoint.sh b/.golib/scripts/internal/reuse-approved-mutation-checkpoint.sh deleted file mode 100755 index 7bd7437..0000000 --- a/.golib/scripts/internal/reuse-approved-mutation-checkpoint.sh +++ /dev/null @@ -1,118 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 9 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -ledger="$1" -checkpoint="$2" -module="$3" -package="$4" -current_input="$5" -tool_version="$6" -verifier_identity="$7" -validated_revision="$8" -output="$9" - -[[ -s "${ledger}" && -s "${checkpoint}" ]] || exit 1 - -report_digest="$({ jq -S -c '.report' "${checkpoint}" || exit 1; } | - shasum -a 256 | awk '{print $1}')" -execution_revision="$(jq -er '.execution_revision' "${checkpoint}")" -previous_input="$(jq -er '.gate_input_digest' "${checkpoint}")" -checkpoint_inputs="$(jq -c ' - ([.gate_input_digest] + - (.identity_lineage // []) + - [(.identity_migration.previous_gate_input_digest // "")]) - | map(select(length > 0)) - | unique -' "${checkpoint}")" - -jq -e \ - --arg module "${module}" \ - --arg package "${package}" \ - --arg execution_revision "${execution_revision}" \ - --arg tool_version "${tool_version}" \ - --arg verifier_identity "${verifier_identity}" \ - --arg report_digest "${report_digest}" \ - --argjson checkpoint_inputs "${checkpoint_inputs}" ' - .schema_version == 3 and - ([.verifier_migrations[]? | select( - .module == $module and - .package == $package and - .gremlins_verifier_sha256 == $verifier_identity and - .execution_revision == $execution_revision and - (.gate_input_digest as $approved_input | - ($checkpoint_inputs | index($approved_input)) != null) and - .gremlins_version == $tool_version and - .report_sha256 == $report_digest - )] | length == 1) - ' "${ledger}" >/dev/null - -jq -e \ - --arg module "${module}" \ - --arg package "${package}" \ - --arg tool_version "${tool_version}" \ - --arg verifier_identity "${verifier_identity}" ' - .schema_version == 3 and - .module == $module and - .package == $package and - .gremlins_version == $tool_version and - ((.gremlins_verifier_sha256 // $verifier_identity) == $verifier_identity) and - (.report.files | type == "array") and - ([.report.files[].mutations[]? | select(.status != "KILLED")] | length == 0) - ' "${checkpoint}" >/dev/null - -if [[ "${previous_input}" == "${current_input}" ]]; then - jq \ - --arg validated_revision "${validated_revision}" \ - --arg verifier_identity "${verifier_identity}" ' - .validated_revision = $validated_revision - | .gremlins_verifier_sha256 = $verifier_identity - | .verifier_identity_source = "approved-semantic-migration" - ' "${checkpoint}" >"${output}" - exit 0 -fi - -jq -e \ - --arg module "${module}" \ - --arg package "${package}" \ - --arg execution_revision "${execution_revision}" \ - --arg previous_input "${previous_input}" \ - --arg current_input "${current_input}" \ - --arg tool_version "${tool_version}" \ - --arg verifier_identity "${verifier_identity}" \ - --arg report_digest "${report_digest}" ' - .schema_version == 3 and - ([.entries[] | select( - .module == $module and - .package == $package and - .execution_revision == $execution_revision and - .gate_input_digest == $previous_input and - .replacement_gate_input_digest == $current_input and - .gremlins_version == $tool_version and - ((.gremlins_verifier_sha256 // $verifier_identity) == $verifier_identity) and - .report_sha256 == $report_digest - )] | length == 1) - ' "${ledger}" >/dev/null - -jq \ - --arg validated_revision "${validated_revision}" \ - --arg current_input "${current_input}" \ - --arg verifier_identity "${verifier_identity}" ' - .identity_lineage = (( - (.identity_lineage // []) + - [.gate_input_digest] + - [(.identity_migration.previous_gate_input_digest // "")] - ) | map(select(length > 0)) | unique) - | .validated_revision = $validated_revision - | .gremlins_verifier_sha256 = $verifier_identity - | .verifier_identity_source = "approved-semantic-migration" - | .identity_migration = { - reason: "approved-input-identity-migration", - previous_gate_input_digest: .gate_input_digest - } - | .gate_input_digest = $current_input - ' "${checkpoint}" >"${output}" diff --git a/.golib/scripts/internal/reuse-mutation-coverage.sh b/.golib/scripts/internal/reuse-mutation-coverage.sh deleted file mode 100755 index c1a0c83..0000000 --- a/.golib/scripts/internal/reuse-mutation-coverage.sh +++ /dev/null @@ -1,42 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 5 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -profile="$1" -identity="$2" -destination="$3" -expected_input_digest="$4" -expected_test_tags="$5" - -[[ -s "${profile}" && -s "${identity}" ]] || exit 1 - -profile_sha256="$(shasum -a 256 "${profile}" | awk '{print $1}')" -elapsed="$(jq -er \ - --arg input_digest "${expected_input_digest}" \ - --arg test_tags "${expected_test_tags}" \ - --arg profile_sha256 "${profile_sha256}" ' - select( - .schema_version == 1 and - .input_digest == $input_digest and - .test_tags == $test_tags and - .profile_sha256 == $profile_sha256 and - (.elapsed | type == "string") and - (.elapsed | test("^[1-9][0-9]*s$")) - ) - | .elapsed - ' "${identity}")" || exit 1 - -temporary="$(mktemp "${destination}.tmp.XXXXXX")" -cleanup() { - rm -f "${temporary}" -} -trap cleanup EXIT HUP INT TERM -cp "${profile}" "${temporary}" -copied_sha256="$(shasum -a 256 "${temporary}" | awk '{print $1}')" -[[ "${copied_sha256}" == "${profile_sha256}" ]] || exit 1 -mv "${temporary}" "${destination}" -printf '%s\n' "${elapsed}" diff --git a/.golib/scripts/internal/run-mutation.sh b/.golib/scripts/internal/run-mutation.sh deleted file mode 100755 index ac46b58..0000000 --- a/.golib/scripts/internal/run-mutation.sh +++ /dev/null @@ -1,686 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 2 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -mode="$1" -module="$2" -if [[ -n "${GOLIB_REAL_GO:-}" ]]; then - PATH="$(dirname "${GOLIB_REAL_GO}"):${PATH}" - export PATH -fi -if [[ "${mode}" != "enforce" && "${mode}" != "discover" ]]; then - printf 'mutation mode must be enforce or discover\n' >&2 - exit 2 -fi -discover_only=0 -if [[ "${mode}" == "discover" ]]; then - discover_only=1 -fi -directory="${root}/${module}" -artifact="${root}/.artifacts/${module}" -report="${artifact}/mutation.json" -checkpoint_directory="${artifact}/mutation-checkpoints" -history_migrations="${root}/.golib/mutation-history-migrations.json" -mkdir -p "${checkpoint_directory}" -active_build_cache="" -# shellcheck disable=SC1091 -source "${root}/.golib/scripts/internal/mutation-scratch.sh" -mutation_scratch_initialize "${artifact}" -mutation_arguments=() -execution_revision="$(git -C "${root}" rev-parse HEAD)" - -# shellcheck disable=SC1091 -source "${root}/.golib/versions.env" -# shellcheck disable=SC1091 -source "${root}/.golib/scripts/internal/mutation-command.sh" -# shellcheck disable=SC1091 -source "${root}/.golib/scripts/internal/configure-mutation-workers.sh" -gremlins_binary="$("${root}/.golib/scripts/build-golib-gremlins.sh")" -gremlins_binary_sha256="$(shasum -a 256 "${gremlins_binary}" | awk '{print $1}')" -gremlins_verifier_sha256="$("${root}/.golib/scripts/mutation-verifier-identity.sh")" -environment_identity="$(go env -json GOVERSION GOOS GOARCH CGO_ENABLED)" -legacy_gate_input_digest="" -# run_directory is initialized by mutation_scratch_initialize. -# shellcheck disable=SC2154 -shared_coverage="${run_directory}/integration.coverage" -shared_coverage_elapsed="" -modfile="" -owned_count="$(jq -r --arg directory "${module}" ' - .modules[] | select(.directory == $directory) | .owned_dependencies | length -' "${root}/modules.json")" -if [[ "${owned_count}" -gt 0 ]]; then - modfile="${run_directory}/golib-mutation.mod" - cp "${directory}/go.mod" "${modfile}" - if [[ -f "${directory}/go.sum" ]]; then - cp "${directory}/go.sum" "${modfile%.mod}.sum" - fi - while IFS=$'\t' read -r owned_path owned_directory; do - [[ -n "${owned_path}" && -n "${owned_directory}" ]] || continue - GOWORK=off go mod edit -modfile="${modfile}" \ - -replace="${owned_path}=${root}/${owned_directory}" - done < <(jq -r '.modules[] - | select(.directory != "." and .kind != "fixture") - | [.module_path, .directory] - | @tsv' "${root}/modules.json") -fi -cd "${directory}" - -tags="$(jq -r --arg directory "${module}" \ - '.modules[] | select(.directory == $directory) - | .test_tags | map(select(. != "interoperability")) | join(",")' \ - "${root}/modules.json")" -mutation_workers=4 -if grep -Eq \ - 'github\.com/testcontainers/testcontainers-go([/[:space:]])' \ - "${directory}/go.mod"; then - mutation_workers=1 -fi -mutation_environment=(env) -if [[ "${module}" == "pkg/ecma-regexp" ]]; then - mutation_environment+=( -u TEST262_ROOT) -fi -packages=() -while IFS= read -r package_directory; do - [[ -n "${package_directory}" ]] && packages+=("${package_directory}") -done < <(jq -r --arg directory "${module}" ' - .modules[] - | select(.directory == $directory) - | .packages[] - | select(.coverage_required == true) - | .directory -' "${root}/modules.json") -[[ "${#packages[@]}" -gt 0 ]] || { - printf 'mutation package selection is empty for %s\n' "${module}" >&2 - exit 1 -} - -expected_packages_file="${run_directory}/expected-packages.json" -printf '%s\n' "${packages[@]}" | jq -R -s \ - 'split("\n") | map(select(length > 0))' >"${expected_packages_file}" -reports=() - -if [[ "${discover_only}" -eq 0 ]]; then - # A new enforced campaign supersedes any older aggregate immediately. - rm -f "${report}" -fi - -ensure_shared_coverage() { - if [[ -s "${shared_coverage}" && -n "${shared_coverage_elapsed}" ]]; then - return - fi - coverage_profile="${artifact}/coverage.out" - coverage_identity="${artifact}/coverage-profile.json" - coverage_input_digest="$( - "${root}/.golib/scripts/gate-input-digest.sh" coverage "${module}" - )" - if shared_coverage_elapsed="$( - "${root}/.golib/scripts/internal/reuse-mutation-coverage.sh" \ - "${coverage_profile}" \ - "${coverage_identity}" \ - "${shared_coverage}" \ - "${coverage_input_digest}" \ - "${tags}" \ - 2>/dev/null - )"; then - printf '[%s] reused content-bound coverage for mutation\n' "${module}" - return - fi - shared_coverage_elapsed="" - if [[ -n "${modfile}" ]]; then - shared_coverage_elapsed="$( - GOWORK=off GOFLAGS="-modfile=${modfile} -mod=mod" \ - "${root}/.golib/scripts/internal/mutation-coverage.sh" \ - "${shared_coverage}" "${tags}" - )" - else - shared_coverage_elapsed="$( - GOWORK=off "${root}/.golib/scripts/internal/mutation-coverage.sh" \ - "${shared_coverage}" "${tags}" - )" - fi - [[ -s "${shared_coverage}" ]] || { - printf 'shared mutation coverage profile is missing for %s\n' \ - "${module}" >&2 - exit 1 - } -} - -write_aggregate() { - local aggregate_tmp validated_revision - aggregate_tmp="$(mktemp "${report}.tmp.XXXXXX")" - validated_revision="$(git -C "${root}" rev-parse HEAD)" - jq -s \ - --arg module "${module}" \ - --arg validated_revision "${validated_revision}" \ - --slurpfile expected "${expected_packages_file}" ' - { - schema_version: 3, - module: $module, - execution_revisions: ([.[].execution_revision] | unique), - validated_revision: $validated_revision, - gate_input_digests: ( - map({key: .package, value: .gate_input_digest}) - | from_entries - ), - gremlins_versions: ([.[].gremlins_version] | unique), - gremlins_verifier_sha256s: ([.[].gremlins_verifier_sha256] | unique), - verifier_identity_sources: ([.[].verifier_identity_source] | unique), - gremlins_binary_sha256s: ( - [.[].gremlins_binary_sha256 | select(. != null)] | unique - ), - environments: ([.[].environment] | unique), - expected_packages: $expected[0], - completed_packages: [.[].package], - complete: (length == ($expected[0] | length)), - packages: . - } - ' "${reports[@]}" >"${aggregate_tmp}" - mv "${aggregate_tmp}" "${report}" -} - -checkpoint_verifier_source_is_valid() { - jq -e ' - ( - .verifier_identity_source == "executed" and - (.gremlins_binary_sha256 | type == "string") and - (.gremlins_binary_sha256 | test("^[0-9a-f]{64}$")) - ) or ( - .verifier_identity_source == "approved-semantic-migration" and - ( - (.gremlins_binary_sha256 // null) == null or - ( - (.gremlins_binary_sha256 | type == "string") and - (.gremlins_binary_sha256 | test("^[0-9a-f]{64}$")) - ) - ) - ) - ' "$1" >/dev/null -} - -for package_directory in "${packages[@]}"; do - target="." - slug="root" - if [[ "${package_directory}" != "." ]]; then - target="./${package_directory}" - slug="${package_directory//\//-}" - fi - package_report="${run_directory}/${slug}.json" - checkpoint="${checkpoint_directory}/${slug}.json" - package_input_digest="$( - "${root}/.golib/scripts/gate-input-digest.sh" \ - mutation "${module}" "${package_directory}" - )" - build_mutation_arguments \ - "${target}" "${package_report}" "${tags}" "${discover_only}" - configure_mutation_workers "${mutation_workers}" - - reviewed_zero_mutant() { - local source_directory source_digest reviewed - source_directory="${module}" - if [[ "${package_directory}" != "." ]]; then - source_directory="${module}/${package_directory}" - fi - source_digest="$("${root}/.golib/scripts/package-source-digest.sh" \ - "${source_directory}")" - reviewed="$(jq --arg module "${module}" \ - --arg package "${package_directory}" \ - --arg digest "${source_digest}" \ - --arg version "${GREMLINS_VERSION}" \ - --arg tool_identity "${gremlins_verifier_sha256}" ' - [.packages[] | select( - .module_directory == $module and - .package_directory == $package and - .source_digest == $digest and - .gremlins_version == $version and - .gremlins_verifier_sha256 == $tool_identity - )] | length - ' "${root}/.golib/mutation-zero-inventory.json")" - [[ "${reviewed}" -eq 1 ]] - } - - if [[ "${discover_only}" -eq 0 && -s "${checkpoint}" ]] && - jq -e \ - --arg module "${module}" \ - --arg package "${package_directory}" \ - --arg digest "${package_input_digest}" \ - --arg version "${GREMLINS_VERSION}" \ - --arg tool_identity "${gremlins_verifier_sha256}" ' - .schema_version == 3 and - .module == $module and - .package == $package and - .gate_input_digest == $digest and - .gremlins_version == $version and - .gremlins_verifier_sha256 == $tool_identity and - (.report.files | type == "array") and - ([.report.files[].mutations[]? | select(.status != "KILLED")] | length == 0) - ' "${checkpoint}" >/dev/null && - checkpoint_verifier_source_is_valid "${checkpoint}"; then - checkpoint_total="$( - jq '[.report.files[].mutations[]?] | length' "${checkpoint}" - )" - if [[ "${checkpoint_total}" -gt 0 ]] || reviewed_zero_mutant; then - checkpoint_tmp="$(mktemp "${checkpoint}.tmp.XXXXXX")" - jq --arg revision "$(git -C "${root}" rev-parse HEAD)" \ - '.validated_revision = $revision' \ - "${checkpoint}" >"${checkpoint_tmp}" - mv "${checkpoint_tmp}" "${checkpoint}" - reports+=("${checkpoint}") - write_aggregate - printf '[%s] %s reused content-identical mutation evidence\n' \ - "${module}" "${target}" - continue - fi - fi - - observer_v1_package_digest="$( - "${root}/.golib/scripts/internal/optional-mutation-digest.sh" \ - observer-v1 "${module}" "${package_directory}" - )" - if [[ "${discover_only}" -eq 0 && - -n "${observer_v1_package_digest}" && -s "${checkpoint}" ]] && - jq -e \ - --arg module "${module}" \ - --arg package "${package_directory}" \ - --arg digest "${observer_v1_package_digest}" \ - --arg version "${GREMLINS_VERSION}" \ - --arg tool_identity "${gremlins_verifier_sha256}" ' - .schema_version == 3 and - .module == $module and - .package == $package and - (( - [.gate_input_digest] + - (.identity_lineage // []) + - [(.identity_migration.previous_gate_input_digest // "")] - ) | index($digest)) != null and - .gremlins_version == $version and - .gremlins_verifier_sha256 == $tool_identity and - (.report.files | type == "array") and - ([.report.files[].mutations[]? | select(.status != "KILLED")] | length == 0) - ' "${checkpoint}" >/dev/null && - checkpoint_verifier_source_is_valid "${checkpoint}"; then - checkpoint_total="$( - jq '[.report.files[].mutations[]?] | length' "${checkpoint}" - )" - if [[ "${checkpoint_total}" -gt 0 ]] || reviewed_zero_mutant; then - checkpoint_tmp="$(mktemp "${checkpoint}.tmp.XXXXXX")" - jq \ - --arg revision "$(git -C "${root}" rev-parse HEAD)" \ - --arg digest "${package_input_digest}" ' - .identity_lineage = (( - (.identity_lineage // []) + - [.gate_input_digest] + - [(.identity_migration.previous_gate_input_digest // "")] - ) | map(select(length > 0)) | unique) - | .validated_revision = $revision - | .identity_migration = { - reason: "dependency-test-isolation", - previous_gate_input_digest: .gate_input_digest - } - | .gate_input_digest = $digest - ' "${checkpoint}" >"${checkpoint_tmp}" - mv "${checkpoint_tmp}" "${checkpoint}" - reports+=("${checkpoint}") - write_aggregate - printf '[%s] %s migrated dependency-test-isolated mutation identity\n' \ - "${module}" "${target}" - continue - fi - fi - - legacy_stable_package_digest="$( - "${root}/.golib/scripts/internal/optional-mutation-digest.sh" \ - legacy-stable "${module}" "${package_directory}" - )" - if [[ "${discover_only}" -eq 0 && - -n "${legacy_stable_package_digest}" && -s "${checkpoint}" ]] && - jq -e \ - --arg module "${module}" \ - --arg package "${package_directory}" \ - --arg digest "${legacy_stable_package_digest}" \ - --arg version "${GREMLINS_VERSION}" \ - --arg tool_identity "${gremlins_verifier_sha256}" ' - .schema_version == 3 and - .module == $module and - .package == $package and - (( - [.gate_input_digest] + - (.identity_lineage // []) + - [(.identity_migration.previous_gate_input_digest // "")] - ) | index($digest)) != null and - .gremlins_version == $version and - .gremlins_verifier_sha256 == $tool_identity and - (.report.files | type == "array") and - ([.report.files[].mutations[]? | select(.status != "KILLED")] | length == 0) - ' "${checkpoint}" >/dev/null && - checkpoint_verifier_source_is_valid "${checkpoint}"; then - checkpoint_total="$( - jq '[.report.files[].mutations[]?] | length' "${checkpoint}" - )" - if [[ "${checkpoint_total}" -gt 0 ]] || reviewed_zero_mutant; then - checkpoint_tmp="$(mktemp "${checkpoint}.tmp.XXXXXX")" - jq \ - --arg revision "$(git -C "${root}" rev-parse HEAD)" \ - --arg digest "${package_input_digest}" ' - .identity_lineage = (( - (.identity_lineage // []) + - [.gate_input_digest] + - [(.identity_migration.previous_gate_input_digest // "")] - ) | map(select(length > 0)) | unique) - | .validated_revision = $revision - | .identity_migration = { - reason: "target-observer-resolution", - previous_gate_input_digest: .gate_input_digest - } - | .gate_input_digest = $digest - ' "${checkpoint}" >"${checkpoint_tmp}" - mv "${checkpoint_tmp}" "${checkpoint}" - reports+=("${checkpoint}") - write_aggregate - printf '[%s] %s migrated module-wide mutation identity\n' \ - "${module}" "${target}" - continue - fi - fi - - legacy_package_digest="$( - "${root}/.golib/scripts/internal/optional-mutation-digest.sh" \ - caller "${module}" "${package_directory}" - )" - if [[ "${discover_only}" -eq 0 && - -n "${legacy_package_digest}" && -s "${checkpoint}" ]] && - jq -e \ - --arg module "${module}" \ - --arg package "${package_directory}" \ - --arg digest "${legacy_package_digest}" \ - --arg version "${GREMLINS_VERSION}" \ - --arg tool_identity "${gremlins_verifier_sha256}" ' - .schema_version == 3 and - .module == $module and - .package == $package and - (( - [.gate_input_digest] + - (.identity_lineage // []) + - [(.identity_migration.previous_gate_input_digest // "")] - ) | index($digest)) != null and - .gremlins_version == $version and - .gremlins_verifier_sha256 == $tool_identity and - (.report.files | type == "array") and - ([.report.files[].mutations[]? | select(.status != "KILLED")] | length == 0) - ' "${checkpoint}" >/dev/null && - checkpoint_verifier_source_is_valid "${checkpoint}"; then - checkpoint_total="$( - jq '[.report.files[].mutations[]?] | length' "${checkpoint}" - )" - if [[ "${checkpoint_total}" -gt 0 ]] || reviewed_zero_mutant; then - checkpoint_tmp="$(mktemp "${checkpoint}.tmp.XXXXXX")" - jq \ - --arg revision "$(git -C "${root}" rev-parse HEAD)" \ - --arg digest "${package_input_digest}" ' - .identity_lineage = (( - (.identity_lineage // []) + - [.gate_input_digest] + - [(.identity_migration.previous_gate_input_digest // "")] - ) | map(select(length > 0)) | unique) - | .validated_revision = $revision - | .identity_migration = { - reason: "canonical-workspace-resolution", - previous_gate_input_digest: .gate_input_digest - } - | .gate_input_digest = $digest - ' "${checkpoint}" >"${checkpoint_tmp}" - mv "${checkpoint_tmp}" "${checkpoint}" - reports+=("${checkpoint}") - write_aggregate - printf '[%s] %s migrated caller-dependent mutation identity\n' \ - "${module}" "${target}" - continue - fi - fi - - if [[ "${discover_only}" -eq 0 && -s "${checkpoint}" ]] && - jq -e \ - --arg module "${module}" \ - --arg package "${package_directory}" \ - --arg version "${GREMLINS_VERSION}" \ - --arg tool_identity "${gremlins_verifier_sha256}" ' - .schema_version == 2 and - .module == $module and - .package == $package and - .gremlins_version == $version and - .gremlins_verifier_sha256 == $tool_identity and - (.report.files | type == "array") and - ([.report.files[].mutations[]? | select(.status != "KILLED")] | length == 0) - ' "${checkpoint}" >/dev/null && - checkpoint_verifier_source_is_valid "${checkpoint}"; then - if [[ -z "${legacy_gate_input_digest}" ]]; then - legacy_gate_input_digest="$( - "${root}/.golib/scripts/gate-input-digest.sh" \ - mutation-legacy "${module}" - )" - fi - if jq -e --arg digest "${legacy_gate_input_digest}" \ - '.gate_input_digest == $digest' "${checkpoint}" >/dev/null; then - checkpoint_total="$( - jq '[.report.files[].mutations[]?] | length' "${checkpoint}" - )" - if [[ "${checkpoint_total}" -gt 0 ]] || reviewed_zero_mutant; then - checkpoint_tmp="$(mktemp "${checkpoint}.tmp.XXXXXX")" - jq \ - --arg revision "$(git -C "${root}" rev-parse HEAD)" \ - --arg digest "${package_input_digest}" ' - .schema_version = 3 - | .validated_revision = $revision - | .legacy_module_gate_input_digest = .gate_input_digest - | .gate_input_digest = $digest - ' "${checkpoint}" >"${checkpoint_tmp}" - mv "${checkpoint_tmp}" "${checkpoint}" - reports+=("${checkpoint}") - write_aggregate - printf '[%s] %s migrated content-identical mutation evidence\n' \ - "${module}" "${target}" - continue - fi - fi - fi - - if [[ "${discover_only}" -eq 0 && -s "${checkpoint}" && - -s "${history_migrations}" ]]; then - checkpoint_tmp="$(mktemp "${checkpoint}.tmp.XXXXXX")" - if "${root}/.golib/scripts/internal/reuse-approved-mutation-checkpoint.sh" \ - "${history_migrations}" \ - "${checkpoint}" \ - "${module}" \ - "${package_directory}" \ - "${package_input_digest}" \ - "${GREMLINS_VERSION}" \ - "${gremlins_verifier_sha256}" \ - "$(git -C "${root}" rev-parse HEAD)" \ - "${checkpoint_tmp}"; then - checkpoint_total="$( - jq '[.report.files[].mutations[]?] | length' \ - "${checkpoint_tmp}" - )" - if [[ "${checkpoint_total}" -gt 0 ]] || reviewed_zero_mutant; then - mv "${checkpoint_tmp}" "${checkpoint}" - reports+=("${checkpoint}") - write_aggregate - printf '[%s] %s reused approved content-identical mutation evidence\n' \ - "${module}" "${target}" - continue - fi - fi - rm -f "${checkpoint_tmp}" - fi - - printf '[%s] mutation package %s\n' "${module}" "${target}" - mutation_scratch_package_cache "${slug}" - ensure_shared_coverage - status=0 - if [[ "${discover_only}" -eq 1 ]]; then - if [[ -n "${modfile}" ]]; then - "${mutation_environment[@]}" \ - GOCACHE="${active_build_cache}" GOWORK=off \ - GOLIB_GREMLINS_COVERAGE_PROFILE="${shared_coverage}" \ - GOLIB_GREMLINS_COVERAGE_ELAPSED="${shared_coverage_elapsed}" \ - GOFLAGS="-modfile=${modfile} -mod=mod" \ - "${gremlins_binary}" "${mutation_arguments[@]}" \ - >"${run_directory}/${slug}.log" 2>&1 || status=$? - else - "${mutation_environment[@]}" \ - GOCACHE="${active_build_cache}" GOWORK=off \ - GOLIB_GREMLINS_COVERAGE_PROFILE="${shared_coverage}" \ - GOLIB_GREMLINS_COVERAGE_ELAPSED="${shared_coverage_elapsed}" \ - "${gremlins_binary}" "${mutation_arguments[@]}" \ - >"${run_directory}/${slug}.log" 2>&1 || status=$? - fi - elif [[ -n "${modfile}" ]]; then - "${mutation_environment[@]}" \ - GOCACHE="${active_build_cache}" GOWORK=off \ - GOLIB_GREMLINS_COVERAGE_PROFILE="${shared_coverage}" \ - GOLIB_GREMLINS_COVERAGE_ELAPSED="${shared_coverage_elapsed}" \ - GOFLAGS="-modfile=${modfile} -mod=mod" \ - "${gremlins_binary}" "${mutation_arguments[@]}" || status=$? - else - "${mutation_environment[@]}" \ - GOCACHE="${active_build_cache}" GOWORK=off \ - GOLIB_GREMLINS_COVERAGE_PROFILE="${shared_coverage}" \ - GOLIB_GREMLINS_COVERAGE_ELAPSED="${shared_coverage_elapsed}" \ - "${gremlins_binary}" "${mutation_arguments[@]}" || status=$? - fi - mutation_scratch_cleanup_package_cache - if [[ "${status}" -ne 0 ]]; then - if [[ "${discover_only}" -eq 1 ]]; then - cat "${run_directory}/${slug}.log" >&2 - fi - printf 'mutation tool failed for %s %s (status %s)\n' \ - "${module}" "${target}" "${status}" >&2 - exit 1 - fi - if [[ "${discover_only}" -eq 1 && ! -s "${package_report}" ]]; then - source_directory="${module}" - if [[ "${package_directory}" != "." ]]; then - source_directory="${module}/${package_directory}" - fi - source_digest="$("${root}/.golib/scripts/package-source-digest.sh" \ - "${source_directory}")" - printf 'ZERO\t%s\t%s\t%s\n' \ - "${module}" "${package_directory}" "${source_digest}" - continue - fi - if [[ ! -s "${package_report}" ]]; then - if reviewed_zero_mutant; then - printf '{"files":[]}\n' >"${package_report}" - else - printf 'mutation report is missing for %s %s\n' \ - "${module}" "${target}" >&2 - exit 1 - fi - fi - if ! jq -e '.files | type == "array"' "${package_report}" >/dev/null; then - printf 'mutation report is malformed for %s %s\n' "${module}" "${target}" >&2 - exit 1 - fi - total="$(jq '[.files[].mutations[]] | length' "${package_report}")" - if [[ "${discover_only}" -eq 1 ]]; then - printf 'MUTANTS\t%s\t%s\t%s\n' \ - "${module}" "${package_directory}" "${total}" - continue - fi - if [[ "${total}" -eq 0 ]]; then - if ! reviewed_zero_mutant; then - printf 'mutation report unexpectedly contains no reviewed mutants for %s %s\n' \ - "${module}" "${target}" >&2 - exit 1 - fi - fi - unexpected="$(jq '[.files[].mutations[] | select(.status != "KILLED")] | length' \ - "${package_report}")" - if [[ "${status}" -ne 0 || "${unexpected}" -ne 0 ]]; then - jq -r '.files[] as $file | $file.mutations[] | select(.status != "KILLED") | "\($file.file_name):\(.line):\(.column) \(.type) \(.status)"' \ - "${package_report}" >&2 - printf 'mutation gate failed for %s %s (tool status %s)\n' \ - "${module}" "${target}" "${status}" >&2 - exit 1 - fi - current_input_digest="$( - "${root}/.golib/scripts/gate-input-digest.sh" \ - mutation "${module}" "${package_directory}" - )" - if [[ "${current_input_digest}" != "${package_input_digest}" ]]; then - printf 'mutation inputs changed while running %s %s\n' \ - "${module}" "${target}" >&2 - exit 1 - fi - checkpoint_tmp="$(mktemp "${checkpoint}.tmp.XXXXXX")" - jq \ - --arg module "${module}" \ - --arg package "${package_directory}" \ - --arg execution_revision "${execution_revision}" \ - --arg validated_revision "$(git -C "${root}" rev-parse HEAD)" \ - --arg gate_input_digest "${package_input_digest}" \ - --arg gremlins_version "${GREMLINS_VERSION}" \ - --arg gremlins_verifier_sha256 "${gremlins_verifier_sha256}" \ - --arg gremlins_binary_sha256 "${gremlins_binary_sha256}" \ - --argjson environment "${environment_identity}" ' - { - schema_version: 3, - module: $module, - package: $package, - execution_revision: $execution_revision, - validated_revision: $validated_revision, - gate_input_digest: $gate_input_digest, - gremlins_version: $gremlins_version, - gremlins_verifier_sha256: $gremlins_verifier_sha256, - verifier_identity_source: "executed", - gremlins_binary_sha256: $gremlins_binary_sha256, - environment: $environment, - report: . - } - ' "${package_report}" >"${checkpoint_tmp}" - mv "${checkpoint_tmp}" "${checkpoint}" - reports+=("${checkpoint}") - write_aggregate - if [[ "${total}" -eq 0 ]]; then - printf '[%s] %s has a hash-pinned zero-viable-mutant review\n' \ - "${module}" "${target}" - else - printf '[%s] %s killed %s/%s viable mutants\n' \ - "${module}" "${target}" "${total}" "${total}" - fi -done - -if [[ "${discover_only}" -eq 1 ]]; then - exit 0 -fi - -if ! jq -e \ - ' - .schema_version == 3 and - .complete == true and - (.gremlins_verifier_sha256s | length == 1) and - ([.packages[] | select(.schema_version != 3)] | length == 0) and - all(.packages[]; - ( - .verifier_identity_source == "executed" and - (.gremlins_binary_sha256 | type == "string") and - (.gremlins_binary_sha256 | test("^[0-9a-f]{64}$")) - ) or ( - .verifier_identity_source == "approved-semantic-migration" and - ( - (.gremlins_binary_sha256 // null) == null or - ( - (.gremlins_binary_sha256 | type == "string") and - (.gremlins_binary_sha256 | test("^[0-9a-f]{64}$")) - ) - ) - ) - ) - ' "${report}" >/dev/null; then - printf 'aggregate mutation report is incomplete for %s\n' "${module}" >&2 - exit 1 -fi diff --git a/.golib/scripts/internal/run-verification-snapshots.sh b/.golib/scripts/internal/run-verification-snapshots.sh deleted file mode 100755 index 59de60e..0000000 --- a/.golib/scripts/internal/run-verification-snapshots.sh +++ /dev/null @@ -1,137 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 4 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$1" -gate="$2" -jobs="$3" -selection="$4" -snapshot_parents=() -snapshot_pids=() -snapshot_process_groups=() -lane_files=() -snapshot_processes_complete=0 -cleanup_started=0 - -# shellcheck disable=SC2329 # Called from the EXIT cleanup trap. -terminate_snapshot_processes() { - local attempt group running - for group in "${snapshot_process_groups[@]}"; do - kill -TERM -- "-${group}" 2>/dev/null || true - done - attempt=0 - while [[ "${attempt}" -lt 100 ]]; do - running=0 - for group in "${snapshot_process_groups[@]}"; do - if kill -0 -- "-${group}" 2>/dev/null; then - running=1 - break - fi - done - [[ "${running}" -eq 1 ]] || break - sleep 0.05 - attempt=$((attempt + 1)) - done - for group in "${snapshot_process_groups[@]}"; do - if kill -0 -- "-${group}" 2>/dev/null; then - kill -KILL -- "-${group}" 2>/dev/null || true - fi - done -} - -# shellcheck disable=SC2329 # Invoked by the EXIT trap. -cleanup_snapshots() { - local status=$? lane_file parent pid - if [[ "${cleanup_started}" -eq 1 ]]; then - return "${status}" - fi - cleanup_started=1 - trap '' HUP INT TERM - if [[ "${snapshot_processes_complete}" -eq 0 ]]; then - terminate_snapshot_processes - fi - for pid in "${snapshot_pids[@]}"; do - wait "${pid}" 2>/dev/null || true - done - for parent in "${snapshot_parents[@]}"; do - if [[ -d "${parent}" ]]; then - find "${parent}" -depth -delete - fi - done - for lane_file in "${lane_files[@]}"; do - rm -f "${lane_file}" - done - return "${status}" -} - -trap cleanup_snapshots EXIT -trap 'exit 129' HUP -trap 'exit 130' INT -trap 'exit 143' TERM - -lane=0 -while [[ "${lane}" -lt "${jobs}" ]]; do - lane_files+=("$(mktemp "${TMPDIR:-/tmp}/golib-lane.XXXXXX")") - lane=$((lane + 1)) -done -lane=0 -while IFS= read -r module; do - [[ -n "${module}" ]] || continue - printf '%s\n' "${module}" >>"${lane_files[${lane}]}" - lane=$(((lane + 1) % jobs)) -done <<<"${selection}" - -printf 'parallel-safe verification snapshot jobs=%s\n' "${jobs}" -# Non-interactive job control gives every background lane its own process group. -set -m -lane=0 -while [[ "${lane}" -lt "${jobs}" ]]; do - snapshot_parent="$( - mktemp -d "${TMPDIR:-/tmp}/golib-verification.XXXXXX" - )" - snapshot="${snapshot_parent}/repository" - snapshot_parents+=("${snapshot_parent}") - "${root}/.golib/scripts/create-verification-snapshot.sh" \ - "${root}" "${snapshot}" - selected_modules="$(paste -sd, - <"${lane_files[${lane}]}")" - ( - cd "${snapshot}" - GOLIB_VERIFICATION_SNAPSHOT=1 \ - ./scripts/run-modules.sh \ - "${gate}" --jobs 1 --modules "${selected_modules}" - ) & - snapshot_pids+=("$!") - snapshot_process_groups+=("$!") - lane=$((lane + 1)) -done - -status=0 -for pid in "${snapshot_pids[@]}"; do - if ! wait "${pid}"; then - status=1 - fi -done -snapshot_processes_complete=1 -for group in "${snapshot_process_groups[@]}"; do - if kill -0 -- "-${group}" 2>/dev/null; then - snapshot_processes_complete=0 - status=1 - break - fi -done -if [[ "${status}" -eq 0 ]]; then - while IFS= read -r module; do - [[ -n "${module}" ]] || continue - if [[ "${gate}" == "check" ]]; then - "${root}/.golib/scripts/audit-goals.sh" "${module}" >/dev/null - else - "${root}/.golib/scripts/verify-gate-evidence.sh" \ - "${module}" "${gate}" - fi - done <<<"${selection}" -fi -exit "${status}" diff --git a/.golib/scripts/mutation-verifier-identity.sh b/.golib/scripts/mutation-verifier-identity.sh deleted file mode 100755 index df9c609..0000000 --- a/.golib/scripts/mutation-verifier-identity.sh +++ /dev/null @@ -1,34 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -root="${GOLIB_ROOT:-$(git rev-parse --show-toplevel)}" - -# shellcheck disable=SC1091 -source "${root}/.golib/versions.env" - -: "${GREMLINS_VERSION:?GREMLINS_VERSION is required}" -: "${GREMLINS_SUM:?GREMLINS_SUM is required}" -: "${GREMLINS_GOMOD_SUM:?GREMLINS_GOMOD_SUM is required}" - -inputs=( - scripts/internal/mutation-command.sh - scripts/internal/mutation-coverage.sh - scripts/patches/gremlins-run-all-mutants.patch - scripts/patches/gremlins-shared-coverage.patch - scripts/patches/gremlins-module-relative-diff.patch -) - -{ - printf 'gremlins-version\t%s\n' "${GREMLINS_VERSION}" - printf 'gremlins-sum\t%s\n' "${GREMLINS_SUM}" - printf 'gremlins-gomod-sum\t%s\n' "${GREMLINS_GOMOD_SUM}" - for input in "${inputs[@]}"; do - [[ -f "${root}/.golib/${input}" ]] || { - printf 'mutation verifier input is missing: %s\n' "${input}" >&2 - exit 1 - } - printf 'file\t%s\t%s\n' "${input}" "$( - shasum -a 256 "${root}/.golib/${input}" | awk '{print $1}' - )" - done -} | shasum -a 256 | awk '{print $1}' diff --git a/.golib/scripts/package-source-digest.sh b/.golib/scripts/package-source-digest.sh deleted file mode 100755 index 3c49ed0..0000000 --- a/.golib/scripts/package-source-digest.sh +++ /dev/null @@ -1,42 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 1 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -package_directory="$1" -package_directory="${package_directory#./}" -case "${package_directory}" in - ""|/*|../*|*/../*|*/..) - printf 'package directory must be repository-relative: %s\n' \ - "${package_directory}" >&2 - exit 2 - ;; -esac -absolute="${root}/${package_directory}" -[[ -d "${absolute}" ]] || { - printf 'package directory does not exist: %s\n' "${package_directory}" >&2 - exit 2 -} - -manifest="$(mktemp "${TMPDIR:-/tmp}/golib-source-digest.XXXXXX")" -cleanup() { - rm -f "${manifest}" -} -trap cleanup EXIT HUP INT TERM - -while IFS= read -r -d '' file; do - relative="${file#"${root}/"}" - digest="$(shasum -a 256 "${file}" | awk '{print $1}')" - printf '%s %s\n' "${digest}" "${relative}" >>"${manifest}" -done < <(find "${absolute}" -maxdepth 1 -type f -name '*.go' \ - ! -name '*_test.go' -print0 | LC_ALL=C sort -z) - -[[ -s "${manifest}" ]] || { - printf 'package has no production Go files: %s\n' "${package_directory}" >&2 - exit 1 -} -shasum -a 256 "${manifest}" | awk '{print $1}' diff --git a/.golib/scripts/patches/gremlins-module-relative-diff.patch b/.golib/scripts/patches/gremlins-module-relative-diff.patch deleted file mode 100644 index c573b4c..0000000 --- a/.golib/scripts/patches/gremlins-module-relative-diff.patch +++ /dev/null @@ -1,89 +0,0 @@ -diff --git a/cmd/unleash.go b/cmd/unleash.go ---- a/cmd/unleash.go -+++ b/cmd/unleash.go -@@ -162 +162 @@ -- fDiff, err := diff.New() -+ fDiff, err := diff.New(mod.Root) -diff --git a/internal/diff/parse.go b/internal/diff/parse.go ---- a/internal/diff/parse.go -+++ b/internal/diff/parse.go -@@ -15,2 +15,2 @@ --func New() (Diff, error) { -- return NewWithCmd(exec.Command) -+func New(root string) (Diff, error) { -+ return NewWithCmd(root, exec.Command) -@@ -24 +24 @@ --func NewWithCmd[T execCmd](cmdContext func(name string, args ...string) T) (Diff, error) { -+func NewWithCmd[T execCmd](root string, cmdContext func(name string, args ...string) T) (Diff, error) { -@@ -33 +33 @@ -- cmd := cmdContext("git", "diff", "--merge-base", diffRef) -+ cmd := cmdContext("git", "-C", root, "diff", "--relative", "--merge-base", diffRef) -diff --git a/internal/diff/diff.go b/internal/diff/diff.go ---- a/internal/diff/diff.go -+++ b/internal/diff/diff.go -@@ -38,10 +38,19 @@ -- if fragment.LinesAdded == 0 { -- continue -- } -- -- startLine := int(fragment.NewPosition + fragment.LeadingContext) -- -- changes = append(changes, Change{ -- StartLine: startLine, -- EndLine: startLine + int(fragment.LinesAdded-1), -- }) -+ lineNumber := int(fragment.NewPosition) -+ for _, line := range fragment.Lines { -+ switch line.Op { -+ case gitdiff.OpAdd: -+ last := len(changes) - 1 -+ if last >= 0 && changes[last].EndLine == lineNumber-1 { -+ changes[last].EndLine = lineNumber -+ } else { -+ changes = append(changes, Change{ -+ StartLine: lineNumber, -+ EndLine: lineNumber, -+ }) -+ } -+ lineNumber++ -+ case gitdiff.OpContext: -+ lineNumber++ -+ case gitdiff.OpDelete: -+ } -+ } -diff --git a/internal/diff/diff_test.go b/internal/diff/diff_test.go ---- a/internal/diff/diff_test.go -+++ b/internal/diff/diff_test.go -@@ -98,0 +99,11 @@ -+ { -+ NewPosition: 500, -+ LinesAdded: 2, -+ Lines: []gitdiff.Line{ -+ {Op: gitdiff.OpContext}, -+ {Op: gitdiff.OpAdd}, -+ {Op: gitdiff.OpContext}, -+ {Op: gitdiff.OpContext}, -+ {Op: gitdiff.OpAdd}, -+ }, -+ }, -@@ -109,0 +121,2 @@ -+ {StartLine: 501, EndLine: 501}, -+ {StartLine: 504, EndLine: 504}, -diff --git a/internal/diff/parse_test.go b/internal/diff/parse_test.go ---- a/internal/diff/parse_test.go -+++ b/internal/diff/parse_test.go -@@ -17 +17 @@ -- d, err := NewWithCmd(m.call) -+ d, err := NewWithCmd("/repo", m.call) -@@ -31 +31 @@ -- _, err := NewWithCmd(m.call) -+ _, err := NewWithCmd("/repo", m.call) -@@ -40 +40 @@ -- expectedArgs := []string{"diff", "--merge-base", "test"} -+ expectedArgs := []string{"-C", "/repo", "diff", "--relative", "--merge-base", "test"} -@@ -56 +56 @@ -- _, err := NewWithCmd(m.call) -+ _, err := NewWithCmd("/repo", m.call) -@@ -73 +73 @@ -- result, err := NewWithCmd(m.call) -+ result, err := NewWithCmd("/repo", m.call) diff --git a/.golib/scripts/patches/gremlins-run-all-mutants.patch b/.golib/scripts/patches/gremlins-run-all-mutants.patch deleted file mode 100644 index 7ac7799..0000000 --- a/.golib/scripts/patches/gremlins-run-all-mutants.patch +++ /dev/null @@ -1,60 +0,0 @@ -diff --git a/internal/engine/engine.go b/internal/engine/engine.go ---- a/internal/engine/engine.go -+++ b/internal/engine/engine.go -@@ -188,5 +188 @@ -- var status mutator.Status -- -- if mu.codeData.Cov.IsCovered(pos) { -- status = mutator.Runnable -- } -+ status := mutator.Runnable -diff --git a/internal/engine/executor.go b/internal/engine/executor.go ---- a/internal/engine/executor.go -+++ b/internal/engine/executor.go -@@ -197,2 +196,0 @@ -- cmd := m.execContext(ctx, "go", m.getTestArgs(pkg)...) -- cmd.Dir = m.mutant.Workdir() -@@ -200 +198,8 @@ -- cmd.Dir = rootDir -+ status := m.runTestCommand( -+ ctx, -+ m.mutant.Workdir(), -+ m.getTestArgsFor(pkg, false, false), -+ ) -+ if status != mutator.Lived { -+ return status -+ } -@@ -201,0 +207,16 @@ -+ -+ workingDir := m.mutant.Workdir() -+ if m.integrationMode { -+ workingDir = rootDir -+ } -+ -+ return m.runTestCommand(ctx, workingDir, m.getTestArgs(pkg)) -+} -+ -+func (m *mutantExecutor) runTestCommand( -+ ctx context.Context, -+ workingDir string, -+ args []string, -+) mutator.Status { -+ cmd := m.execContext(ctx, "go", args...) -+ cmd.Dir = workingDir -@@ -219,0 +241,7 @@ -+ return m.getTestArgsFor(pkg, m.integrationMode, true) -+} -+ -+func (m *mutantExecutor) getTestArgsFor( -+ pkg string, -+ integrationMode, includeBuildTags bool, -+) []string { -@@ -221 +249 @@ -- if m.buildTags != "" { -+ if includeBuildTags && m.buildTags != "" { -@@ -231 +259 @@ -- args = append(args, fmt.Sprintf("-cpu %d", m.testCPU)) -+ args = append(args, "-cpu", fmt.Sprint(m.testCPU)) -@@ -235 +263 @@ -- if m.integrationMode { -+ if integrationMode { diff --git a/.golib/scripts/patches/gremlins-shared-coverage.patch b/.golib/scripts/patches/gremlins-shared-coverage.patch deleted file mode 100644 index 86e8912..0000000 --- a/.golib/scripts/patches/gremlins-shared-coverage.patch +++ /dev/null @@ -1,51 +0,0 @@ -diff --git a/internal/coverage/coverage.go b/internal/coverage/coverage.go ---- a/internal/coverage/coverage.go -+++ b/internal/coverage/coverage.go -@@ -102,0 +103,5 @@ -+ if c.integrationMode { -+ if err := c.executeUnitBaseline(); err != nil { -+ return Result{}, fmt.Errorf("unit test baseline failed: %w", err) -+ } -+ } -@@ -143,0 +149,11 @@ -+func (c *Coverage) executeUnitBaseline() error { -+ cmd := c.cmdContext("go", "test", "-count=1", "./...") -+ if out, err := cmd.CombinedOutput(); err != nil { -+ log.Infof("\n%s\n", string(out)) -+ -+ return err -+ } -+ -+ return nil -+} -+ -@@ -144,0 +161,29 @@ -+ if sourcePath := os.Getenv("GOLIB_GREMLINS_COVERAGE_PROFILE"); sourcePath != "" { -+ elapsedText := os.Getenv("GOLIB_GREMLINS_COVERAGE_ELAPSED") -+ elapsed, err := time.ParseDuration(elapsedText) -+ if err != nil || elapsed <= 0 { -+ return 0, fmt.Errorf("invalid external coverage elapsed duration %q", elapsedText) -+ } -+ source, err := os.Open(sourcePath) -+ if err != nil { -+ return 0, fmt.Errorf("open external coverage profile: %w", err) -+ } -+ defer func() { -+ _ = source.Close() -+ }() -+ destination, err := os.Create(c.filePath()) -+ if err != nil { -+ return 0, fmt.Errorf("create coverage profile: %w", err) -+ } -+ defer func() { -+ _ = destination.Close() -+ }() -+ if _, err := io.Copy(destination, source); err != nil { -+ return 0, fmt.Errorf("copy external coverage profile: %w", err) -+ } -+ if err := destination.Close(); err != nil { -+ return 0, fmt.Errorf("close coverage profile: %w", err) -+ } -+ -+ return elapsed, nil -+ } diff --git a/.golib/scripts/release.sh b/.golib/scripts/release.sh deleted file mode 100755 index a2ed9d3..0000000 --- a/.golib/scripts/release.sh +++ /dev/null @@ -1,59 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -dry_run=0 -public=0 -while [[ $# -gt 0 ]]; do - case "$1" in - --dry-run) dry_run=1; shift ;; - --public) public=1; shift ;; - *) break ;; - esac -done -if [[ "${dry_run}" -ne 1 || $# -ne 1 ]]; then - printf 'usage: %s --dry-run [--public] \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -module="$1" -record="$(jq -ce --arg directory "${module}" \ - '.modules[] | select(.directory == $directory and .releasable == true)' \ - "${root}/modules.json")" -module_path="$(jq -r '.module_path' <<<"${record}")" -tag_prefix="$(jq -r '.tag_prefix' <<<"${record}")" -version="v$(jq -r '.version' <<<"${record}")" -tag="${tag_prefix}${version#v}" -directory="${root}/${module}" - -[[ "$(sed -n 's/^module[[:space:]]\+//p' "${directory}/go.mod")" == "${module_path}" ]] -if grep -Eq '^[[:space:]]*replace([[:space:]]|$)' "${directory}/go.mod"; then - printf 'release module contains a replace directive: %s\n' "${module}" >&2 - exit 1 -fi -if git -C "${root}" show-ref --verify --quiet "refs/tags/${tag}"; then - printf 'release tag already exists: %s\n' "${tag}" >&2 - exit 1 -fi - -task="$(mktemp -d "${TMPDIR:-/tmp}/golib-release.XXXXXX")" -# shellcheck disable=SC2329 # Invoked by the release EXIT trap. -cleanup() { - chmod -R u+w "${task}" 2>/dev/null || true - find "${task}" -depth -delete 2>/dev/null || true -} -trap cleanup EXIT HUP INT TERM - -if [[ "${public}" -eq 1 ]]; then - GOPROXY="https://proxy.golang.org,direct" GOWORK=off \ - go list -m "${module_path}@${version}" >/dev/null -else - proxy="${task}/proxy" - mkdir "${proxy}" - "${root}/.golib/scripts/build-local-proxy.sh" "${proxy}" "${version}" - GOPROXY="file://${proxy},https://proxy.golang.org,direct" \ - GONOSUMDB="github.com/faustbrian/go-*" GOWORK=off \ - go list -m "${module_path}@${version}" >/dev/null -fi - -printf 'release dry-run passed: %s %s\n' "${module_path}" "${tag}" diff --git a/.golib/scripts/repository-check.sh b/.golib/scripts/repository-check.sh deleted file mode 100755 index 353adb2..0000000 --- a/.golib/scripts/repository-check.sh +++ /dev/null @@ -1,46 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -root="$(git rev-parse --show-toplevel)" -repository="github.com/faustbrian/go-lease" - -jq -e --arg repository "${repository}" ' - .repository == $repository and - (.modules | length > 0) and - all(.modules[]; - (.directory == "." or (.directory | startswith("/") | not)) and - ( - .releasable == false or - .module_path == $repository or - (.module_path | startswith($repository + "/")) - ) - ) -' "${root}/modules.json" >/dev/null - -while IFS= read -r module; do - directory="$(jq -r --arg module "${module}" \ - '.modules[] | select(.module_path == $module) | .directory' \ - "${root}/modules.json")" - [[ "$(sed -n 's/^module[[:space:]]\+//p' "${root}/${directory}/go.mod")" == "${module}" ]] - if grep -Eq '^[[:space:]]*replace([[:space:]]|$)' "${root}/${directory}/go.mod"; then - printf 'committed replace directive in %s\n' "${directory}/go.mod" >&2 - exit 1 - fi -done < <(jq -r '.modules[].module_path' "${root}/modules.json") - -if grep -REnI \ - --exclude-dir='.git' \ - --exclude-dir='.artifacts' \ - --exclude='go.sum' \ - --exclude='CHANGELOG.md' \ - --exclude='repository-check.sh' \ - 'github\.com/faustbrian/golib/pkg|/Users/[^/]+/Developer|\.\./go-' \ - "${root}"; then - printf 'monorepo or sibling-checkout reference remains\n' >&2 - exit 1 -fi - -git -C "${root}" ls-files --error-unmatch package-lock.json >/dev/null - -git diff --check -printf 'standalone repository contract passed\n' diff --git a/.golib/scripts/restore-ci-mutation-evidence.sh b/.golib/scripts/restore-ci-mutation-evidence.sh deleted file mode 100755 index 5fe812b..0000000 --- a/.golib/scripts/restore-ci-mutation-evidence.sh +++ /dev/null @@ -1,273 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -lt 1 || $# -gt 2 ]]; then - printf 'usage: %s [artifact-archive]\n' "$0" >&2 - exit 2 -fi - -root="${GOLIB_ROOT:-$(git rev-parse --show-toplevel)}" -module="$1" -archive="${2:-}" - -if ! jq -e --arg directory "${module}" \ - '.modules[] | select(.directory == $directory)' \ - "${root}/modules.json" >/dev/null; then - printf 'cannot restore mutation evidence for unknown module: %s\n' \ - "${module}" >&2 - exit 2 -fi - -task="$(mktemp -d "${TMPDIR:-/tmp}/golib-ci-evidence.XXXXXX")" -cleanup() { - find "${task}" -depth -delete 2>/dev/null || true -} -trap cleanup EXIT HUP INT TERM - -checkpoint_directory="${root}/.artifacts/${module}/mutation-checkpoints" -mkdir -p "${checkpoint_directory}" -packages="${task}/packages" -jq -r --arg directory "${module}" ' - .modules[] - | select(.directory == $directory) - | .packages[] - | select(.coverage_required == true) - | .directory -' "${root}/modules.json" | LC_ALL=C sort >"${packages}" -expected="$(wc -l <"${packages}" | tr -d ' ')" -restored=0 -require_exact_digest=0 -digest_directory="${task}/input-digests" -fallback_directory="${task}/fallback-checkpoints" -history_migrations="${root}/.golib/mutation-history-migrations.json" -current_verifier_identity="" -current_revision="" -current_gremlins_version="" - -command -v unzip >/dev/null || { - printf 'unzip is required to restore CI mutation evidence\n' >&2 - exit 1 -} - -restore_archive() { - local candidate_archive="$1" - local entries package_directory slug entry candidate destination fallback - local expected_digest migrated - if [[ ! -s "${candidate_archive}" ]] || - ! unzip -tqq "${candidate_archive}" >/dev/null; then - return 1 - fi - - entries="${task}/entries" - unzip -Z1 "${candidate_archive}" >"${entries}" - while IFS= read -r package_directory; do - [[ -n "${package_directory}" ]] || continue - slug=root - if [[ "${package_directory}" != "." ]]; then - slug="${package_directory//\//-}" - fi - destination="${checkpoint_directory}/${slug}.json" - [[ ! -s "${destination}" ]] || continue - fallback="${fallback_directory}/${slug}.json" - expected_digest="" - if [[ "${require_exact_digest}" -eq 1 ]]; then - [[ -s "${digest_directory}/${slug}" ]] || continue - expected_digest="$(<"${digest_directory}/${slug}")" - fi - entry="mutation-checkpoints/${slug}.json" - if [[ "$(grep -Fxc -- "${entry}" "${entries}")" -ne 1 ]]; then - continue - fi - candidate="$(mktemp "${task}/checkpoint.XXXXXX")" - if ! unzip -p "${candidate_archive}" "${entry}" >"${candidate}" || - ! jq -e \ - --arg module "${module}" \ - --arg package "${package_directory}" \ - ' - .schema_version == 3 and - .module == $module and - .package == $package and - (.execution_revision | type == "string") and - (.execution_revision | test("^[0-9a-f]{40}$")) and - (.gate_input_digest | type == "string") and - (.gate_input_digest | test("^[0-9a-f]{64}$")) and - (.gremlins_version | type == "string") and - (.gremlins_version | length > 0) and - ( - (.gremlins_verifier_sha256 // null) == null or - ( - (.gremlins_verifier_sha256 | type == "string") and - (.gremlins_verifier_sha256 | test("^[0-9a-f]{64}$")) - ) - ) and - (.report.files | type == "array") and - ([.report.files[].mutations[]? | select(.status != "KILLED")] | length == 0) - ' "${candidate}" >/dev/null; then - find "${candidate}" -delete 2>/dev/null || true - continue - fi - if [[ "${require_exact_digest}" -eq 0 ]]; then - mv "${candidate}" "${destination}" - restored=$((restored + 1)) - continue - fi - if jq -e \ - --arg digest "${expected_digest}" \ - --arg version "${current_gremlins_version}" \ - --arg verifier "${current_verifier_identity}" ' - .gate_input_digest == $digest and - .gremlins_version == $version and - .gremlins_verifier_sha256 == $verifier and - ( - ( - .verifier_identity_source == "executed" and - (.gremlins_binary_sha256 | type == "string") and - (.gremlins_binary_sha256 | test("^[0-9a-f]{64}$")) - ) or - ( - .verifier_identity_source == "approved-semantic-migration" and - ( - (.gremlins_binary_sha256 // null) == null or - ( - (.gremlins_binary_sha256 | type == "string") and - (.gremlins_binary_sha256 | test("^[0-9a-f]{64}$")) - ) - ) - ) - ) - ' "${candidate}" >/dev/null; then - mv "${candidate}" "${destination}" - restored=$((restored + 1)) - continue - fi - migrated="$(mktemp "${task}/migrated-checkpoint.XXXXXX")" - if [[ -s "${history_migrations}" ]] && - "${root}/.golib/scripts/internal/reuse-approved-mutation-checkpoint.sh" \ - "${history_migrations}" \ - "${candidate}" \ - "${module}" \ - "${package_directory}" \ - "${expected_digest}" \ - "${current_gremlins_version}" \ - "${current_verifier_identity}" \ - "${current_revision}" \ - "${migrated}"; then - mv "${migrated}" "${destination}" - find "${candidate}" -delete 2>/dev/null || true - restored=$((restored + 1)) - continue - fi - find "${migrated}" -delete 2>/dev/null || true - mkdir -p "${fallback_directory}" - if [[ ! -s "${fallback}" ]]; then - mv "${candidate}" "${fallback}" - else - find "${candidate}" -delete 2>/dev/null || true - fi - done <"${packages}" -} - -if [[ -z "${archive}" ]]; then - repository="${GITHUB_REPOSITORY:-}" - repository_id="${GITHUB_REPOSITORY_ID:-}" - run_id="${GITHUB_RUN_ID:-}" - if [[ -z "${GH_TOKEN:-}" || -z "${repository}" || - ! "${repository_id}" =~ ^[0-9]+$ || ! "${run_id}" =~ ^[0-9]+$ ]]; then - printf '[%s] prior CI mutation evidence is unavailable\n' "${module}" - exit 0 - fi - command -v gh >/dev/null || { - printf 'GitHub CLI is required to restore CI mutation evidence\n' >&2 - exit 1 - } - - artifact_slug="${module//\//-}" - [[ "${artifact_slug}" != "." ]] || artifact_slug=root - artifact_name="evidence-${artifact_slug}" - if ! artifacts="$( - gh api --method GET \ - "repos/${repository}/actions/artifacts" \ - -f "name=${artifact_name}" \ - -f per_page=100 - )"; then - printf '[%s] prior CI mutation evidence lookup failed; executing fresh\n' \ - "${module}" - exit 0 - fi - artifact_ids="$( - jq -r \ - --argjson repository_id "${repository_id}" \ - --argjson run_id "${run_id}" ' - [ - .artifacts[] - | select( - .expired == false and - .workflow_run.head_repository_id == $repository_id and - .workflow_run.head_branch == "main" and - .workflow_run.id != $run_id - ) - ] - | sort_by(.created_at) - | reverse - | .[].id - ' <<<"${artifacts}" - )" - if [[ -z "${artifact_ids}" ]]; then - printf '[%s] no trusted prior CI mutation evidence found\n' "${module}" - exit 0 - fi - mkdir -p "${digest_directory}" - # shellcheck disable=SC1091 - source "${root}/.golib/versions.env" - current_gremlins_version="${GREMLINS_VERSION}" - current_verifier_identity="$( - "${root}/.golib/scripts/mutation-verifier-identity.sh" - )" - current_revision="${GITHUB_SHA:-$(git -C "${root}" rev-parse HEAD)}" - require_exact_digest=1 - while IFS= read -r package_directory; do - [[ -n "${package_directory}" ]] || continue - slug=root - if [[ "${package_directory}" != "." ]]; then - slug="${package_directory//\//-}" - fi - if ! "${root}/.golib/scripts/gate-input-digest.sh" \ - mutation "${module}" "${package_directory}" \ - >"${digest_directory}/${slug}"; then - find "${digest_directory}/${slug}" -delete 2>/dev/null || true - fi - done <"${packages}" - while IFS= read -r artifact_id; do - [[ "${artifact_id}" =~ ^[0-9]+$ ]] || continue - candidate_archive="${task}/evidence-${artifact_id}.zip" - if ! gh api \ - "repos/${repository}/actions/artifacts/${artifact_id}/zip" \ - >"${candidate_archive}"; then - continue - fi - restore_archive "${candidate_archive}" || continue - [[ "${restored}" -lt "${expected}" ]] || break - done <<<"${artifact_ids}" - while IFS= read -r package_directory; do - [[ -n "${package_directory}" ]] || continue - slug=root - if [[ "${package_directory}" != "." ]]; then - slug="${package_directory//\//-}" - fi - destination="${checkpoint_directory}/${slug}.json" - fallback="${fallback_directory}/${slug}.json" - if [[ ! -s "${destination}" && -s "${fallback}" ]]; then - mv "${fallback}" "${destination}" - restored=$((restored + 1)) - fi - done <"${packages}" -else - if ! restore_archive "${archive}"; then - printf '[%s] prior CI mutation evidence archive is invalid; executing fresh\n' \ - "${module}" - exit 0 - fi -fi - -printf '[%s] restored %d prior content-addressed mutation checkpoints\n' \ - "${module}" "${restored}" diff --git a/.golib/scripts/run-gate-with-evidence.sh b/.golib/scripts/run-gate-with-evidence.sh deleted file mode 100755 index 366d64f..0000000 --- a/.golib/scripts/run-gate-with-evidence.sh +++ /dev/null @@ -1,234 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 2 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -module="$1" -gate="$2" -artifact="${root}/.artifacts/${module}/evidence" -legacy_evidence="${artifact}/${gate}.json" -legacy_log="${artifact}/${gate}.log" -lock_root="${artifact}/.locks" -lock="${lock_root}/${gate}.lock" -temporary_evidence="" -temporary_log="" -temporary_legacy_evidence="${legacy_evidence}.tmp.$$" -temporary_legacy_log="${legacy_log}.tmp.$$" -started_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" -execution_revision="$(git rev-parse HEAD)" -lock_acquired=0 - -# shellcheck disable=SC2329 # Invoked by the signal and exit trap. -cleanup() { - if [[ -n "${temporary_evidence}" ]]; then - rm -f "${temporary_evidence}" - fi - if [[ -n "${temporary_log}" ]]; then - rm -f "${temporary_log}" - fi - rm -f "${temporary_legacy_evidence}" "${temporary_legacy_log}" - if [[ "${lock_acquired}" -eq 1 ]]; then - owner="$(readlink "${lock}" 2>/dev/null || true)" - if [[ "${owner}" == "$$" ]]; then - rm -f "${lock}" - fi - fi -} - -forward_gate_output() { - local line output_open=1 - trap '' PIPE - while IFS= read -r line || [[ -n "${line}" ]]; do - if [[ "${output_open}" -eq 1 ]] && - ! printf '%s\n' "${line}" 2>/dev/null; then - output_open=0 - fi - done - return 0 -} - -emit_evidence_status() ( - trap '' PIPE - printf '[%s] %s evidence: %s\n' "$1" "$2" "$3" 2>/dev/null || true -) - -trap cleanup EXIT -trap 'exit 130' HUP INT TERM -if [[ -x "${root}/.golib/scripts/internal/migrate-runner-isolation-evidence.sh" ]]; then - "${root}/.golib/scripts/internal/migrate-runner-isolation-evidence.sh" \ - "${module}" "${gate}" -fi -mkdir -p "${artifact}" -mkdir -p "${lock_root}" -while ! ln -s "$$" "${lock}" 2>/dev/null; do - if [[ -L "${lock}" ]]; then - owner="$(readlink "${lock}" 2>/dev/null || true)" - if [[ "${owner}" =~ ^[0-9]+$ ]] && ! kill -0 "${owner}" 2>/dev/null; then - rm -f "${lock}" - continue - fi - elif [[ -d "${lock}" ]]; then - owner="$(cat "${lock}/owner" 2>/dev/null || true)" - if [[ "${owner}" =~ ^[0-9]+$ ]] && ! kill -0 "${owner}" 2>/dev/null; then - rm -f "${lock}/owner" - rmdir "${lock}" 2>/dev/null || true - continue - fi - if [[ -z "${owner}" ]] && rmdir "${lock}" 2>/dev/null; then - continue - fi - elif [[ -f "${lock}" ]]; then - owner="$(cat "${lock}" 2>/dev/null || true)" - if [[ "${owner}" =~ ^[0-9]+$ ]] && ! kill -0 "${owner}" 2>/dev/null; then - rm -f "${lock}" - continue - fi - if [[ -z "${owner}" ]]; then - rm -f "${lock}" - continue - fi - fi - sleep 0.05 -done -lock_acquired=1 - -input_digest="$("${root}/.golib/scripts/gate-input-digest.sh" "${gate}" "${module}")" -digest_artifact="${artifact}/by-input/${gate}" -evidence="${digest_artifact}/${input_digest}.json" -log="${digest_artifact}/${input_digest}.log" -temporary_evidence="${evidence}.tmp.$$" -temporary_log="${log}.tmp.$$" -mkdir -p "${digest_artifact}" -rm -f "${temporary_evidence}" "${temporary_log}" - -source_evidence="${evidence}" -source_log="${log}" -if [[ ! -f "${source_evidence}" || ! -f "${source_log}" ]]; then - source_evidence="${legacy_evidence}" - source_log="${legacy_log}" -fi -if [[ -f "${source_evidence}" && -f "${source_log}" ]]; then - recorded_log_sha256="$(jq -er '.log_sha256' "${source_evidence}" 2>/dev/null || true)" - current_log_sha256="$(shasum -a 256 "${source_log}" | awk '{print $1}')" - if jq -e \ - --arg module "${module}" \ - --arg gate "${gate}" \ - --arg input_digest "${input_digest}" \ - --arg log_sha256 "${current_log_sha256}" \ - ' - .schema_version == 1 and - .module == $module and - .gate == $gate and - (.result == "passed" or .result == "not_applicable" or .result == "advisory") and - .exit_code == 0 and - .input_digest == $input_digest and - .completed_input_digest == $input_digest and - .log_sha256 == $log_sha256 - ' "${source_evidence}" >/dev/null 2>&1 && - [[ "${recorded_log_sha256}" == "${current_log_sha256}" ]]; then - revalidated_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" - reused_result="$(jq -r '.result' "${source_evidence}")" - if grep -Fq ': not applicable by catalog policy' "${source_log}"; then - reused_result=not_applicable - elif [[ "${gate}" == "nilaway" ]] && - grep -Eq 'NilAway advisory exit status: [1-9][0-9]*$' "${source_log}"; then - reused_result=advisory - fi - jq \ - --arg revalidated_revision "${execution_revision}" \ - --arg revalidated_at "${revalidated_at}" \ - --arg result "${reused_result}" \ - ' - .result = $result | - .revalidated_revision = $revalidated_revision | - .revalidated_at = $revalidated_at | - .reuse_count = ((.reuse_count // 0) + 1) - ' "${source_evidence}" >"${temporary_evidence}" - cp "${source_log}" "${temporary_log}" - mv "${temporary_log}" "${log}" - mv "${temporary_evidence}" "${evidence}" - cp "${log}" "${temporary_legacy_log}" - cp "${evidence}" "${temporary_legacy_evidence}" - mv "${temporary_legacy_log}" "${legacy_log}" - mv "${temporary_legacy_evidence}" "${legacy_evidence}" - emit_evidence_status "${module}" "${gate}" reused - exit 0 - fi -fi - -set +e -"${root}/.golib/scripts/check-module.sh" "${module}" "${gate}" 2>&1 | - tee "${temporary_log}" | - forward_gate_output -pipeline_status=("${PIPESTATUS[@]}") -command_status=${pipeline_status[0]} -tee_status=${pipeline_status[1]} -if [[ "${command_status}" -eq 0 && "${tee_status}" -ne 0 ]]; then - command_status="${tee_status}" -fi -set -e - -completed_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" -completed_revision="$(git rev-parse HEAD)" -completed_digest="$("${root}/.golib/scripts/gate-input-digest.sh" "${gate}" "${module}")" -log_sha256="$(shasum -a 256 "${temporary_log}" | awk '{print $1}')" -result=passed -if [[ "${command_status}" -ne 0 ]]; then - result=failed -elif [[ "${completed_digest}" != "${input_digest}" ]]; then - result=invalidated - command_status=1 -elif grep -Fq ': not applicable by catalog policy' "${temporary_log}"; then - result=not_applicable -elif [[ "${gate}" == "nilaway" ]] && - grep -Eq 'NilAway advisory exit status: [1-9][0-9]*$' "${temporary_log}"; then - result=advisory -fi - -jq -n \ - --arg module "${module}" \ - --arg gate "${gate}" \ - --arg result "${result}" \ - --argjson exit_code "${command_status}" \ - --arg execution_revision "${execution_revision}" \ - --arg completed_revision "${completed_revision}" \ - --arg input_digest "${input_digest}" \ - --arg completed_digest "${completed_digest}" \ - --arg log_sha256 "${log_sha256}" \ - --arg started_at "${started_at}" \ - --arg completed_at "${completed_at}" \ - --arg go_version "$(go env GOVERSION)" \ - --arg goos "$(go env GOOS)" \ - --arg goarch "$(go env GOARCH)" \ - '{ - schema_version: 1, - module: $module, - gate: $gate, - result: $result, - exit_code: $exit_code, - execution_revision: $execution_revision, - completed_revision: $completed_revision, - input_digest: $input_digest, - completed_input_digest: $completed_digest, - log_sha256: $log_sha256, - started_at: $started_at, - completed_at: $completed_at, - environment: { - go_version: $go_version, - goos: $goos, - goarch: $goarch - } - }' >"${temporary_evidence}" - -mv "${temporary_log}" "${log}" -mv "${temporary_evidence}" "${evidence}" -cp "${log}" "${temporary_legacy_log}" -cp "${evidence}" "${temporary_legacy_evidence}" -mv "${temporary_legacy_log}" "${legacy_log}" -mv "${temporary_legacy_evidence}" "${legacy_evidence}" -emit_evidence_status "${module}" "${gate}" "${result}" -exit "${command_status}" diff --git a/.golib/scripts/run-modules.sh b/.golib/scripts/run-modules.sh deleted file mode 100755 index f22fd9d..0000000 --- a/.golib/scripts/run-modules.sh +++ /dev/null @@ -1,45 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -lt 2 ]]; then - printf 'usage: %s <--all|--modules LIST>\n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -gate="$1" -shift -case "$1" in - --all) - selection="$(jq -r '.modules[].directory' "${root}/modules.json")" - ;; - --modules) - [[ $# -eq 2 ]] || exit 2 - selection="${2//,/\\n}" - ;; - *) - printf 'unknown module selection: %s\n' "$1" >&2 - exit 2 - ;; -esac - -while IFS= read -r module; do - [[ -n "${module}" ]] || continue - ( - task="$(mktemp -d "${TMPDIR:-/tmp}/golib-services.XXXXXX")" - environment="${task}/environment" - state="${task}/state" - # shellcheck disable=SC2329 # Invoked by the subshell EXIT trap. - cleanup() { - "${root}/.golib/scripts/stop-services.sh" "${state}" || true - find "${task}" -depth -delete 2>/dev/null || true - } - trap cleanup EXIT HUP INT TERM - "${root}/.golib/scripts/start-services.sh" "${module}" "${environment}" "${state}" - set -a - # shellcheck source=/dev/null - source "${environment}" - set +a - "${root}/.golib/scripts/check-module.sh" "${module}" "${gate}" - ) -done <<<"${selection}" diff --git a/.golib/scripts/stage-ci-evidence.sh b/.golib/scripts/stage-ci-evidence.sh deleted file mode 100755 index 955435e..0000000 --- a/.golib/scripts/stage-ci-evidence.sh +++ /dev/null @@ -1,144 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 3 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="${GOLIB_ROOT:-$(git rev-parse --show-toplevel)}" -module="$1" -destination="$2" -outcome="$3" -script_directory="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -source="${root}/.artifacts" -if [[ "${module}" != "." ]]; then - source="${source}/${module}" -fi -case "${outcome}" in - success|failure|cancelled|skipped) ;; - *) - printf 'invalid CI contract outcome for %s: %s\n' \ - "${module}" "${outcome}" >&2 - exit 2 - ;; -esac -source_available=0 -if [[ -d "${source}" && ! -L "${source}" ]]; then - source="$(cd "${source}" && pwd -P)" - source_available=1 -elif [[ -e "${source}" || -L "${source}" ]]; then - printf 'invalid attributable evidence source for %s\n' "${module}" >&2 - exit 1 -fi -destination_name="$(basename "${destination}")" -destination_parent="$(cd "$(dirname "${destination}")" && pwd -P)" || { - printf 'CI evidence destination parent is unavailable: %s\n' \ - "${destination}" >&2 - exit 1 -} -[[ "${destination_name}" != "." && "${destination_name}" != ".." ]] || { - printf 'invalid CI evidence destination: %s\n' "${destination}" >&2 - exit 1 -} -destination="${destination_parent}/${destination_name}" -[[ ! -e "${destination}" && ! -L "${destination}" ]] || { - printf 'CI evidence destination already exists: %s\n' \ - "${destination}" >&2 - exit 1 -} -if [[ "${source_available}" -eq 1 ]]; then - case "${destination}" in - "${source}"|"${source}"/*) - printf 'CI evidence destination must be outside its source: %s\n' \ - "${destination}" >&2 - exit 1 - ;; - esac -fi - -# Cancellation can bypass the mutation runner's signal trap. Reclaim only -# scratch directories whose recorded owner process is demonstrably gone. -# shellcheck disable=SC1091 -. "${script_directory}/internal/mutation-scratch.sh" -if [[ "${source_available}" -eq 1 ]]; then - mutation_scratch_recover_abandoned "${source}" -fi - -mkdir -p "${destination}" -inventory="$(mktemp "${TMPDIR:-/tmp}/golib-ci-evidence.XXXXXXXX")" -stage_complete=0 -cleanup() { - local exit_code=$? - trap - EXIT HUP INT TERM - find "${inventory}" -delete 2>/dev/null || true - if [[ "${stage_complete}" -eq 0 && -d "${destination}" && - ! -L "${destination}" ]]; then - find "${destination}" -depth -delete 2>/dev/null || true - fi - exit "${exit_code}" -} -trap cleanup EXIT -trap 'exit 129' HUP -trap 'exit 130' INT -trap 'exit 143' TERM - -files=0 -if [[ "${source_available}" -eq 1 ]]; then - find "${source}" -mindepth 1 \ - \( -type d \( \ - -name 'mutation-run-*' -o \ - -name '*.go-cache-*' -o \ - -name 'go-cache' -o \ - -name '.locks' -o \ - -name '*.lock' -o \ - -name '*.tmp.*' \ - \) -prune \) -o -print0 >"${inventory}" -else - : >"${inventory}" -fi -while IFS= read -r -d '' candidate; do - relative="${candidate#"${source}/"}" - basename="${candidate##*/}" - case "${basename}" in - *.tmp.*|*.lock) - continue - ;; - esac - if [[ -L "${candidate}" ]]; then - printf 'refusing to stage symbolic link: %s\n' "${relative}" >&2 - exit 1 - fi - if [[ -d "${candidate}" ]]; then - mkdir -p "${destination}/${relative}" - continue - fi - if [[ ! -f "${candidate}" ]]; then - printf 'refusing to stage unsupported artifact: %s\n' \ - "${relative}" >&2 - exit 1 - fi - mkdir -p "$(dirname "${destination}/${relative}")" - cp -p "${candidate}" "${destination}/${relative}" - files=$((files + 1)) -done <"${inventory}" - -jq -n \ - --arg module "${module}" \ - --arg outcome "${outcome}" \ - --arg repository "${GITHUB_REPOSITORY:-}" \ - --arg run_id "${GITHUB_RUN_ID:-}" \ - --arg run_attempt "${GITHUB_RUN_ATTEMPT:-}" \ - --arg revision "${GITHUB_SHA:-}" \ - '{ - schema_version: 1, - module: $module, - outcome: $outcome, - repository: $repository, - run_id: $run_id, - run_attempt: $run_attempt, - revision: $revision - }' >"${destination}/ci-result.json" -files=$((files + 1)) -stage_complete=1 -printf '[%s] staged %d durable evidence files\n' "${module}" "${files}" diff --git a/.golib/scripts/start-services.sh b/.golib/scripts/start-services.sh deleted file mode 100755 index 44b9082..0000000 --- a/.golib/scripts/start-services.sh +++ /dev/null @@ -1,296 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 3 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -module="$1" -environment_file="$2" -state_file="$3" -slug="$(printf '%s' "${module}" | tr '/.' '--')-${RANDOM}" - -# shellcheck source=/dev/null -source "${root}/.golib/versions.env" -: >"${environment_file}" -: >"${state_file}" - -services="$(jq -r --arg directory "${module}" \ - '.modules[] | select(.directory == $directory) | .required_services[]' \ - "${root}/modules.json")" -[[ -n "${services}" ]] || exit 0 -command -v docker >/dev/null || { - printf 'Docker is required by %s for: %s\n' "${module}" "${services}" >&2 - exit 1 -} - -record() { - printf '%s\n' "$1" >>"${state_file}" -} - -record_resource() { - local kind="$1" - shift - printf '%s' "${kind}" >>"${state_file}" - printf '\t%s' "$@" >>"${state_file}" - printf '\n' >>"${state_file}" -} - -wait_for() { - local container="$1" - shift - for _ in {1..90}; do - if docker exec "${container}" "$@" >/dev/null 2>&1; then - return 0 - fi - if [[ "$(docker inspect --format '{{.State.Running}}' "${container}" 2>/dev/null || true)" != "true" ]]; then - break - fi - sleep 1 - done - docker logs "${container}" >&2 || true - printf 'service did not become healthy: %s\n' "${container}" >&2 - exit 1 -} - -published_port() { - docker port "$1" "$2/tcp" | tail -1 | sed 's/.*://' -} - -acquire_rabbitstream_lock() { - local lock owner started - lock="${TMPDIR:-/tmp}/golib-rabbitstream-fixture.lock" - started="${SECONDS}" - while ! mkdir "${lock}" 2>/dev/null; do - owner="$(cat "${lock}/owner" 2>/dev/null || true)" - if [[ "${owner}" =~ ^[0-9]+$ ]] && ! kill -0 "${owner}" 2>/dev/null; then - rm -f "${lock}/owner" - rmdir "${lock}" 2>/dev/null || true - continue - fi - if ((SECONDS - started >= 1200)); then - printf 'timed out waiting for the RabbitStream fixture lock\n' >&2 - exit 1 - fi - sleep 1 - done - printf '%s\n' "${PPID}" >"${lock}/owner" - record_resource lock "${lock}" "${PPID}" -} - -write_service_environment() { - local name="$1" - local value="$2" - printf '%s=%q\n' "${name}" "${value}" >>"${environment_file}" -} - -start_rabbitstream() { - local mode="$1" - local integration fixture_id standalone_project cluster_project tls_project - local fixture_parent tls_runtime - - command -v openssl >/dev/null || { - printf 'OpenSSL is required by %s for RabbitStream fixtures\n' "${module}" >&2 - exit 1 - } - acquire_rabbitstream_lock - integration="${root}/.golib/services/rabbitstream" - fixture_id="codex-rabbitstream-${RANDOM}-$$" - standalone_project="${fixture_id}-single" - cluster_project="${fixture_id}-cluster" - tls_project="${fixture_id}-tls" - - RABBITSTREAM_USER="rabbitstream-$(openssl rand -hex 8)" - RABBITSTREAM_PASSWORD="$(openssl rand -hex 24)" - RABBITSTREAM_ERLANG_COOKIE="$(openssl rand -hex 32)" - RABBITSTREAM_RESTRICTED_USER="restricted-$(openssl rand -hex 8)" - RABBITSTREAM_RESTRICTED_PASSWORD="$(openssl rand -hex 24)" - export RABBITSTREAM_USER RABBITSTREAM_PASSWORD RABBITSTREAM_ERLANG_COOKIE - export RABBITSTREAM_RESTRICTED_USER RABBITSTREAM_RESTRICTED_PASSWORD - - record_resource compose "${integration}" "${integration}/standalone-compose.yaml" \ - "${standalone_project}" - ( - cd "${integration}" - COMPOSE_PROJECT_NAME="${standalone_project}" ./standalone-setup.sh - ) - write_service_environment RABBITSTREAM_TEST_HOST localhost - write_service_environment RABBITSTREAM_TEST_PORT 15552 - write_service_environment RABBITSTREAM_TEST_USER "${RABBITSTREAM_USER}" - write_service_environment RABBITSTREAM_TEST_PASSWORD "${RABBITSTREAM_PASSWORD}" - write_service_environment RABBITSTREAM_TEST_RESTART_CONTAINER \ - "${standalone_project}-rabbit-1" - write_service_environment RABBITSTREAM_TEST_PROXY_API http://127.0.0.1:18474 - write_service_environment RABBITSTREAM_TEST_PROXY_NAME rabbitstream - - [[ "${mode}" == "full" ]] || return 0 - - record_resource compose "${integration}" "${integration}/compose.yaml" \ - "${cluster_project}" - ( - cd "${integration}" - COMPOSE_PROJECT_NAME="${cluster_project}" ./setup.sh - ) - write_service_environment RABBITSTREAM_CLUSTER_PORTS 15561,15562,15563 - write_service_environment RABBITSTREAM_CLUSTER_CONTAINERS \ - "15561=${cluster_project}-rabbit1-1,15562=${cluster_project}-rabbit2-1,15563=${cluster_project}-rabbit3-1" - write_service_environment RABBITSTREAM_CLUSTER_PROJECT "${cluster_project}" - write_service_environment RABBITSTREAM_ERLANG_COOKIE "${RABBITSTREAM_ERLANG_COOKIE}" - write_service_environment RABBITSTREAM_UPGRADE_IMAGE \ - "rabbitmq@sha256:397fde82bc04522d88680b57cbf5d70caae715a76c957404e52e3f0fa056b8f3" - write_service_environment RABBITSTREAM_UPGRADE_FROM_VERSION 4.3.4 - write_service_environment RABBITSTREAM_UPGRADE_TO_VERSION 4.3.5 - - fixture_parent="$(mktemp -d "${TMPDIR:-/tmp}/golib-rabbitstream.XXXXXX")" - tls_runtime="${fixture_parent}/tls" - record_resource directory "${fixture_parent}" - record_resource compose "${integration}" "${integration}/tls-compose.yaml" \ - "${tls_project}" - ( - cd "${integration}" - RABBITSTREAM_TLS_RUNTIME="${tls_runtime}" \ - COMPOSE_PROJECT_NAME="${tls_project}" ./tls-setup.sh - ) - write_service_environment RABBITSTREAM_TLS_HOST localhost - write_service_environment RABBITSTREAM_TLS_PORT 15571 - write_service_environment RABBITSTREAM_TLS_USER "${RABBITSTREAM_USER}" - write_service_environment RABBITSTREAM_TLS_PASSWORD "${RABBITSTREAM_PASSWORD}" - write_service_environment RABBITSTREAM_TLS_RUNTIME "${tls_runtime}" - write_service_environment RABBITSTREAM_RESTRICTED_USER \ - "${RABBITSTREAM_RESTRICTED_USER}" - write_service_environment RABBITSTREAM_RESTRICTED_PASSWORD \ - "${RABBITSTREAM_RESTRICTED_PASSWORD}" -} - -while IFS= read -r service; do - case "${service}" in - postgresql) - container="golib-postgres-${slug}" - postgres_version="${POSTGRES_IMAGE#postgres:}" - postgres_version="${postgres_version%-alpine}" - docker run --detach --name "${container}" -p 127.0.0.1::5432 \ - -e POSTGRES_USER=golib -e POSTGRES_PASSWORD=golib \ - -e POSTGRES_DB=golib "${POSTGRES_IMAGE}" >/dev/null - record "${container}" - wait_for "${container}" pg_isready -U golib -d golib - port="$(published_port "${container}" 5432)" - cat >>"${environment_file}" </dev/null - record "${container}" - wait_for "${container}" valkey-cli ping - port="$(published_port "${container}" 6379)" - cat >>"${environment_file}" </dev/null - record "${container}" - wait_for "${container}" redis-cli ping - port="$(published_port "${container}" 6379)" - cat >>"${environment_file}" </dev/null - record "${container}" - sleep 2 - port="$(published_port "${container}" 4222)" - printf 'NATS_URL=nats://127.0.0.1:%s\n' "${port}" >>"${environment_file}" - ;; - nsq) - container="golib-nsq-${slug}" - docker run --detach --name "${container}" -p 127.0.0.1::4150 \ - "${NSQ_IMAGE}" /nsqd --broadcast-address=127.0.0.1 >/dev/null - record "${container}" - sleep 2 - port="$(published_port "${container}" 4150)" - printf 'NSQD_TCP_ADDRESS=127.0.0.1:%s\n' "${port}" >>"${environment_file}" - ;; - rabbitmq) - container="golib-rabbitmq-${slug}" - docker run --detach --name "${container}" --hostname "${container}" \ - --user rabbitmq \ - -p 127.0.0.1::5672 \ - "${RABBITMQ_IMAGE}" >/dev/null - record "${container}" - wait_for "${container}" rabbitmq-diagnostics -q ping - port="$(published_port "${container}" 5672)" - printf 'RABBITMQ_URL=amqp://guest:guest@127.0.0.1:%s/\n' \ - "${port}" >>"${environment_file}" - ;; - rabbitstream) - start_rabbitstream full - ;; - rabbitstream-standalone) - start_rabbitstream standalone - ;; - opensearch) - # shellcheck source=/dev/null - source "${root}/.golib/services/opensearch/opensearch-images.env" - container="golib-opensearch-${slug}" - opensearch_image="${opensearch_image_repository}@${opensearch_new_digest}" - docker run --detach --name "${container}" -p 127.0.0.1::9200 \ - --cpus=1 --memory=1g --pids-limit=512 \ - --ulimit nofile=1024:1024 \ - -e discovery.type=single-node \ - -e DISABLE_SECURITY_PLUGIN=true \ - -e OPENSEARCH_JAVA_OPTS='-Xms512m -Xmx512m' \ - "${opensearch_image}" >/dev/null - record "${container}" - port="$(published_port "${container}" 9200)" - ready=0 - for _ in {1..120}; do - if curl --connect-timeout 2 --max-time 5 --fail --silent \ - "http://127.0.0.1:${port}/" >/dev/null; then - ready=1 - break - fi - if [[ "$(docker inspect --format '{{.State.Running}}' "${container}" 2>/dev/null || true)" != "true" ]]; then - break - fi - sleep 1 - done - if [[ "${ready}" -ne 1 ]]; then - docker logs "${container}" >&2 || true - printf 'service did not become healthy: %s\n' "${container}" >&2 - exit 1 - fi - cat >>"${environment_file}" <&2 - exit 1 - ;; - esac -done <<<"${services}" diff --git a/.golib/scripts/stop-services.sh b/.golib/scripts/stop-services.sh deleted file mode 100755 index a3ac576..0000000 --- a/.golib/scripts/stop-services.sh +++ /dev/null @@ -1,110 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 1 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -state_file="$1" -[[ -f "${state_file}" ]] || exit 0 -cleanup_timeout="${GOLIB_DOCKER_CLEANUP_TIMEOUT_SECONDS:-30}" -if [[ ! "${cleanup_timeout}" =~ ^[1-9][0-9]*$ ]]; then - printf 'GOLIB_DOCKER_CLEANUP_TIMEOUT_SECONDS must be a positive integer\n' >&2 - exit 2 -fi - -remove_container() { - local container="$1" - local started="${SECONDS}" - - docker rm --force "${container}" >/dev/null 2>&1 & - local docker_pid=$! - while kill -0 "${docker_pid}" >/dev/null 2>&1; do - if ((SECONDS - started >= cleanup_timeout)); then - kill -TERM "${docker_pid}" >/dev/null 2>&1 || true - sleep 0.1 - kill -KILL "${docker_pid}" >/dev/null 2>&1 || true - wait "${docker_pid}" >/dev/null 2>&1 || true - printf 'timed out removing Docker container %s after %ss\n' \ - "${container}" "${cleanup_timeout}" >&2 - return - fi - sleep 0.1 - done - wait "${docker_pid}" >/dev/null 2>&1 || true -} - -remove_compose_project() { - local directory="$1" - local file="$2" - local project="$3" - local started="${SECONDS}" - - env RABBITSTREAM_USER=cleanup RABBITSTREAM_PASSWORD=cleanup \ - RABBITSTREAM_ERLANG_COOKIE=cleanup \ - docker compose --project-directory "${directory}" -f "${file}" \ - -p "${project}" down --volumes --remove-orphans >/dev/null 2>&1 & - local docker_pid=$! - while kill -0 "${docker_pid}" >/dev/null 2>&1; do - if ((SECONDS - started >= cleanup_timeout)); then - kill -TERM "${docker_pid}" >/dev/null 2>&1 || true - sleep 0.1 - kill -KILL "${docker_pid}" >/dev/null 2>&1 || true - wait "${docker_pid}" >/dev/null 2>&1 || true - printf 'timed out removing Docker Compose project %s after %ss\n' \ - "${project}" "${cleanup_timeout}" >&2 - return - fi - sleep 0.1 - done - wait "${docker_pid}" >/dev/null 2>&1 || true -} - -remove_owned_directory() { - local directory="$1" - case "$(basename "${directory}")" in - golib-rabbitstream.*) ;; - *) - printf 'refusing to remove unexpected service directory: %s\n' \ - "${directory}" >&2 - return 1 - ;; - esac - [[ -d "${directory}" ]] || return - chmod -R u+w "${directory}" 2>/dev/null || true - find "${directory}" -depth -delete -} - -lock_path="" -lock_owner="" - -while IFS=$'\t' read -r kind first second third; do - [[ -n "${kind}" ]] || continue - case "${kind}" in - container) remove_container "${first}" ;; - compose) remove_compose_project "${first}" "${second}" "${third}" ;; - directory) remove_owned_directory "${first}" ;; - lock) - lock_path="${first}" - lock_owner="${second}" - ;; - *) remove_container "${kind}" ;; - esac -done <"${state_file}" - -if [[ -n "${lock_path}" ]]; then - case "$(basename "${lock_path}")" in - golib-rabbitstream-fixture.lock) ;; - *) - printf 'refusing to remove unexpected service lock: %s\n' \ - "${lock_path}" >&2 - exit 1 - ;; - esac - current_owner="$(cat "${lock_path}/owner" 2>/dev/null || true)" - if [[ "${current_owner}" == "${lock_owner}" ]]; then - rm -f "${lock_path}/owner" - rmdir "${lock_path}" 2>/dev/null || true - fi -fi diff --git a/.golib/scripts/test-oidc-keycloak-interoperability.sh b/.golib/scripts/test-oidc-keycloak-interoperability.sh deleted file mode 100755 index b87f798..0000000 --- a/.golib/scripts/test-oidc-keycloak-interoperability.sh +++ /dev/null @@ -1,81 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -root="$(git rev-parse --show-toplevel)" -# shellcheck source=/dev/null -source "${root}/.golib/versions.env" -: "${KEYCLOAK_IMAGE:?KEYCLOAK_IMAGE must be pinned}" -expected_version="26.3.2" -fixture="${root}/pkg/authentication/oidc/testdata/keycloak-26.3.2-realm.json" -run_directory="$(mktemp -d)" -state_file="${run_directory}/containers" -token_file="${run_directory}/id-token" -container="golib-oidc-keycloak-${$}-${RANDOM}" -: >"${state_file}" - -cleanup() { - "${root}/.golib/scripts/stop-services.sh" "${state_file}" - find "${run_directory}" -depth -delete -} -trap cleanup EXIT - -command -v docker >/dev/null || { - printf 'Docker is required for OIDC Keycloak interoperability\n' >&2 - exit 1 -} - -printf '%s\n' "${container}" >"${state_file}" -docker run --detach --name "${container}" --publish 127.0.0.1::8080 \ - --mount "type=bind,src=${fixture},dst=/opt/keycloak/data/import/realm.json,readonly" \ - --env KC_BOOTSTRAP_ADMIN_USERNAME=admin \ - --env KC_BOOTSTRAP_ADMIN_PASSWORD=interoperability-admin \ - "${KEYCLOAK_IMAGE}" start-dev --import-realm --hostname-strict=false >/dev/null -port="$(docker port "${container}" 8080/tcp | tail -1 | sed 's/.*://')" -issuer="http://127.0.0.1:${port}/realms/oidc-hardening" -ready=false -for _ in {1..120}; do - if curl --fail --silent --show-error \ - "${issuer}/.well-known/openid-configuration" >/dev/null 2>&1; then - ready=true - break - fi - if [[ "$(docker inspect --format '{{.State.Running}}' "${container}" 2>/dev/null || true)" != "true" ]]; then - break - fi - sleep 1 -done -if [[ "${ready}" != "true" ]]; then - docker logs "${container}" >&2 || true - printf 'Keycloak did not become ready\n' >&2 - exit 1 -fi - -reported_version="$(docker exec "${container}" /opt/keycloak/bin/kc.sh --version)" -if [[ "${reported_version}" != *"${expected_version}"* ]]; then - printf 'unexpected Keycloak version: %s\n' "${reported_version}" >&2 - exit 1 -fi - -token_response="$(curl --fail --silent --show-error \ - --request POST "${issuer}/protocol/openid-connect/token" \ - --data-urlencode grant_type=password \ - --data-urlencode client_id=oidc-client \ - --data-urlencode username=alice \ - --data-urlencode password=interoperability-password \ - --data-urlencode scope=openid)" -id_token="$(jq -er '.id_token | strings | select(length > 0)' <<<"${token_response}")" -umask 077 -printf '%s\n' "${id_token}" >"${token_file}" -unset token_response id_token - -OIDC_INTEROP_ISSUER="${issuer}" \ -OIDC_INTEROP_TOKEN_FILE="${token_file}" \ -OIDC_INTEROP_CLIENT_ID=oidc-client \ -GOCACHE="${run_directory}/go-cache" \ -GOWORK=off \ -go test ./... -tags=integration \ - -run '^(TestGoogleProviderMetadataSnapshot|TestRepresentativeProviderMetadataProfiles|TestKeycloakProviderIssuedIDToken)$' \ - -count=1 - -printf 'Keycloak interoperability passed: version=%s image=%s\n' \ - "${expected_version}" "${KEYCLOAK_IMAGE}" diff --git a/.golib/scripts/update-api-baseline.sh b/.golib/scripts/update-api-baseline.sh deleted file mode 100755 index 69f9085..0000000 --- a/.golib/scripts/update-api-baseline.sh +++ /dev/null @@ -1,48 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 1 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -# shellcheck disable=SC1091 # Canonical repository tool versions. -source "${root}/.golib/versions.env" -case "$1" in - /*) directory="$1" ;; - *) directory="${root}/$1" ;; -esac -if [[ ! -f "${directory}/go.mod" ]]; then - printf 'module has no go.mod: %s\n' "$1" >&2 - exit 1 -fi - -module_path="$( - cd "${directory}" - GOWORK=off go mod edit -json | jq -er '.Module.Path' -)" -baseline="${directory}/api/baseline.txt" -temporary="$(mktemp "${TMPDIR:-/tmp}/golib-api-baseline.XXXXXX")" -cleanup() { - rm -f "${temporary}" -} -trap cleanup EXIT HUP INT TERM - -run_apidiff() { - if [[ -n "${GOLIB_APIDIFF:-}" ]]; then - GOWORK=off go exec-tool "${GOLIB_APIDIFF}" "$@" - else - GOWORK=off go run \ - "golang.org/x/exp/cmd/apidiff@${APIDIFF_VERSION}" "$@" - fi -} - -mkdir -p "$(dirname "${baseline}")" -( - cd "${directory}" - run_apidiff -m -w "${temporary}" "${module_path}" -) -test -s "${temporary}" -mv "${temporary}" "${baseline}" -trap - EXIT HUP INT TERM diff --git a/.golib/scripts/verify-gate-evidence.sh b/.golib/scripts/verify-gate-evidence.sh deleted file mode 100755 index 356d3ee..0000000 --- a/.golib/scripts/verify-gate-evidence.sh +++ /dev/null @@ -1,118 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -ne 2 ]]; then - printf 'usage: %s \n' "$0" >&2 - exit 2 -fi - -root="$(git rev-parse --show-toplevel)" -module="$1" -gate="$2" -artifact="${root}/.artifacts/${module}/evidence" -lock_root="${artifact}/.locks" -lock="${lock_root}/${gate}.lock" -lock_acquired=0 - -cleanup() { - if [[ "${lock_acquired}" -eq 1 ]]; then - owner="$(readlink "${lock}" 2>/dev/null || true)" - if [[ "${owner}" == "$$" ]]; then - rm -f "${lock}" - fi - fi -} - -trap cleanup EXIT -trap 'exit 129' HUP -trap 'exit 130' INT -trap 'exit 143' TERM -mkdir -p "${lock_root}" -while ! ln -s "$$" "${lock}" 2>/dev/null; do - if [[ -L "${lock}" ]]; then - owner="$(readlink "${lock}" 2>/dev/null || true)" - if [[ "${owner}" =~ ^[0-9]+$ ]] && ! kill -0 "${owner}" 2>/dev/null; then - rm -f "${lock}" - continue - fi - elif [[ -d "${lock}" ]]; then - owner="$(cat "${lock}/owner" 2>/dev/null || true)" - if [[ "${owner}" =~ ^[0-9]+$ ]] && ! kill -0 "${owner}" 2>/dev/null; then - rm -f "${lock}/owner" - rmdir "${lock}" 2>/dev/null || true - continue - fi - if [[ -z "${owner}" ]] && rmdir "${lock}" 2>/dev/null; then - continue - fi - elif [[ -f "${lock}" ]]; then - owner="$(cat "${lock}" 2>/dev/null || true)" - if [[ "${owner}" =~ ^[0-9]+$ ]] && ! kill -0 "${owner}" 2>/dev/null; then - rm -f "${lock}" - continue - fi - if [[ -z "${owner}" ]]; then - rm -f "${lock}" - continue - fi - fi - sleep 0.05 -done -lock_acquired=1 - -input_digest="$("${root}/.golib/scripts/gate-input-digest.sh" "${gate}" "${module}")" -digest_artifact="${artifact}/by-input/${gate}" -evidence="${digest_artifact}/${input_digest}.json" -log="${digest_artifact}/${input_digest}.log" -legacy_evidence="${artifact}/${gate}.json" -legacy_log="${artifact}/${gate}.log" - -if [[ ! -f "${evidence}" || ! -f "${log}" ]]; then - evidence="${legacy_evidence}" - log="${legacy_log}" -fi - -[[ -f "${evidence}" ]] || { - printf '[%s] missing %s evidence\n' "${module}" "${gate}" >&2 - exit 1 -} -[[ -f "${log}" ]] || { - printf '[%s] missing %s evidence log\n' "${module}" "${gate}" >&2 - exit 1 -} - -log_sha256="$(shasum -a 256 "${log}" | awk '{print $1}')" -jq -e \ - --arg module "${module}" \ - --arg gate "${gate}" \ - --arg input_digest "${input_digest}" \ - --arg log_sha256 "${log_sha256}" \ - ' - .schema_version == 1 and - .module == $module and - .gate == $gate and - ( - .result == "passed" or - .result == "not_applicable" or - ($gate == "nilaway" and .result == "advisory") - ) and - .exit_code == 0 and - .input_digest == $input_digest and - .completed_input_digest == $input_digest and - .log_sha256 == $log_sha256 - ' "${evidence}" >/dev/null || { - printf '[%s] stale or invalid %s evidence\n' "${module}" "${gate}" >&2 - exit 1 -} - -if [[ "$(jq -r '.result' "${evidence}")" == "advisory" ]] && - ! grep -Eq 'NilAway advisory exit status: [1-9][0-9]*$' "${log}"; then - printf '[%s] stale or invalid %s evidence\n' "${module}" "${gate}" >&2 - exit 1 -fi - -if [[ "$(jq -r '.result' "${evidence}")" == "not_applicable" ]] && - ! grep -Fq ': not applicable by catalog policy' "${log}"; then - printf '[%s] stale or invalid %s evidence\n' "${module}" "${gate}" >&2 - exit 1 -fi diff --git a/.golib/scripts/with-disposable-go-cache.sh b/.golib/scripts/with-disposable-go-cache.sh deleted file mode 100755 index 35b3ba3..0000000 --- a/.golib/scripts/with-disposable-go-cache.sh +++ /dev/null @@ -1,18 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -if [[ $# -eq 0 ]]; then - printf 'usage: %s [arguments...]\n' "$0" >&2 - exit 2 -fi - -gocache="$(mktemp -d "${TMPDIR:-/tmp}/golib-gocache.XXXXXX")" -gomodcache="$(mktemp -d "${TMPDIR:-/tmp}/golib-modcache.XXXXXX")" -cleanup() { - chmod -R u+w "${gocache}" "${gomodcache}" 2>/dev/null || true - find "${gocache}" -depth -delete 2>/dev/null || true - find "${gomodcache}" -depth -delete 2>/dev/null || true -} -trap cleanup EXIT HUP INT TERM - -GOCACHE="${gocache}" GOMODCACHE="${gomodcache}" "$@" diff --git a/.golib/versions.env b/.golib/versions.env deleted file mode 100644 index c7e8a97..0000000 --- a/.golib/versions.env +++ /dev/null @@ -1,20 +0,0 @@ -GO_VERSION=1.26.6 -GOLANGCI_LINT_VERSION=v2.12.2 -STATICCHECK_VERSION=v0.7.0 -NILAWAY_VERSION=v0.0.0-20260720194628-9fd1b8d7bac8 -GOVULNCHECK_VERSION=v1.6.0 -GREMLINS_VERSION=v0.6.0 -GREMLINS_SUM=h1:3G2ROO0I3q4bb5bxElQIUITTuEbl1iOfVYFqunGwrJI= -GREMLINS_GOMOD_SUM=h1:LLbvJR33CWsu1sgvQ4qMzU2rqkwYJK3Qy/Al59eHKjA= -GITLEAKS_VERSION=v8.30.1 -GO_LICENSES_VERSION=v2.0.1 -CYCLONEDX_VERSION=v1.10.0 -ACTIONLINT_VERSION=v1.7.12 -APIDIFF_VERSION=v0.0.0-20260718201538-764159d718ef -POSTGRES_IMAGE=postgres:18.4-alpine -VALKEY_IMAGE=valkey/valkey:9.1.0-alpine -REDIS_IMAGE=redis:8.6.4-alpine -NATS_IMAGE=nats:2.14.2-alpine -NSQ_IMAGE=nsqio/nsq:v1.3.0 -RABBITMQ_IMAGE=rabbitmq:4.3.2-management-alpine -KEYCLOAK_IMAGE=quay.io/keycloak/keycloak@sha256:98fab020a3a490aba0978f237e2a06cd0ea42bf149c6cf10f11c0aaf27728ff2 diff --git a/.golib/mutation-bootstrap/root.zip b/.verification/mutation/bootstrap/root.zip similarity index 100% rename from .golib/mutation-bootstrap/root.zip rename to .verification/mutation/bootstrap/root.zip diff --git a/.golib/mutation-history-migrations.json b/.verification/mutation/migration-ledger.json similarity index 100% rename from .golib/mutation-history-migrations.json rename to .verification/mutation/migration-ledger.json diff --git a/.golib/mutation-zero-inventory.json b/.verification/mutation/zero-inventory.json similarity index 100% rename from .golib/mutation-zero-inventory.json rename to .verification/mutation/zero-inventory.json diff --git a/AGENTS.md b/AGENTS.md index 6ea988c..c5b3d40 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -11,8 +11,8 @@ shown here. - This file is the canonical policy for the complete repository. - Package policies MAY add stricter domain rules but MUST NOT weaken this file. - `CLAUDE.md` and tool-specific files MUST point here rather than duplicate it. -- Historical `.ai/GOAL*.md` files are requirements and evidence, not proof of - completion. Current executable evidence is REQUIRED. +- Historical implementation plans belong in repository history or issue + tracking, not in the released source tree. Current checks MUST pass. ## Repository Structure @@ -177,7 +177,11 @@ shown here. - Comments MUST explain why a constraint or non-obvious implementation exists; they MUST NOT narrate obvious syntax. - Every public module MUST provide a quick start, API reference, examples, - adoption guidance, tradeoffs, security notes, FAQ, and release notes. + guidance on when to use it, explicit limitations, security notes, FAQ, and + release notes. +- The root README MUST remain a concise entry point. Detailed guides, + operations, audits, and maintainer material belong under `docs/` and MUST be + linked through `docs/README.md`. - Documentation and examples MUST compile and be checked in CI. ## Changelogs diff --git a/CHANGELOG.md b/CHANGELOG.md index bd9f5d4..fed863c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,16 @@ ## Unreleased +### Changed + +- Replace copied repository tooling with the pinned `go-library-tools` v1.0.4 + contract while retaining package-owned policy and verification evidence. + +### Documentation + +- Replace archived monorepo links and completed execution artifacts with a + standalone, human-oriented documentation structure. + ## 1.0.0 - 2026-08-25 ### Changed @@ -23,7 +33,7 @@ ### Documentation -- Link the package README to the repository-wide Golib documentation portal. +- Link the package README to package-owned documentation. ### Changed diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b20fa76..5d9f2db 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -18,6 +18,11 @@ New direct dependencies and dependency updates must follow the [dependency governance policy](AGENTS.md#dependencies-and-supply-chain). Package-local update bots are forbidden; the root policy owns every module and action update. +Repository verification is provided by the released `go-library-tools` CLI. +Keep package-specific policy in `modules.json`, `packages.json`, +the repository configuration, and `verification/package.mk`; do not recreate +shared tooling in the consumer repository. + Specification-backed changes must follow the [specification governance contract](AGENTS.md#design), update the affected stable decision entries, and complete the Specification Decisions diff --git a/Makefile b/Makefile index 16b94f5..3098c3e 100644 --- a/Makefile +++ b/Makefile @@ -1,11 +1,13 @@ -SHELL := /usr/bin/env bash +GOLIB ?= golib .PHONY: check ci inventory repository-check check: - ./.golib/scripts/with-disposable-go-cache.sh ./.golib/scripts/run-modules.sh check --all + $(GOLIB) check --all -ci: repository-check check +ci: + $(GOLIB) repository check + $(GOLIB) check --all inventory repository-check: - ./.golib/scripts/repository-check.sh + $(GOLIB) repository check diff --git a/README.md b/README.md index 9f0d9d4..06d5210 100644 --- a/README.md +++ b/README.md @@ -62,7 +62,7 @@ the returned error. No acquisition order or starvation guarantee is provided. See the [documentation index](docs/README.md), [security policy](SECURITY.md), and [changelog](CHANGELOG.md). -## Ecosystem +## Documentation -Use the [Golib documentation portal](https://github.com/faustbrian/golib/blob/main/docs/index.md) -to choose companion packages, supported stacks, recipes, and operations guidance. +Use the [documentation index](docs/README.md) for package-owned guides, +operational contracts, examples, and maintainer references. diff --git a/docs/README.md b/docs/README.md index 09224aa..a371ec5 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,20 +1,39 @@ # Documentation -- Concepts: [state machine](state-machine.md), [fencing](fencing.md), - [backend guarantees](backend-guarantees.md), [API](api.md) -- Setup: [Valkey](quickstart-valkey.md), - [PostgreSQL](quickstart-postgres.md), [migrations](migrations.md) -- Integration: [unique jobs](unique-jobs.md), [schedulers](schedulers.md), - [protected writes](protected-writes.md), [service shutdown](shutdown.md) -- Operations: [renewal and loss](renewal-and-loss.md), - [Kubernetes](kubernetes.md), [failover](failover.md), - [operations](operations.md), [troubleshooting](troubleshooting.md) -- Assurance: [threat model](threat-model.md), - [acceptance audit](audit-evidence.md), - [failure matrix](failure-matrix.md), [resource budgets](resource-budgets.md), - [hardening evidence](hardening.md), [performance](performance.md), - [benchmark baseline](benchmark-baseline.md), [fuzz report](fuzz-report.md), - [race and stress report](race-stress-report.md), - [mutation report](mutation-report.md), [compatibility](compatibility.md) -- Migration: [Laravel locks and unique jobs](laravel-migration.md) -- Help: [FAQ](faq.md) +## Getting started + +- [Valkey quickstart](quickstart-valkey.md) +- [PostgreSQL quickstart](quickstart-postgres.md) +- [API reference](api.md) +- [Database migrations](migrations.md) + +## Concepts and integrations + +- [State machine](state-machine.md) +- [Fencing](fencing.md) +- [Backend guarantees](backend-guarantees.md) +- [Protected writes](protected-writes.md) +- [Unique jobs](unique-jobs.md) +- [Schedulers](schedulers.md) +- [Renewal and loss](renewal-and-loss.md) +- [Service shutdown](shutdown.md) + +## Operations + +- [Operations guide](operations.md) +- [Kubernetes](kubernetes.md) +- [Failover](failover.md) +- [Failure matrix](failure-matrix.md) +- [Resource budgets](resource-budgets.md) +- [Threat model](threat-model.md) +- [Performance](performance.md) +- [Troubleshooting](troubleshooting.md) + +## Reference and maintenance + +- [Compatibility](compatibility.md) +- [Benchmark baseline](benchmark-baseline.md) +- [Laravel migration](laravel-migration.md) +- [FAQ](faq.md) +- [Contributing](../CONTRIBUTING.md) +- [Release history](../CHANGELOG.md) diff --git a/docs/audit-evidence.md b/docs/audit-evidence.md deleted file mode 100644 index 45586bd..0000000 --- a/docs/audit-evidence.md +++ /dev/null @@ -1,94 +0,0 @@ -# Acceptance audit evidence - -This audit maps the fenced distributed lease goal to the current local tree. -Hosted CI remains the maintainer's final external verification step. - -## Core model - -| Requirement | Evidence | -|---|---| -| bounded namespaced keys | `key.go`, `TestKeyIsBoundedAndNamespaced` | -| immutable acquisition policy | `policy.go`, `TestPolicyIsValidatedAndImmutable` | -| owner, fence, times, state | `record.go`, `handle.go`, lifecycle tests | -| try, bounded wait, renew, validate, release | `client.go`, `handle.go` | -| managed renewal and loss | `managed.go`, uncertainty and shutdown tests | -| stable errors | `errors.go`, classification tests | -| deterministic clock/retry | `leasetest.Clock`, injected retry tests | -| waiter, operation, goroutine bounds | client capacity and timeout tests | -| backend skew, client rollback/freeze, response pause | dual local deadline tests | - -## Correctness and backends - -| Requirement | Evidence | -|---|---| -| monotonically increasing success fences | shared conformance and model fuzzing | -| atomic owner plus token renewal | Valkey Lua and PostgreSQL conditional SQL | -| successor-safe release | shared conformance and stale release tests | -| backend time | Valkey `TIME`; PostgreSQL `clock_timestamp()` | -| Valkey cluster-safe layout | hashed same-slot lease/counter key test | -| `NOSCRIPT` recovery | live `SCRIPT FLUSH` integration test | -| PostgreSQL durable continuity | separate `lease_fences` table and migration | -| cleanup without fence reset | bounded cleanup tests and retained counters | -| restore/flush continuity limits | backend guarantees and failover guides | - -Disposable local runs passed under `-race` for PostgreSQL 14, 15, 16, 17, and -18 and Valkey 9. Both backends passed after container restart; Valkey also -passed after script-cache flush. Client-owned TLS, ACL, reconnect, pool, and -command-timeout failures surface through fail-closed adapter tests. The -reproducible `make backend-hardening` target seeds and snapshots fixed fences, -proves continuity across restart, detects reuse after older-snapshot restore, -proves destructive reset returns token 1, promotes streaming PostgreSQL and -Valkey replicas, and repeats script-cache and live partition classification -checks in CI. Its secure Valkey phase rotates the CA, server certificate, named -ACL user, and password; old trust and old ACL credentials fail closed before -the new client proves fence continuity. - -The PostgreSQL operational fault phase forces a transaction abort and a real -`40P01` deadlock, races bounded cleanup against successor acquisition, churns -caller-owned pools, runs acquisition at serializable isolation, and exercises -both additive-compatible and fail-closed incompatible rolling schemas. Fence -history must remain monotonic after every phase. PostgreSQL uses transactional -fence rows instead of sequences; the abort phase proves a rolled-back increment -does not create a committed jump and the next acquisition advances exactly -once. - -The physical-replica phase authorizes SCRAM replication only from the -disposable primary's directly connected network before `pg_basebackup`; the -rule is reloaded explicitly and disappears with the fault container. - -Valkey rolling-script tests accept the documented v1 response and reject -added, removed, or changed response fields as unavailable rather than treating -an incompatible response as ownership. - -## Integrations and security - -Queue workers and scheduler callbacks receive the fence; their direct loss -tests advance authoritative time beyond expiry and prove each callback context -is canceled with `ErrLost`. Service lifecycle bounds handles, stops renewers, -and reports remote release failure. The protected-write example race-tests -concurrent writers and rejects stale and replayed tokens atomically. - -Cryptographic 192-bit owners, hashed backend keys, redacted observations, -redacted classified driver errors, observer panic isolation, token overflow -checks, bounded nonblocking observer slots, bounded cleanup, and the threat -model cover spoofing, collision, -leakage, stale writers, split brain, rollback, restore, malicious contention, -and resource exhaustion. - -## Quality and release - -- `make check`: format, vet, unit, race, exact 100.0% production statement - coverage, repeated lifecycle stress, fuzz smoke, benchmarks, docs, examples, - and API baseline -- `make lint staticcheck`: strict analyzer gates -- `make mutation`: 23 Go mutants plus four adapter comparison classes killed, - zero lived, 100% efficacy and mutant coverage -- `make vuln`: no known reachable Go vulnerabilities -- `make workflows`: pinned workflow syntax validation -- `make nilaway`: visible advisory analysis; findings do not block by policy -- CI: PostgreSQL 14-18 and Valkey 9 matrices, mutation, vulnerability, lint, - advisory NilAway, release workflow, and the locally reproducible gates - -No unsupported fairness, consensus, multi-key atomicity, stopped-expired-work, -or distributed-transaction claim is made. Every dangerous stale effect still -requires protected-resource fencing. diff --git a/docs/fuzz-report.md b/docs/fuzz-report.md deleted file mode 100644 index cd00d43..0000000 --- a/docs/fuzz-report.md +++ /dev/null @@ -1,9 +0,0 @@ -# Fuzz report - -`make fuzz` runs bounded smoke fuzzing for canonical key parsing, policy bounds, -and the memory lease state model. The model limits each input to 128 operations -and mixes acquire, renew, release, expiry jumps, and clock rollback. - -Long-running qualification should extend `FUZZ_TIME` and preserve any generated -corpus. A crash, non-monotonic successful fence, accepted out-of-bound policy, -or unexpected state-machine error is a release blocker. diff --git a/docs/hardening.md b/docs/hardening.md deleted file mode 100644 index c2f27fc..0000000 --- a/docs/hardening.md +++ /dev/null @@ -1,37 +0,0 @@ -# Hardening evidence - -| Requirement | Executable evidence | -|---|---| -| successor-safe stale rejection | `leasetest.RunBackendConformance` | -| independent owner/token rejection | shared conformance forged-identity matrix | -| late/corrupt successful response | `TestHandleRejectsMismatchedSuccessfulResponses` | -| state model and clock jumps | `FuzzLeaseStateModel` | -| backend skew, rollback, and frozen client clock | local dual-deadline tests | -| bounded retry and jitter | `TestAcquireUsesInjectedBoundedJitter` | -| contention race | `TestContentionElectsExactlyOneOwner`, `make race` | -| repeated lifecycle concurrency | `make stress` | -| renewal uncertainty | `TestManagedRenewalReportsUncertaintyAndStopsAdmission` | -| observer re-entrancy | `TestObserverCanInspectHandleDuringStateTransition` | -| blocking observer isolation | `TestBlockingObserverCannotDelayLeaseTransition` | -| queue and scheduler loss cancellation | direct integration loss tests | -| concurrent handle operations | `TestHandleRejectsConcurrentOperationsWithoutBlockingState` | -| deadline during remote operation | `TestHandleFailsClosedWhenDeadlinePassesDuringOperation` | -| shutdown semantics | `leaseservice` hardening tests | -| response corruption | Valkey/PostgreSQL hardening tests | -| script owner/token comparison | `TestScriptsUseBackendTimeAndAtomicComparisons` | -| meaningful production coverage | `make coverage` requires exactly 100.0% | -| fuzz smoke | `make fuzz` | -| ownership mutation resistance | `make mutation`, including Lua/SQL predicates | -| live backend parity | `make integration` with backend environment variables | -| restart, restore, reset, partition, backend promotion | `make backend-hardening` | -| PostgreSQL abort, deadlock, cleanup race, pool churn, isolation | `TestLiveOperationalFaults` | -| Valkey TLS and named ACL rotation | secure continuity phases in backend hardening | -| allocation and latency baseline | `make benchmark` | - -Local release verification uses `make check lint staticcheck nilaway mutation -vuln workflows`, `make backend-hardening`, and live PostgreSQL 14-18 and Valkey -9 matrices. The hardening target owns disposable PostgreSQL 18 and Valkey 9 -containers, repeats conformance after restart, exercises script-cache recovery, -and requires partitioned mutations to return `ErrAmbiguousOutcome`. Hosted CI -is the final external verification step and is not used to block local -progress. diff --git a/docs/mutation-report.md b/docs/mutation-report.md deleted file mode 100644 index 183ee38..0000000 --- a/docs/mutation-report.md +++ /dev/null @@ -1,19 +0,0 @@ -# Mutation report - -`make mutation` pins Gremlins v0.6.0, two workers, and timeout coefficient 10. -It targets the memory reference state machine, where mutations to owner/token -comparison, expiry boundaries, counter increments, active state, and release -conditions are exercised by conformance, model, and hardening tests. It then -runs `scripts/check-fence-mutations.sh` against disposable source copies. - -The adapter mutation gate changes Valkey owner/token comparisons from unequal -to equal and PostgreSQL owner/token comparisons from equal to unequal. The -exact contract tests must fail for every mutated copy. This covers the Lua and -SQL predicates that Go mutation tools cannot rewrite directly. -Shared conformance independently forges only the owner and only the token for -renew, validate, and release, then revalidates the successor after every -rejection. This detects an adapter that accidentally compares only one field. - -The current reference run generated 23 Go mutants: 23 killed, zero lived, zero -uncovered, 100.00% efficacy, and 100.00% mutant coverage. The adapter run -generated four comparison mutation classes: four killed and zero lived. diff --git a/docs/race-stress-report.md b/docs/race-stress-report.md deleted file mode 100644 index c3740b1..0000000 --- a/docs/race-stress-report.md +++ /dev/null @@ -1,22 +0,0 @@ -# Race and lifecycle stress report - -`make stress` repeats the ownership-critical concurrency matrix 25 times under -Go's race detector with shuffled test order. The gate covers contended -acquisition, cancellation, explicit renewal and release, concurrent handle -operations, late responses, managed renewal and loss notification, stale-owner -successor protection, queue and scheduler loss cancellation, and service -shutdown races. - -The command is deterministic in scope and bounded in repetitions. Increase -`STRESS_COUNT` for soak runs; the blocking local and hosted gate keeps the -documented default so execution time remains bounded. - -The release gate is: - -```text -go test -race -shuffle=on -count=25 -run ./... -``` - -Passing this report does not replace backend fault tests. PostgreSQL and Valkey -failover, restore, partition, and credential phases remain in -`make backend-hardening`. diff --git a/modules.json b/modules.json index aeb5469..7f23da0 100644 --- a/modules.json +++ b/modules.json @@ -200,89 +200,9 @@ "specifications": [], "conformance_corpora": [], "provenance": [], - "goal_files": [ - ".ai/GOAL.md", - ".ai/GOAL_HARDEN.md" - ], - "goal_status": "implementation-evidence-inventoried", - "goal_evidence": [ - { - "file": ".ai/GOAL.md", - "requirements_sha256": "504032dfaf2d089b625af60b1a0669e1189f10e8e01728795921379d4b2b4d14", - "implementation_evidence": [ - "CHANGELOG.md", - "README.md", - "docs/README.md", - "docs/api.md", - "docs/compatibility.md", - "docs/hardening.md", - "docs/performance.md", - "docs/threat-model.md" - ], - "verification_gates": [ - "format-check", - "tidy-check", - "safety", - "vet", - "test", - "race", - "coverage", - "lint", - "staticcheck", - "vulnerability", - "secrets", - "licenses", - "sbom", - "fuzz", - "mutation", - "nilaway", - "docs", - "api", - "conformance", - "interoperability", - "benchmark" - ], - "implementation_status": "implemented-requires-fresh-verification" - }, - { - "file": ".ai/GOAL_HARDEN.md", - "requirements_sha256": "54066fea5f68f0f83a505da2ce71d3606ad5257722a084e96bde7a1a70a7330d", - "implementation_evidence": [ - "CHANGELOG.md", - "README.md", - "docs/README.md", - "docs/api.md", - "docs/compatibility.md", - "docs/hardening.md", - "docs/performance.md", - "docs/threat-model.md" - ], - "verification_gates": [ - "format-check", - "tidy-check", - "safety", - "vet", - "test", - "race", - "coverage", - "lint", - "staticcheck", - "vulnerability", - "secrets", - "licenses", - "sbom", - "fuzz", - "mutation", - "nilaway", - "docs", - "api", - "conformance", - "interoperability", - "benchmark" - ], - "implementation_status": "implemented-requires-fresh-verification" - } - ], + "goal_files": [], + "goal_status": "not-applicable", + "goal_evidence": [], "gates": { "api_compatibility": true, "benchmarks": true, diff --git a/scripts/check-api-compat.sh b/scripts/check-api-compat.sh deleted file mode 100755 index 97d08c3..0000000 --- a/scripts/check-api-compat.sh +++ /dev/null @@ -1,5 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -root="$(git rev-parse --show-toplevel)" -exec "${root}/.golib/scripts/check-api-baseline.sh" . diff --git a/scripts/check-coverage.sh b/scripts/check-coverage.sh deleted file mode 100755 index b16f397..0000000 --- a/scripts/check-coverage.sh +++ /dev/null @@ -1,12 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -profile="${TMPDIR:-/tmp}/lease-coverage.out" -packages="$(go list ./... | grep -v '/leasetest$' | grep -v '/examples/' | paste -sd, -)" -go test ./... -coverpkg="$packages" -coverprofile="$profile" -total="$(go tool cover -func="$profile" | awk '/^total:/ {print $3}')" -if [[ "$total" != "100.0%" ]]; then - echo "meaningful production statement coverage: $total (required 100.0%)" >&2 - exit 1 -fi -echo "meaningful production statement coverage: $total" diff --git a/scripts/check-docs.sh b/scripts/check-docs.sh index deee6e4..5599cbe 100755 --- a/scripts/check-docs.sh +++ b/scripts/check-docs.sh @@ -3,15 +3,14 @@ set -euo pipefail required=( README.md CHANGELOG.md SECURITY.md CONTRIBUTING.md - docs/api.md docs/audit-evidence.md docs/backend-guarantees.md docs/failure-matrix.md - docs/benchmark-baseline.md docs/compatibility.md docs/fuzz-report.md - docs/faq.md docs/fencing.md docs/hardening.md docs/kubernetes.md + docs/api.md docs/backend-guarantees.md docs/failure-matrix.md + docs/benchmark-baseline.md docs/compatibility.md + docs/faq.md docs/fencing.md docs/kubernetes.md docs/laravel-migration.md docs/migrations.md docs/operations.md - docs/mutation-report.md docs/performance.md docs/protected-writes.md docs/quickstart-postgres.md docs/quickstart-valkey.md docs/renewal-and-loss.md docs/resource-budgets.md docs/schedulers.md docs/shutdown.md - docs/state-machine.md docs/race-stress-report.md docs/threat-model.md + docs/state-machine.md docs/threat-model.md docs/troubleshooting.md docs/unique-jobs.md ) diff --git a/scripts/check-fuzz.sh b/scripts/check-fuzz.sh deleted file mode 100755 index 5166237..0000000 --- a/scripts/check-fuzz.sh +++ /dev/null @@ -1,7 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -duration="${1:-2s}" -go test . -run '^$' -fuzz '^FuzzKeyParsing$' -fuzztime="$duration" -go test . -run '^$' -fuzz '^FuzzPolicyBounds$' -fuzztime="$duration" -go test ./memory -run '^$' -fuzz '^FuzzLeaseStateModel$' -fuzztime="$duration" diff --git a/scripts/check-stress.sh b/scripts/check-stress.sh deleted file mode 100755 index 7f8fef6..0000000 --- a/scripts/check-stress.sh +++ /dev/null @@ -1,7 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -count="${STRESS_COUNT:-25}" -pattern='^(TestContentionElectsExactlyOneOwner|TestAcquirePropagatesBackendAndCancellation|TestHandleLifecycleFailsClosed|TestHandleRejectsConcurrentOperationsWithoutBlockingState|TestHandleFailsClosedWhenDeadlinePassesDuringOperation|TestManagedRenewalReportsUncertaintyAndStopsAdmission|TestManagedStopHonorsCallerDeadline|TestStaleOwnerCannotAffectSuccessor|TestAcquireRacingShutdownReleasesReservation|TestManagerBoundsHandlesAndReleasesOnShutdown|TestWorkerCancelsOwnershipSensitiveJobOnLoss|TestCoordinatorCancelsOwnershipSensitiveTaskOnLoss)$' - -go test -race -shuffle=on -count="$count" -run "$pattern" ./... diff --git a/verification/package.mk b/verification/package.mk new file mode 100644 index 0000000..8e3fae4 --- /dev/null +++ b/verification/package.mk @@ -0,0 +1,10 @@ +.PHONY: docs fence-mutations backend-hardening + +docs: + ./scripts/check-docs.sh + +fence-mutations: + ./scripts/check-fence-mutations.sh + +backend-hardening: + ./scripts/check-backend-faults.sh