docs: add TLS scenario guide and re-verify release-facing claims #283

Workflow file for this run

.github/workflows/benchmark.yml at 7487a23

	name: Performance Benchmark

	on:
	pull_request:
	branches: [ "main" ]
	types: [ closed ]
	push:
	tags: [ "*" ]
	workflow_dispatch:

	# Declare default permissions as read only (principle of least privilege)
	permissions:
	contents: read

	# Prevent concurrent benchmark runs to avoid interference
	concurrency:
	group: ${{ github.workflow }}-${{ github.ref }}
	cancel-in-progress: false # Don't cancel in-progress runs as benchmarks are expensive

	jobs:
	benchmark:
	name: Run Integration Benchmarks
	runs-on: ubuntu-latest
	# Only run on merged PRs, not just closed ones
	if: github.event_name != 'pull_request' \|\| github.event.pull_request.merged == true
	# Add timeout to prevent long-running jobs (increased for integration benchmarks).
	# The -Pbenchmark profile DECLARES TWELVE k6 goals: the eight cross-gateway matrix aspects, the
	# two retained non-matrix health benchmarks (healthLiveCheck, gatewayHealth), and the two
	# API-Sheriff-only passthrough-relay executions (mapped relay throughput and the empty-mode
	# no-regression run). All TWELVE of them execute: uploadLarge and websocketEcho still carry a
	# <skip> bound to their own property, but both properties are false in benchmarks/pom.xml, so
	# both goals run. The two stay ordered last in the profile as defense-in-depth, so a fail-fast
	# abort from either cannot discard a goal that would otherwise produce a result.
	#
	# Budget, from per-goal timings measured locally 2026-07-28 rather than estimated:
	# * goal execution ~16 min -- sized on all TWELVE goals, which now all execute.
	# 11 request-rate goals x (60s k6 window + ~10s compose/k6 start and
	# summary write) = ~13 min, plus uploadLarge. uploadLarge is NOT a
	# request-rate goal: it is transfer-bound at reduced concurrency
	# (k6.vus.upload.large = 5), so it does not finish in the same wall time as
	# a request-rate run — it is budgeted at ~3 min on its own. Re-derive this
	# term when a skip property is flipped, or when a goal is added or removed.
	# * stack startup ~120 s -- the lane boots SEVEN native gateway instances (api-sheriff,
	# api-sheriff-mtls, api-sheriff-cookie, api-sheriff-cookie-2,
	# api-sheriff-ws-admission for the WebSocket relay-permit exhaustion
	# regression, api-sheriff-plain-mgmt for the plain-HTTP management opt-out,
	# and api-sheriff-passthrough-empty for the benchmark's empty-passthrough_sni
	# arm) alongside Keycloak, go-httpbin, nginx-static, passthrough-backend,
	# grpc-echo, toxiproxy, asset-origin and prometheus, because
	# start-integration-container.sh runs a bare `up -d`. Each added
	# gateway instance costs roughly another 10 s of the readiness wait — they
	# share one native image, so the term grows with instance count, not with
	# image builds; the two instances added since this term was last derived are
	# what take it from ~100 s to ~120 s. Re-derive this term whenever an
	# api-sheriff* service is added to or removed from
	# integration-tests/docker-compose.yml.
	# * native compile the dominant and most variable term on a cold cache.
	#
	# 75 minutes holds with headroom for the cold native compile; it is kept unchanged because the
	# measured run does not approach the cap.
	timeout-minutes: 75
	permissions:
	# READ, not write. The former `contents: write` was justified in-comment as "needed to upload
	# artifacts", which is not what that scope does: actions/upload-artifact uses the Actions
	# runtime token, not GITHUB_TOKEN's contents scope. Every genuine write in this job goes to
	# the OTHER repository (cuioss.github.io) and is authenticated by the GitHub App token minted
	# below — the checkout of that repo passes it explicitly and the `git push` inherits it. So
	# nothing here needs write access to THIS repository's contents.
	contents: read

	steps:
	- name: Harden the runner (Audit all outbound calls)
	uses: step-security/harden-runner@e14015d583714f6e62063499dc959a02595150a1 # v2.21.1
	with:
	egress-policy: audit

	- name: Create GitHub App token for deployment
	id: app-token
	uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0
	with:
	app-id: ${{ secrets.RELEASE_APP_ID }}
	private-key: ${{ secrets.RELEASE_APP_PRIVATE_KEY }}
	repositories: ${{ github.event.repository.name }},cuioss.github.io
	owner: cuioss

	- name: Checkout code
	uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	with:
	fetch-depth: 0 # Fetch all history for proper versioning
	persist-credentials: false # Prevent token from overriding app token during deploy

	- name: Set up JDK 25
	uses: actions/setup-java@dd06d9cba3e5552c54d9f8ea23572deb30010f7c # v6.0.0
	with:
	java-version: '25'
	distribution: 'temurin'
	cache: maven

	- name: Build api-sheriff
	run: \|
	# Build all modules first to ensure artifacts are available for benchmarking
	./mvnw --no-transfer-progress clean install -DskipTests

	- name: Fetch Previous History
	uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	with:
	repository: cuioss/cuioss.github.io
	ref: main
	path: previous-pages
	sparse-checkout: \|
	api-sheriff/benchmarks/integration/history
	continue-on-error: true

	- name: Prepare Historical Data for Benchmarks
	run: \|
	python3 benchmarks/scripts/benchmark-pages.py prepare-history \
	--previous-pages-dir previous-pages/api-sheriff/benchmarks \
	--output-dir "${GITHUB_WORKSPACE}/benchmark-history"

	- name: Run Integration Benchmarks with k6
	id: run-benchmarks
	run: \|
	# Run k6-based integration benchmarks with native image. The -Pbenchmark profile declares
	# twelve goals: the eight cross-gateway matrix aspects (unauth, bearer, http2,
	# graphql, upload-1MB, upload-50MB, ws, grpc), the two retained non-matrix health
	# benchmarks (healthLiveCheck, gatewayHealth), and the two API-Sheriff-only
	# passthrough-relay executions (mapped and empty). All twelve execute and are
	# expected to produce a CI result. The on-demand APISIX comparison lane is deliberately NOT run
	# here. Maven still fails fast on the goals that do run, so the coverage step below
	# remains what proves which goals actually produced a result.
	echo "Running k6 integration benchmarks with native Quarkus..."
	./mvnw --no-transfer-progress clean verify -pl benchmarks -Pbenchmark \
	-Dbenchmark.history.dir="${GITHUB_WORKSPACE}/benchmark-history/integration"

	# Verify artifacts were generated
	echo "Integration benchmark artifacts generated:"
	ls -la benchmarks/target/benchmark-results/

	# Maven fails fast, so a goal that errors aborts every goal after it and a goal that is never
	# reached leaves no trace in the job output — a suite that ran a handful of the executing goals
	# renders exactly like one that ran all of them. This step diffs the goals that MUST produce a
	# summary document against the documents k6 actually wrote, so a partial suite is visibly
	# partial and a goal that silently vanished fails the job. When Maven failed, it also states
	# in words whether the suite was truncated by that failure. Goals that are not expected to run
	# are named with their reason and their skip property rather than omitted, so "not run" is
	# never indistinguishable from "forgotten".
	#
	# Both the expected set and the not-expected table are DERIVED from the build-generated
	# execution manifest (benchmarks/pom.xml, generate-benchmark-manifest, bound to initialize),
	# not maintained by hand here. That is deliberate: the previous hand-maintained list mirrored
	# the POM with nothing enforcing agreement, and it drifted — the goal set moved from eleven to
	# twelve while prose in this file still said eleven. Deriving both from the manifest makes that
	# drift structurally impossible, and an absent manifest is itself a hard failure rather than an
	# empty expected set that would vacuously report full coverage.
	#
	# The rendering lives in the generator's own `summarise` subcommand rather than being
	# reimplemented in bash here, so the manifest has exactly ONE consumer. Splitting it across two
	# implementations would recreate, one level up, the duplicated-contract defect this manifest was
	# introduced to remove — and did: an earlier bash/jq version of this step diverged on the table
	# heading and mislabelled an inline `<skip>` as "not wired".
	- name: Summarise benchmark coverage
	if: always()
	env:
	BENCHMARK_OUTCOME: ${{ steps.run-benchmarks.outcome }}
	run: \|
	python3 benchmarks/scripts/benchmark-manifest.py summarise \
	--manifest benchmarks/target/benchmark-execution-manifest.json \
	--results-dir benchmarks/target/benchmark-results/k6 \
	--benchmark-outcome "${BENCHMARK_OUTCOME}" \
	--summary-file "$GITHUB_STEP_SUMMARY"

	# The commit SHA reaches the shell through the environment, never through `${{ }}` expansion
	# inside the script body — expression interpolation into a run block is a script-injection
	# sink, and maven.yml and release.yml already hold this line for every value they pass.
	- name: Assemble benchmark artifacts for deployment
	env:
	COMMIT_SHA: ${{ github.sha }}
	run: \|
	python3 benchmarks/scripts/benchmark-pages.py assemble \
	--integration-results benchmarks/target/benchmark-results/gh-pages-ready \
	--previous-pages-dir previous-pages/api-sheriff/benchmarks \
	--output-dir gh-pages \
	--commit-sha "${COMMIT_SHA}"

	- name: Upload benchmark results
	uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
	with:
	name: benchmark-results
	path: gh-pages/
	retention-days: 90 # Keep results for 90 days

	- name: Checkout cuioss.github.io for deployment
	uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	with:
	repository: cuioss/cuioss.github.io
	path: _pages-deploy
	sparse-checkout: api-sheriff/benchmarks
	token: ${{ steps.app-token.outputs.token }}

	- name: Deploy to cuioss.github.io
	env:
	COMMIT_SHA: ${{ github.sha }}
	run: \|
	TARGET_DIR="_pages-deploy/api-sheriff/benchmarks"
	rm -rf "$TARGET_DIR"
	mkdir -p "$TARGET_DIR"
	cp -r gh-pages/* "$TARGET_DIR/"
	cd _pages-deploy
	git config user.name "cuioss-release-bot[bot]"
	git config user.email "cuioss-release-bot[bot]@users.noreply.github.com"
	git add .
	git diff --staged --quiet \|\| git commit -m "Deploy benchmark results from ${COMMIT_SHA}"
	git push

Navigation Menu

Uh oh!

Uh oh!

docs: add TLS scenario guide and re-verify release-facing claims #283

Workflow file

docs: add TLS scenario guide and re-verify release-facing claims #283

Uh oh!

Workflow file for this run