Skip to content

Commit 01fa9f9

Browse files
dimitridimitri
committed
ci: unified workflow with build-once image sharing and cross-run layer cache
Replace the two independent workflows (run-tests.yml, run-pgaftest.yml) with a single ci.yml that builds each Postgres version's images once and shares them across both the pytest suite and the pgaftest TAP suite. Problem with the old approach ------------------------------ On every push both workflows independently rebuilt all images from scratch. A typical PR triggered six separate image builds for PG14-17 (two workflows × four versions), each re-compiling Postgres + Citus + pg_auto_failover from source. The base→citus→build chain alone takes ~8-10 min per version. New ci.yml (unified, single workflow) -------------------------------------- One build matrix job per Postgres major version replaces both workflows' build stages: style ──► build (PG14..17) ──► pytest (PG14-17) ├─► pgaftest (PG14-17, all schedules) ├─► installcheck (PG14-17) └─► upgrade (PG16) Each build job: - Uses Docker Buildx with GHA layer cache (scope build-pgNN) so the expensive base→citus→build chain is a cache hit on subsequent runs when neither the Dockerfile nor Postgres/Citus versions have changed. - Builds all required targets sequentially (test, run, debian, testrun, pgaftest for PG17); Docker's local daemon cache means the shared build layers are compiled once and reused for every subsequent target in the same job. - Uploads each image as a GitHub Actions artifact (retention: 1 day) for downstream test jobs to download and load. The pytest suite and pgaftest suite both need: [build], not each other, so all test jobs run fully in parallel after the build fan-out. New base-images.yml -------------------- Separate workflow maintains cached base images in GHCR for cross-run layer reuse. Two schedules: nightly (02:00 UTC) — rebuilds PG19 from PostgreSQL master so its frequently-changing headers stay current. weekly (Mon 03:00) — checks github.com/postgres/postgres for new minor-release tags (REL_NN_X pattern, numeric minor only). If a new tag appears, rebuilds the base image, pushes to ghcr.io/hapostgres/pg-cache:pgNN, and commits the updated .github/pg-base-versions/pgNN.ref so the next weekly check knows what's already been built. manual — workflow_dispatch with an optional pgversion input to trigger a rebuild for a specific major version immediately. .github/pg-base-versions/ -------------------------- Seed files recording the current latest minor release for each stable major version (PG14-18), used by the weekly check job as its baseline.
1 parent 53a1a9d commit 01fa9f9

9 files changed

Lines changed: 615 additions & 459 deletions

File tree

‎.github/pg-base-versions/pg14.ref‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
REL_14_23

‎.github/pg-base-versions/pg15.ref‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
REL_15_18

‎.github/pg-base-versions/pg16.ref‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
REL_16_14

‎.github/pg-base-versions/pg17.ref‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
REL_17_10

‎.github/pg-base-versions/pg18.ref‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
REL_18_4

‎.github/workflows/base-images.yml‎

Lines changed: 178 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,178 @@
1+
name: Base images
2+
3+
# Maintains cached base images in GHCR so CI build jobs skip re-compiling
4+
# Postgres and Citus from source on every PR.
5+
#
6+
# Two triggers:
7+
#
8+
# nightly — rebuilds PG19 (tracking PostgreSQL master), every day at 02:00
9+
# UTC. PG19 is in active development so its headers change often
10+
# enough that a daily refresh is worthwhile.
11+
#
12+
# weekly — checks whether a new Postgres minor release has been tagged on
13+
# github.com/postgres/postgres for each stable major version
14+
# (PG14–18). If so, rebuilds and pushes a fresh base image and
15+
# commits the updated .github/pg-base-versions/pgNN.ref file so
16+
# the next weekly check knows what was already built.
17+
#
18+
# manual — workflow_dispatch lets you rebuild any single major version on
19+
# demand (e.g. right after a minor release ships, without waiting
20+
# for the next weekly cron).
21+
#
22+
# Image naming in GHCR:
23+
# ghcr.io/hapostgres/pg-cache:pgNN
24+
#
25+
# The ci.yml build jobs pull this as --cache-from, which makes the
26+
# base→citus→build chain a cache hit when Postgres/Citus haven't changed.
27+
28+
on:
29+
schedule:
30+
# Nightly at 02:00 UTC — rebuilds PG19 (master).
31+
- cron: '0 2 * * *'
32+
# Weekly on Monday 03:00 UTC — checks stable minor versions PG14-18.
33+
- cron: '0 3 * * 1'
34+
35+
workflow_dispatch:
36+
inputs:
37+
pgversion:
38+
description: 'Postgres major version to rebuild (e.g. 17). Leave blank to run the full weekly check.'
39+
required: false
40+
type: string
41+
42+
env:
43+
REGISTRY: ghcr.io
44+
CACHE_IMAGE: ghcr.io/${{ github.repository_owner }}/pg-cache
45+
46+
jobs:
47+
# ---------------------------------------------------------------------------
48+
# Nightly: rebuild PG19 from PostgreSQL master.
49+
# ---------------------------------------------------------------------------
50+
nightly-pg19:
51+
name: Nightly rebuild (PG19 / master)
52+
if: >
53+
github.event_name == 'schedule' && github.event.schedule == '0 2 * * *' ||
54+
github.event_name == 'workflow_dispatch' && github.event.inputs.pgversion == '19'
55+
runs-on: ubuntu-latest
56+
permissions:
57+
contents: read
58+
packages: write
59+
60+
steps:
61+
- uses: actions/checkout@v4
62+
63+
- name: Set up Docker Buildx
64+
uses: docker/setup-buildx-action@v3
65+
66+
- name: Log in to GHCR
67+
uses: docker/login-action@v3
68+
with:
69+
registry: ghcr.io
70+
username: ${{ github.actor }}
71+
password: ${{ secrets.GITHUB_TOKEN }}
72+
73+
- name: Generate git-version.h
74+
run: make version
75+
76+
- name: Build and cache PG19 base image
77+
run: |
78+
docker buildx build \
79+
--cache-from type=registry,ref=${{ env.CACHE_IMAGE }}:pg19 \
80+
--cache-to type=registry,ref=${{ env.CACHE_IMAGE }}:pg19,mode=max \
81+
--build-arg PGVERSION=19 \
82+
--build-arg CITUSTAG=none \
83+
--target build \
84+
.
85+
86+
# ---------------------------------------------------------------------------
87+
# Weekly: detect new Postgres minor releases for stable versions (PG14-18).
88+
# ---------------------------------------------------------------------------
89+
check-minor-releases:
90+
name: Check minor releases (PG${{ matrix.PGVERSION }})
91+
if: >
92+
github.event_name == 'schedule' && github.event.schedule == '0 3 * * 1' ||
93+
github.event_name == 'workflow_dispatch' && (
94+
github.event.inputs.pgversion == '' ||
95+
github.event.inputs.pgversion == matrix.PGVERSION_STR
96+
)
97+
runs-on: ubuntu-latest
98+
permissions:
99+
contents: write
100+
packages: write
101+
strategy:
102+
fail-fast: false
103+
matrix:
104+
include:
105+
- { PGVERSION: 14, PGVERSION_STR: '14', CITUSTAG: v12.1.5 }
106+
- { PGVERSION: 15, PGVERSION_STR: '15', CITUSTAG: v12.1.5 }
107+
- { PGVERSION: 16, PGVERSION_STR: '16', CITUSTAG: v13.2.0 }
108+
- { PGVERSION: 17, PGVERSION_STR: '17', CITUSTAG: v13.2.0 }
109+
- { PGVERSION: 18, PGVERSION_STR: '18', CITUSTAG: v14.1.0 }
110+
111+
steps:
112+
- uses: actions/checkout@v4
113+
with:
114+
# Need push access to update ref files.
115+
token: ${{ secrets.GITHUB_TOKEN }}
116+
117+
- name: Detect latest Postgres ${{ matrix.PGVERSION }} minor release
118+
id: detect
119+
run: |
120+
LATEST=$(git ls-remote --tags https://github.com/postgres/postgres \
121+
"refs/tags/REL_${{ matrix.PGVERSION }}_*" \
122+
| grep -v '\^{}' \
123+
| awk '{print $2}' \
124+
| sed 's|refs/tags/||' \
125+
| grep -E "^REL_${{ matrix.PGVERSION }}_[0-9]+$" \
126+
| sort -V \
127+
| tail -1)
128+
129+
STORED=$(cat .github/pg-base-versions/pg${{ matrix.PGVERSION }}.ref 2>/dev/null || echo "none")
130+
131+
echo "latest=${LATEST}" >> $GITHUB_OUTPUT
132+
echo "stored=${STORED}" >> $GITHUB_OUTPUT
133+
134+
if [ "${LATEST}" != "${STORED}" ] && [ -n "${LATEST}" ]; then
135+
echo "rebuild=true" >> $GITHUB_OUTPUT
136+
echo "PG${{ matrix.PGVERSION }}: new minor release ${LATEST} (was ${STORED}), rebuilding"
137+
else
138+
echo "rebuild=false" >> $GITHUB_OUTPUT
139+
echo "PG${{ matrix.PGVERSION }}: already at ${STORED}, nothing to do"
140+
fi
141+
142+
- name: Set up Docker Buildx
143+
if: steps.detect.outputs.rebuild == 'true'
144+
uses: docker/setup-buildx-action@v3
145+
146+
- name: Log in to GHCR
147+
if: steps.detect.outputs.rebuild == 'true'
148+
uses: docker/login-action@v3
149+
with:
150+
registry: ghcr.io
151+
username: ${{ github.actor }}
152+
password: ${{ secrets.GITHUB_TOKEN }}
153+
154+
- name: Generate git-version.h
155+
if: steps.detect.outputs.rebuild == 'true'
156+
run: make version
157+
158+
- name: Build and cache PG${{ matrix.PGVERSION }} base image
159+
if: steps.detect.outputs.rebuild == 'true'
160+
run: |
161+
docker buildx build \
162+
--cache-from type=registry,ref=${{ env.CACHE_IMAGE }}:pg${{ matrix.PGVERSION }} \
163+
--cache-to type=registry,ref=${{ env.CACHE_IMAGE }}:pg${{ matrix.PGVERSION }},mode=max \
164+
--build-arg PGVERSION=${{ matrix.PGVERSION }} \
165+
--build-arg CITUSTAG=${{ matrix.CITUSTAG }} \
166+
--target build \
167+
.
168+
169+
- name: Update stored ref
170+
if: steps.detect.outputs.rebuild == 'true'
171+
run: |
172+
mkdir -p .github/pg-base-versions
173+
echo "${{ steps.detect.outputs.latest }}" > .github/pg-base-versions/pg${{ matrix.PGVERSION }}.ref
174+
git config user.name "github-actions[bot]"
175+
git config user.email "github-actions[bot]@users.noreply.github.com"
176+
git add .github/pg-base-versions/pg${{ matrix.PGVERSION }}.ref
177+
git commit -m "ci: update PG${{ matrix.PGVERSION }} base image to ${{ steps.detect.outputs.latest }}"
178+
git push

0 commit comments

Comments
 (0)