Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,12 @@ change between minor versions; both are versioned and documented when they do.
GitLab and 18% on Discourse — each one a validation query against your
database, which is why the number is published here.

- **The detection section no longer prints a heading with nothing under it.**
A new dimension — the name a schema gives its primary keys — was added to the
model and counted as evidence, but the terminal report did not know how to
print it. A schema whose only detected convention was that one got a section
announcing conventions and naming none.

### Changed

- The lexical fallback no longer extracts a table's trigrams once per column in
Expand Down
33 changes: 31 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,10 @@ VERSION ?= $(shell git describe --tags --always --dirty 2>/dev/null || echo dev)
COMMIT ?= $(shell git rev-parse --short HEAD 2>/dev/null || echo none)
DATE ?= $(shell date -u +%Y-%m-%dT%H:%M:%SZ)

LDFLAGS := -s -w \
# Recursive on purpose: demo-svg overrides VERSION for its own build, and an
# immediate assignment here would have expanded VERSION before that override
# exists.
LDFLAGS = -s -w \
-X '$(PKG)/internal/buildinfo.Version=$(VERSION)' \
-X '$(PKG)/internal/buildinfo.Commit=$(COMMIT)' \
-X '$(PKG)/internal/buildinfo.Date=$(DATE)'
Expand All @@ -13,7 +16,7 @@ LDFLAGS := -s -w \
# has to stay trivial.
export CGO_ENABLED := 0

.PHONY: build test test-integration corpus benchmark lint fmt cover crosscheck release-check image-check clean help
.PHONY: demo-svg build test test-integration corpus benchmark lint fmt cover crosscheck release-check image-check clean help

## build: compile the binary into ./bin
build:
Expand Down Expand Up @@ -112,6 +115,32 @@ image-check:
docker buildx build --platform linux/amd64,linux/arm64 --output=type=cacheonly "$$ctx"
@echo "image ok"

## demo-svg: re-record the README image from a real run
#
# The image is generated rather than screenshotted so it can be regenerated when
# the report format changes — see docs/RELEASING.md. It needs Docker.
#
# It builds at the latest tag rather than at the working tree's version, because
# the tool stamps its own version into the report and a recording that says
# v0.1.2-16-g837ec2a-dirty tells the reader about the maintainer's checkout
# instead of about their database.
demo-svg: VERSION = $(shell git describe --tags --abbrev=0 2>/dev/null || echo dev)
demo-svg: build
@ctr=pgfathom-demo-svg; \
docker rm -f $$ctr >/dev/null 2>&1 || true; \
docker run -d --name $$ctr -e POSTGRES_PASSWORD=demo -e POSTGRES_DB=loja -p 55440:5432 \
-v "$$PWD/demo/schema.sql:/docker-entrypoint-initdb.d/schema.sql:ro" postgres:16-alpine >/dev/null; \
for i in $$(seq 40); do \
docker exec $$ctr psql -U postgres -d loja -c 'SELECT 1 FROM nota_fiscal LIMIT 1' >/dev/null 2>&1 && break; \
sleep 1; \
done; \
COLORTERM=truecolor PGFATHOM_DSN='postgres://postgres:demo@localhost:55440/loja?sslmode=disable' \
./bin/$(BINARY) discover --schema public --full --color always 2>/dev/null > /tmp/pgfathom-demo.ansi; \
docker rm -f $$ctr >/dev/null; \
go run scripts/termsvg.go < /tmp/pgfathom-demo.ansi > assets/demo.svg; \
echo "assets/demo.svg regenerated; the plain text below goes in the README"; \
sed 's/\x1b\[[0-9;]*m//g' /tmp/pgfathom-demo.ansi

## clean: remove build artifacts
clean:
rm -rf bin dist coverage.out
Expand Down
21 changes: 18 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,19 @@

`pgfathom` finds the relationships your database has but never declared — and proves them against the data instead of guessing from column names.

<p align="center">
<img src="assets/demo.svg" alt="pgfathom discover finding one broken and two confirmed relationships" width="900">
</p>

Try it against a throwaway database, with nothing of yours involved:

```console
$ docker compose -f demo/compose.yaml up
```

That starts PostgreSQL, loads [a schema](demo/schema.sql) that declares almost none of its
own integrity, and prints exactly what you see above.

> [!IMPORTANT]
> **Early, and measured.**
> `pgfathom audit` and `pgfathom discover` run end to end, verdicts and reviewable
Expand Down Expand Up @@ -78,7 +91,7 @@ $ pgfathom discover --schema public

```

pgfathom v0.1.1 · PostgreSQL 16.14 · profile pt-br · threshold 0.50
pgfathom v0.1.2 · PostgreSQL 16.14 · profile pt-br · threshold 0.50
full validation — every row was examined; verdicts are conclusive

BROKEN — the relationship is real; its integrity is not (1)
Expand All @@ -100,9 +113,11 @@ $ pgfathom discover --schema public
────────────────────────────────────────────────────────────────────────────────────────────────
none

Nothing detected from 6 tables and 1 declared key; the pt-br profile applies alone.
DETECTED — conventions read from the schema itself, added to pt-br
────────────────────────────────────────────────────────────────────────────────────────────────
key name id 5 occurrences (100%) — e.g. cliente, funcionario, nota_fiscal

1 broken · 2 confirmed · 0 weak · 0 unvalidated · 0 discarded · 134ms
1 broken · 2 confirmed · 0 weak · 0 unvalidated · 0 discarded · 133ms
6 tables · 6 analyzed (100%)
stats prefilter: 3 checked · 0 rejected · 0 without statistics
! statistics reset time unknown — usage counters carry no meaning
Expand Down
1 change: 1 addition & 0 deletions assets/demo.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
36 changes: 36 additions & 0 deletions demo/compose.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# The shortest path from "what is this" to seeing it work.
#
# docker compose -f demo/compose.yaml up
#
# It starts a throwaway PostgreSQL, loads a schema that declares almost none of
# its own integrity, and runs discover against it. Nothing here touches anything
# you already have: the database lives in the container and dies with it.
services:
db:
image: postgres:16-alpine
environment:
POSTGRES_PASSWORD: demo
POSTGRES_DB: loja
volumes:
- ./schema.sql:/docker-entrypoint-initdb.d/schema.sql:ro
healthcheck:
# The report has to run against a loaded schema, and postgres reports
# ready once before running the init scripts. Asking for a table that the
# script creates last is the only signal that the load actually finished.
test: ["CMD-SHELL", "psql -U postgres -d loja -c 'SELECT 1 FROM nota_fiscal LIMIT 1' || exit 1"]
interval: 1s
timeout: 5s
retries: 30

pgfathom:
image: ghcr.io/lvcas-dotcom/pgfathom:latest
depends_on:
db:
condition: service_healthy
command:
- discover
- --dsn
- postgres://postgres:demo@db:5432/loja?sslmode=disable
- --schema
- public
- --full
71 changes: 71 additions & 0 deletions demo/schema.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
-- The demo schema. It is the single copy: docs/DEMO.md points here, and
-- demo/compose.yaml loads it, so the two cannot drift apart.
--
-- Every table here exists to exercise one verdict. See docs/DEMO.md for what
-- each one is for.

CREATE TABLE cliente (
id bigserial PRIMARY KEY,
nome text NOT NULL
);

CREATE TABLE funcionario (
id bigserial PRIMARY KEY,
nome text NOT NULL
);

CREATE TABLE pedido (
id bigserial PRIMARY KEY,
cliente_id bigint NOT NULL,
total numeric(12,2) NOT NULL DEFAULT 0
);

CREATE TABLE item_pedido (
pedido_id bigint NOT NULL,
seq integer NOT NULL,
descricao text NOT NULL,
PRIMARY KEY (pedido_id, seq)
);

CREATE TABLE os_servico (
id bigserial PRIMARY KEY,
resp_tecnico bigint,
aberta_em date NOT NULL DEFAULT current_date
);

INSERT INTO cliente (nome) SELECT 'cliente ' || g FROM generate_series(1, 4000) g;
INSERT INTO funcionario (nome) SELECT 'func ' || g FROM generate_series(1, 300) g;

-- Every order points at a customer that exists: this relationship is sound, and
-- nobody ever declared it.
INSERT INTO pedido (cliente_id, total)
SELECT 1 + (g % 4000), (g % 500)::numeric
FROM generate_series(1, 20000) g;

INSERT INTO item_pedido (pedido_id, seq, descricao)
SELECT p.id, s, 'item ' || s
FROM pedido p, generate_series(1, 3) s;

-- One in forty service orders names a technician who is not in the table. There
-- was never a constraint to stop it.
INSERT INTO os_servico (resp_tecnico)
SELECT CASE WHEN g % 40 = 0 THEN 900000 + g ELSE 1 + (g % 300) END
FROM generate_series(1, 8000) g;

-- The report that ships with the system joins the two tables, which is how the
-- relationship survived in the database even though the constraint did not.
CREATE VIEW os_por_tecnico AS
SELECT f.nome, count(*) AS abertas
FROM os_servico o
JOIN funcionario f ON f.id = o.resp_tecnico
GROUP BY f.nome;

-- A table added later, by people who did declare their keys.
CREATE TABLE nota_fiscal (
id bigserial PRIMARY KEY,
pedido_id bigint NOT NULL REFERENCES pedido(id),
emitida_em date NOT NULL DEFAULT current_date
);
INSERT INTO nota_fiscal (pedido_id) SELECT id FROM pedido WHERE id % 3 = 0;

ANALYZE;
74 changes: 7 additions & 67 deletions docs/DEMO.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,76 +11,16 @@ $ docker run -d --name pgfathom-demo -e POSTGRES_PASSWORD=demo \
-e POSTGRES_DB=loja -p 55432:5432 postgres:16-alpine
```

Load it:
The schema is [`demo/schema.sql`](../demo/schema.sql), kept as one copy so this page and
the compose file cannot drift apart. Load it:

```sql
CREATE TABLE cliente (
id bigserial PRIMARY KEY,
nome text NOT NULL
);

CREATE TABLE funcionario (
id bigserial PRIMARY KEY,
nome text NOT NULL
);

CREATE TABLE pedido (
id bigserial PRIMARY KEY,
cliente_id bigint NOT NULL,
total numeric(12,2) NOT NULL DEFAULT 0
);

CREATE TABLE item_pedido (
pedido_id bigint NOT NULL,
seq integer NOT NULL,
descricao text NOT NULL,
PRIMARY KEY (pedido_id, seq)
);

CREATE TABLE os_servico (
id bigserial PRIMARY KEY,
resp_tecnico bigint,
aberta_em date NOT NULL DEFAULT current_date
);

INSERT INTO cliente (nome) SELECT 'cliente ' || g FROM generate_series(1, 4000) g;
INSERT INTO funcionario (nome) SELECT 'func ' || g FROM generate_series(1, 300) g;

-- Every order points at a customer that exists: this relationship is sound, and
-- nobody ever declared it.
INSERT INTO pedido (cliente_id, total)
SELECT 1 + (g % 4000), (g % 500)::numeric
FROM generate_series(1, 20000) g;

INSERT INTO item_pedido (pedido_id, seq, descricao)
SELECT p.id, s, 'item ' || s
FROM pedido p, generate_series(1, 3) s;

-- One in forty service orders names a technician who is not in the table. There
-- was never a constraint to stop it.
INSERT INTO os_servico (resp_tecnico)
SELECT CASE WHEN g % 40 = 0 THEN 900000 + g ELSE 1 + (g % 300) END
FROM generate_series(1, 8000) g;

-- The report that ships with the system joins the two tables, which is how the
-- relationship survived in the database even though the constraint did not.
CREATE VIEW os_por_tecnico AS
SELECT f.nome, count(*) AS abertas
FROM os_servico o
JOIN funcionario f ON f.id = o.resp_tecnico
GROUP BY f.nome;

-- A table added later, by people who did declare their keys.
CREATE TABLE nota_fiscal (
id bigserial PRIMARY KEY,
pedido_id bigint NOT NULL REFERENCES pedido(id),
emitida_em date NOT NULL DEFAULT current_date
);
INSERT INTO nota_fiscal (pedido_id) SELECT id FROM pedido WHERE id % 3 = 0;

ANALYZE;
```console
$ docker exec -i pgfathom-demo psql -U postgres -d loja < demo/schema.sql
```

Or skip both steps — `docker compose -f demo/compose.yaml up` does all of it and runs the
report.

Run it:

```console
Expand Down
3 changes: 2 additions & 1 deletion docs/RELEASING.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,8 @@ pedir autenticação a quem só quer experimentar.
$ scripts/release-notes.sh v0.1.2
```

3. Se o formato do relatório mudou nesta versão, regrave o exemplo do README.
3. Se o formato do relatório mudou nesta versão, rode `make demo-svg` e cole o
texto que ele imprime no bloco de exemplo do README.
Ele é execução real contra o schema de `docs/DEMO.md`, e o que o torna útil
é ser verdade — um exemplo que já foi verdade é pior que nenhum, porque
ninguém desconfia dele. O número de versão na gravação pode ficar para trás;
Expand Down
3 changes: 3 additions & 0 deletions internal/report/discover.go
Original file line number Diff line number Diff line change
Expand Up @@ -327,6 +327,9 @@ func writeDetection(b *strings.Builder, v DiscoverView) {
{"reference suffix", v.Detection.ColumnSuffixes},
{"reference prefix", v.Detection.ColumnPrefixes},
{"table prefix", v.Detection.TablePrefixes},
// Counted by Empty(), so leaving it out here printed the heading with
// nothing under it — a section announcing conventions and listing none.
{"key name", v.Detection.PrimaryKeyNames},
} {
for _, e := range group.items {
detail := fmt.Sprintf("%d occurrences (%.0f%%)", e.Occurrences, 100*e.Share)
Expand Down
39 changes: 39 additions & 0 deletions internal/report/terminal_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -374,3 +374,42 @@ func TestDetectionCountsReadAsEnglish(t *testing.T) {
t.Errorf("counts of one must read as singular, got:\n%s", got)
}
}

// TestDetectionSectionIsNeverEmpty guards the seam between what Empty() counts
// and what the renderer knows how to print. They drifted once: a detection
// dimension was added to the model and to Empty(), the heading started
// appearing for schemas that had only that dimension, and it appeared with
// nothing beneath it — a section claiming to have read conventions and naming
// none.
func TestDetectionSectionIsNeverEmpty(t *testing.T) {
dimensions := map[string]model.NamingDetection{
"reference suffix": {Enabled: true, ColumnSuffixes: []model.NamingEvidence{{Affix: "_id", Occurrences: 9}}},
"reference prefix": {Enabled: true, ColumnPrefixes: []model.NamingEvidence{{Affix: "id_", Occurrences: 9}}},
"table prefix": {Enabled: true, TablePrefixes: []model.NamingEvidence{{Affix: "tb_", Occurrences: 9}}},
"key name": {Enabled: true, PrimaryKeyNames: []model.NamingEvidence{{Affix: "id", Occurrences: 9}}},
}

for name, detection := range dimensions {
if detection.Empty() {
t.Fatalf("%s: fixture must not be empty, or it tests nothing", name)
}

var b bytes.Buffer
view := report.DiscoverView{
Result: model.NewResult("test", "en", time.Unix(0, 0).UTC(),
model.Coverage{TablesTotal: 1, TablesAnalyzed: 1}),
Detection: detection,
}
if err := report.Discover(&b, view); err != nil {
t.Fatalf("%s: Discover: %v", name, err)
}

if !strings.Contains(b.String(), "DETECTED") {
t.Errorf("%s: a non-empty detection must show the section", name)
continue
}
if !strings.Contains(b.String(), "9 occurrences") {
t.Errorf("%s: the section must name what it detected:\n%s", name, b.String())
}
}
}
Loading
Loading