Skip to content

Commit 89a2eef

Browse files
committed
test: track the exhibit that shows the GR tests assert nothing
Three .ci files that were scratch under a gitignored directory are now tracked. They are the measurement behind the 2026-09-08 row in plan/journal/unwired-feature.md: Ze's Graceful Restart functional tests pass with Graceful Restart unreachable. gr-vacuity-mark-stale-without-gr.ci and gr-vacuity-llgr-transition-without-gr.ci are test/plugin/gr-mark-stale.ci and test/plugin/llgr-transition.ci with the gr plugin removed, so no GR or LLGR code runs at all. Both PASS. The conn=2 re-announcement those tests read as GR evidence is RIBManager.handleState (internal/component/bgp/plugins/rib/rib.go) replaying ribOut, which is deleted in the withdraw path and never on peer-down. gr-vacuity-control-must-fail.ci is the first of those with one octet changed in the conn=2 NLRI. It MUST fail, and it is what stops the two passes being read as a broken runner. Measured this run: mark-stale-without PASS 7.3s, llgr-transition-without PASS 7.3s, control FAIL on 'NLRI: expected=192.168.153.0/24, got=192.168.1.0/24', which is the reason its header names. Placement. The tree has no home for a DEMONSTRATION as opposed to a regression test, and this commit does not build one. test/plugin/ was rejected on both counts: the two passing files would cost 15s on every sweep to assert nothing a regression could break, and the control would redden the suite for every session in this checkout. The must-fail suite (internal/test/runner/testdata/mustfail/) is the right concept and the wrong mechanism, because it runs its fixtures from a Go unit test with no built ze, and the control needs the daemon and the test peer; a missing binary would fail it for a reason its '# must-fail:' line does not name, which is how that gate reddens. So they stay under test/draft/plugin/, where every recursive .ci reader already skips them and TestDraftDirIsInvisibleToRepoChecks holds those readers to it, and .gitignore gains three lines that track this one exhibit by name. They cost no gate a second and run on demand through the plugin suite's draft discovery, named in test/draft/README.md. Two more files were measured and are NOT committed. The 'with' arm of each pair was a verbatim copy of the tracked test it clones (diff is empty against test/plugin/gr-mark-stale.ci and test/plugin/llgr-transition.ci), so committing them would duplicate a test and its vacuity. The 'with' arm of the measurement is the tracked original, which is the point. Every header was rewritten before landing. Each clone still carried the original VALIDATES line -- 'TCP close (not NOTIFICATION) triggers GR activation and mark-stale code path' -- which is the claim the measurement disproves, and the inline comment crediting GR with answering the close went with it.
1 parent dea545a commit 89a2eef

6 files changed

Lines changed: 328 additions & 5 deletions

File tree

.gitignore

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,19 @@ etc/ze/database.zefs*
3737
test/draft/*
3838
!test/draft/README.md
3939

40+
# The Graceful Restart vacuity exhibit, tracked by exception. The three files
41+
# are DEMONSTRATIONS, not regression tests: two pass with Graceful Restart
42+
# unreachable, and the third MUST fail. The tree has no home for a
43+
# demonstration, so they stay here, where they cost no gate second and can
44+
# redden nothing. The four recursive .ci readers skip this directory and
45+
# TestDraftDirIsInvisibleToRepoChecks holds them to it, so for these three that
46+
# skip, rather than the ignore above, is what keeps them out of every gate.
47+
# The directory must be re-included before its contents can be excluded: git
48+
# does not descend into an excluded DIRECTORY.
49+
!test/draft/plugin/
50+
test/draft/plugin/*
51+
!test/draft/plugin/gr-vacuity-*.ci
52+
4053
# MRT data files (large, downloaded)
4154
test/internet/*.gz
4255
test/internet/*.bz2

plan/verification-debt/5e2d7854.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ Clear rows only through `le commit debt-clear` after the named gate exits 0.
55

66
| Date | Session | Subject | Gate owed | Reason | Status |
77
|------|---------|---------|-----------|--------|--------|
8-
| 2026-09-07 | 5e2d7854 | isis: cite RFC 1195 for the level-agnostic point-to-point IIH (+75 more) | full native verification (not FRESH-green) | verify-status is not FRESH-green: STALE: last verify failed (exit=130, at 2026-09-05T17:25:15Z) | open |
8+
| 2026-09-07 | 5e2d7854 | isis: cite RFC 1195 for the level-agnostic point-to-point IIH (+76 more) | full native verification (not FRESH-green) | verify-status is not FRESH-green: STALE: last verify failed (exit=130, at 2026-09-05T17:25:15Z) | open |
99
| 2026-09-07 | 5e2d7854 | isis: cite RFC 1195 for the level-agnostic point-to-point IIH (+23 more) | full native verification over this commit's Go | no full native verification covers this commit's Go | open |
1010
| 2026-09-07 | 5e2d7854 | fix(hub): wire request shutdown before a plugin can ask for it | discovery-index freshness | le discovery-index check reports 765 packages up to date over this tree: the diff adds one unexported function to internal/test/peer and one wiring move in cmd/ze/hub, and creates no package, so ai/PACKAGE-MAP.md has nothing to record | open |
1111
| 2026-09-08 | 5e2d7854 | plan: close qemu-targets-boot-the-shipped-kernel | full native verification (not FRESH-green) | le verify worktree is STALE and was not run: three other sessions own most of the working tree and a full gate would judge their edits, not this closure. Ran instead: the spec's own package tests under -mod=mod (vendor/modules.txt is deleted in another session's working tree), le repository check and le commit audit. Known pre-existing reds carried, none from this closure. | open |

test/draft/README.md

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,22 @@
22

33
A `.ci` file here is **invisible to every suite and every repo-wide gate**. Write
44
and iterate on it as long as you like; it cannot redden `./le verify current mode full`, and
5-
because the directory is gitignored it does not exist in CI at all.
6-
7-
This directory is tracked only for this README. Everything else in it is ignored
8-
(`.gitignore`: `test/draft/` plus `!test/draft/README.md`).
5+
because the directory is gitignored it does not exist in CI at all. The
6+
`gr-vacuity-*.ci` exhibit below is the one tracked exception, and CI does check
7+
it out: it is invisible there because every reader skips this directory, not
8+
because the file is absent.
9+
10+
This directory is tracked for this README and for the `gr-vacuity-*.ci` exhibit
11+
under `plugin/`. Everything else in it is ignored (`.gitignore`: `test/draft/*`
12+
plus the negations).
13+
14+
The exhibit is the one thing here that is NOT a test under development. Its
15+
three files are DEMONSTRATIONS: two pass with Graceful Restart unreachable, and
16+
the third MUST fail. A demonstration has no home in `test/`, because every
17+
directory a suite reads runs its files on every sweep, and these three would
18+
buy a gate nothing (`plan/journal/unwired-feature.md`, 2026-09-08). They live
19+
here for the property this directory already has, which is that no gate reads
20+
them. Run them with `ze-test bgp plugin --draft --pattern gr-vacuity`.
921

1022
## Why this exists
1123

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
# Exhibit: the control. This file MUST FAIL, and that is what it is for.
2+
#
3+
# It is gr-vacuity-mark-stale-without-gr.ci with one octet changed in the
4+
# conn=2 expectation: C0A899 in place of C0A801, so it asks for
5+
# 192.168.153.0/24 where ze announces 192.168.1.0/24.
6+
#
7+
# DEMONSTRATES: the harness discriminates. Its two sibling exhibits pass with
8+
# Graceful Restart unreachable because the assertion is insensitive to GR, and
9+
# NOT because .ci runs in this shape always report green. Without this control
10+
# a reader could take the passes for a broken runner.
11+
#
12+
# FAILS ON: the conn=2 NLRI comparison --
13+
# "NLRI: expected=192.168.153.0/24, got=192.168.1.0/24".
14+
# A run of this file that PASSES, or that fails anywhere else, means the
15+
# sibling exhibits' verdicts no longer carry the meaning stated in them.
16+
#
17+
# Journal row: plan/journal/unwired-feature.md, 2026-09-08.
18+
# Handover: plan/handoff-graceful-restart-sender-facts.md.
19+
20+
# ze-peer expectations
21+
stdin=peer:terminator=EOF_PEER
22+
option=tcp_connections:value=2
23+
option=timeout:value=15s
24+
25+
# Connection 1: Static route + EOR from ze
26+
expect=bgp:conn=1:seq=1:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF00300200000015400101004002004003040A0000014005040000006418C0A801
27+
expect=bgp:conn=1:seq=1:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF00170200000000
28+
# TCP close (no NOTIFICATION). No gr plugin is loaded, so nothing activates:
29+
# this is the stimulus the original test credits GR with answering.
30+
action=close:conn=1:seq=2
31+
32+
# Connection 2: ribOut replay — static route (config + RIB replay) + EOR
33+
expect=bgp:conn=2:seq=1:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF00300200000015400101004002004003040A0000014005040000006418C0A899
34+
expect=bgp:conn=2:seq=1:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF00170200000000
35+
EOF_PEER
36+
37+
# ze bgp configuration
38+
stdin=ze-bgp:terminator=EOF_CONF
39+
plugin {
40+
internal bgp-rib {
41+
use bgp-rib
42+
}
43+
}
44+
45+
bgp {
46+
peer peer1 {
47+
connection {
48+
remote {
49+
ip 127.0.0.1
50+
}
51+
local {
52+
ip 127.0.0.1
53+
accept false
54+
}
55+
}
56+
session {
57+
asn {
58+
local 1
59+
remote 1
60+
}
61+
router-id 1.2.3.4
62+
family {
63+
ipv4/unicast { prefix { maximum 10000; } }
64+
}
65+
capability {
66+
graceful-restart {
67+
restart-time 120
68+
}
69+
}
70+
}
71+
behavior {
72+
group-updates disable
73+
}
74+
timer { connect-retry 5; }
75+
76+
77+
update {
78+
attribute {
79+
origin igp
80+
local-preference 100
81+
next-hop 10.0.0.1
82+
}
83+
nlri {
84+
ipv4/unicast add 192.168.1.0/24
85+
}
86+
}
87+
attach process bgp-rib {
88+
receive [ update state refresh ]
89+
send [ update ]
90+
}
91+
}
92+
}
93+
EOF_CONF
94+
95+
# Process orchestration
96+
cmd=background:seq=1:exec=ze-peer --port $PORT:stdin=peer
97+
cmd=foreground:seq=2:exec=ze -:stdin=ze-bgp:timeout=20s
Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
# Exhibit: test/plugin/llgr-transition.ci passes with Graceful Restart unreachable
2+
#
3+
# This file is a DEMONSTRATION, not a regression test. It is a copy of
4+
# test/plugin/llgr-transition.ci with one change: the `gr` plugin is not loaded,
5+
# so neither Graceful Restart nor Long-Lived Graceful Restart code runs at all.
6+
# It still PASSES.
7+
#
8+
# DEMONSTRATES: the assertion in llgr-transition.ci is not about LLGR. That test
9+
# claims routes survive restart-time expiry because LLGR retained them. The
10+
# conn=2 re-announcement it reads is RIBManager.handleState
11+
# (internal/component/bgp/plugins/rib/rib.go) replaying ribOut, which peer-down
12+
# never deletes. With no gr plugin there is no restart-time timer to expire and
13+
# no LLGR state to retain anything, and the same bytes still arrive.
14+
#
15+
# ASSERTS ABOUT LONG-LIVED GRACEFUL RESTART: nothing. Its claimed AC-3 and AC-8
16+
# (restart-time expiry entering an LLGR period instead of purging) are
17+
# unproven by it.
18+
#
19+
# Journal row: plan/journal/unwired-feature.md, 2026-09-08.
20+
# Handover: plan/handoff-graceful-restart-sender-facts.md.
21+
22+
# ze-peer expectations
23+
stdin=peer:terminator=EOF_PEER
24+
option=tcp_connections:value=2
25+
option=timeout:value=15s
26+
27+
# Connection 1: Static route + EOR from ze
28+
expect=bgp:conn=1:seq=1:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF00300200000015400101004002004003040A0000014005040000006418C0A801
29+
expect=bgp:conn=1:seq=1:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF00170200000000
30+
# TCP close (no NOTIFICATION). No gr plugin is loaded, so nothing activates:
31+
# this is the stimulus the original test credits GR with answering.
32+
action=close:conn=1:seq=2
33+
34+
# Connection 2: ribOut replay — static route (config + RIB replay) + EOR
35+
# The same bytes arrive with no LLGR state in the process to retain anything.
36+
expect=bgp:conn=2:seq=1:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF00300200000015400101004002004003040A0000014005040000006418C0A801
37+
expect=bgp:conn=2:seq=1:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF00170200000000
38+
EOF_PEER
39+
40+
# ze bgp configuration
41+
stdin=ze-bgp:terminator=EOF_CONF
42+
plugin {
43+
internal bgp-rib {
44+
use bgp-rib
45+
}
46+
}
47+
48+
bgp {
49+
peer peer1 {
50+
connection {
51+
remote {
52+
ip 127.0.0.1
53+
}
54+
local {
55+
ip 127.0.0.1
56+
accept false
57+
}
58+
}
59+
session {
60+
asn {
61+
local 1
62+
remote 1
63+
}
64+
router-id 1.2.3.4
65+
family {
66+
ipv4/unicast { prefix { maximum 10000; } }
67+
}
68+
capability {
69+
graceful-restart {
70+
restart-time 1
71+
long-lived-stale-time 60
72+
}
73+
}
74+
}
75+
behavior {
76+
group-updates disable
77+
}
78+
timer { connect-retry 3; }
79+
80+
update {
81+
attribute {
82+
origin igp
83+
local-preference 100
84+
next-hop 10.0.0.1
85+
}
86+
nlri {
87+
ipv4/unicast add 192.168.1.0/24
88+
}
89+
}
90+
attach process bgp-rib {
91+
receive [ update state refresh ]
92+
send [ update ]
93+
}
94+
}
95+
}
96+
EOF_CONF
97+
98+
# Process orchestration
99+
cmd=background:seq=1:exec=ze-peer --port $PORT:stdin=peer
100+
cmd=foreground:seq=2:exec=ze -:stdin=ze-bgp:timeout=20s
Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
# Exhibit: test/plugin/gr-mark-stale.ci passes with Graceful Restart unreachable
2+
#
3+
# This file is a DEMONSTRATION, not a regression test. It is a copy of
4+
# test/plugin/gr-mark-stale.ci with one change: the `gr` plugin is not loaded,
5+
# so no Graceful Restart code runs at all. It still PASSES.
6+
#
7+
# DEMONSTRATES: the assertion in gr-mark-stale.ci is not about Graceful Restart.
8+
# The conn=2 re-announcement that test reads as GR evidence is produced by
9+
# RIBManager.handleState (internal/component/bgp/plugins/rib/rib.go) replaying
10+
# ribOut, which is deleted in the withdraw path and never on peer-down. The
11+
# replay happens whether or not GR exists.
12+
# DEMONSTRATES: mark-stale is not reached even when the plugin IS loaded.
13+
# parseGRCapValue (internal/component/bgp/plugins/gr/gr.go) emits two octets
14+
# with no AFI/SAFI tuples, so the capability carries zero families, decodeGR
15+
# yields nil families, and onSessionDown returns false at its
16+
# len(staleFamilies) == 0 guard.
17+
#
18+
# ASSERTS ABOUT GRACEFUL RESTART: nothing. Green here and green in
19+
# gr-mark-stale.ci are the same green, produced by the same RIB replay.
20+
#
21+
# Journal row: plan/journal/unwired-feature.md, 2026-09-08.
22+
# Handover: plan/handoff-graceful-restart-sender-facts.md.
23+
24+
# ze-peer expectations
25+
stdin=peer:terminator=EOF_PEER
26+
option=tcp_connections:value=2
27+
option=timeout:value=15s
28+
29+
# Connection 1: Static route + EOR from ze
30+
expect=bgp:conn=1:seq=1:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF00300200000015400101004002004003040A0000014005040000006418C0A801
31+
expect=bgp:conn=1:seq=1:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF00170200000000
32+
# TCP close (no NOTIFICATION). No gr plugin is loaded, so nothing activates:
33+
# this is the stimulus the original test credits GR with answering.
34+
action=close:conn=1:seq=2
35+
36+
# Connection 2: ribOut replay — static route (config + RIB replay) + EOR
37+
expect=bgp:conn=2:seq=1:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF00300200000015400101004002004003040A0000014005040000006418C0A801
38+
expect=bgp:conn=2:seq=1:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF00170200000000
39+
EOF_PEER
40+
41+
# ze bgp configuration
42+
stdin=ze-bgp:terminator=EOF_CONF
43+
plugin {
44+
internal bgp-rib {
45+
use bgp-rib
46+
}
47+
}
48+
49+
bgp {
50+
peer peer1 {
51+
connection {
52+
remote {
53+
ip 127.0.0.1
54+
}
55+
local {
56+
ip 127.0.0.1
57+
accept false
58+
}
59+
}
60+
session {
61+
asn {
62+
local 1
63+
remote 1
64+
}
65+
router-id 1.2.3.4
66+
family {
67+
ipv4/unicast { prefix { maximum 10000; } }
68+
}
69+
capability {
70+
graceful-restart {
71+
restart-time 120
72+
}
73+
}
74+
}
75+
behavior {
76+
group-updates disable
77+
}
78+
timer { connect-retry 5; }
79+
80+
81+
update {
82+
attribute {
83+
origin igp
84+
local-preference 100
85+
next-hop 10.0.0.1
86+
}
87+
nlri {
88+
ipv4/unicast add 192.168.1.0/24
89+
}
90+
}
91+
attach process bgp-rib {
92+
receive [ update state refresh ]
93+
send [ update ]
94+
}
95+
}
96+
}
97+
EOF_CONF
98+
99+
# Process orchestration
100+
cmd=background:seq=1:exec=ze-peer --port $PORT:stdin=peer
101+
cmd=foreground:seq=2:exec=ze -:stdin=ze-bgp:timeout=20s

0 commit comments

Comments
 (0)