Skip to content

Commit 10663c5

Browse files
author
SqlRush
committed
test(cluster): pin the gated rtvis self-heal as a RED->GREEN twin (t/403 L10)
One driver, no cleanout bypass: a node0 xid whose TT slot is provably lapped (>=2 pool rollovers, hard-floor assert) is read from node1 twice. RED twin under the default cluster.crossnode_runtime_visibility = off: the read MUST fail closed 53R97 ("cluster TT status unknown") with node1's rtvis_verdict_wire_count provably flat -- a succeeding read is a hard fail, never retried, so the GREEN result cannot come from an overlay / hint / delayed-cleanout side door. GREEN twin after flipping ONLY the GUC (PGC_SUSET reload, both nodes): the same read returns the exact terminal 64|2080 and node1's verdict counters move (wire=64, below_horizon=64) -- every row resolved through the origin-verdict COMMITTED_BELOW_HORIZON arm (cluster_runtime_visibility.c). cluster.tt_status_hint_emit_mode is disabled (PGC_SIGHUP) before the fixture write so the V4 hint wire cannot install the terminal status into node1's TT overlay at commit time: this deterministically manufactures the "hint never arrived" state the resolve path exists for (the outbound hint ring drops under storm load anyway) -- it forces the mechanism under test, it does not weaken any assertion. Known structural limit (registered follow-up, not this leg): with cluster.xid_striping off (this rig) classify_ref's recycled-ref derivation branch is unreachable -- only the fresh-ref AUTHORITATIVE ask is served; a striping-on variant leg needs a PGC_POSTMASTER re-provision. Gates: t/403 33/33 x3 independent runs; L1-L9 assertions unweakened; test-only diff. Spec: spec-5.22e-undo-cluster-retention-horizon.md
1 parent df93b68 commit 10663c5

1 file changed

Lines changed: 240 additions & 0 deletions

File tree

‎src/test/cluster_tap/t/403_undo_idle_peer_floor_pin_2node.pl‎

Lines changed: 240 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,14 @@
5353
# L8 wake-up self-heal: node1's first cross-node read after the
5454
# storm succeeds (bounded retries allowed: the inadmissible
5555
# BELOW_HORIZON arm is fail-closed and observe heals it)
56+
# L10 GATED self-heal proof (runs before L9), RED->GREEN twin
57+
# on ONE driver with NO cleanout bypass: a recycled node0
58+
# xid first fails closed 53R97 under the default
59+
# cluster.crossnode_runtime_visibility = off (verdict wire
60+
# provably untouched), then -- GUC flipped on, nothing else
61+
# changed -- resolves through the COMMITTED_BELOW_HORIZON
62+
# origin-verdict arm: exact data terminal + rtvis verdict
63+
# counter evidence
5664
# L9 restart clean
5765
#
5866
# Spec: spec-5.22e (cluster undo retention brake) S2.1 sampling rule +
@@ -500,6 +508,238 @@ sub floor_lag
500508
}
501509
is($sum, '64|t', "L8 node1 cross-node read healed (tries=$tries)");
502510

511+
# ============================================================
512+
# L10: the GATED wake-up self-heal, for real -- the
513+
# COMMITTED_BELOW_HORIZON resolve arm under
514+
# cluster.crossnode_runtime_visibility = on, with NO owner-side
515+
# cleanout bypass (no FREEZE, no CHECKPOINT).
516+
#
517+
# L8 above only proves the read heals AFTER a VACUUM (FREEZE)
518+
# removed the remote ITL refs -- the rtvis resolve machinery
519+
# (cluster_runtime_visibility.c rtvis_try_origin_verdict) never ran
520+
# there because the GUC defaults off. This leg drives it as a
521+
# RED->GREEN twin on ONE driver:
522+
#
523+
# 1. cluster.tt_status_hint_emit_mode = disabled (PGC_SIGHUP,
524+
# default all_status) BEFORE the fixture write: otherwise the
525+
# V4 hint wire installs the fixture xid's terminal status into
526+
# node1's TT overlay at commit time and the read resolves from
527+
# the overlay without ever touching the verdict wire. This is
528+
# NOT an assertion-weakening bypass: it deterministically
529+
# manufactures the "hint never arrived" state the resolve path
530+
# exists for (the 256-slot outbound ring drops hints under
531+
# storm load anyway), FORCING the mechanism under test to carry
532+
# the read. cluster.crossnode_runtime_visibility stays at its
533+
# default (off) until the RED twin below has run.
534+
# 2. Fixture: ONE node0 transaction inserts 64 rows -> a single
535+
# fresh remote ITL ref (local_xid == raw_xid). With
536+
# cluster.xid_striping off in this rig, cluster_xid_is_mine()
537+
# is always false at the origin, so ONLY the spec-5.22f
538+
# fresh-ref AUTHORITATIVE ask is ever served -- the fixture must
539+
# stay fresh (single writer xact, page never revisited by
540+
# node0, so no delayed cleanout stamps it).
541+
# 3. Churn: a node0 writer laps the TT pool (same
542+
# tt_retention_rollover_count evidence as L2), so the fixture
543+
# xid's TT slot is provably rebound; the origin's complete
544+
# by-xid scan then 0-matches and -- retention legs (a)-(d) +
545+
# CLOG committed -- serves COMMITTED_BELOW_HORIZON{H}.
546+
# 4. RED twin: with the GUC still off, node1's read of the fixture
547+
# MUST fail closed 53R97 ("cluster TT status unknown",
548+
# heapam_visibility.c) and node1's verdict-wire counter MUST
549+
# stay flat (cluster_runtime_visibility.c gates the whole
550+
# resolve on the GUC before any wire touch). This proves the
551+
# recycled-slot window is genuinely open at the moment the
552+
# GREEN twin runs -- the GREEN result cannot be an overlay /
553+
# hint / cleanout side door.
554+
# 5. cluster.crossnode_runtime_visibility = on (PGC_SUSET, reload)
555+
# on BOTH nodes: the requester's classify / fresh-ref widening
556+
# gates on it AND the origin's lms_undo_verdict_serve refuses
557+
# while it is off (cluster_cr_server.c).
558+
# 6. GREEN twin: node1 re-reads the same fixture. A snapshot
559+
# read_scn behind the shipped horizon takes the inadmissible
560+
# fail-closed arm and the Lamport observe heals the NEXT
561+
# snapshot (bounded retries, the documented self-heal); the
562+
# healed snapshot admits the bound and every row resolves
563+
# visible.
564+
#
565+
# HARD ASSERTS: RED twin fails closed with the wire flat + exact
566+
# terminal data on the GREEN twin (count=64 AND sum=2080 -- not
567+
# L8's sum >= 0 shape) + node1's rtvis verdict counters moved
568+
# (below_horizon / exact), proving the verdict wire resolved the
569+
# read rather than an overlay / hint / cleanout side door.
570+
#
571+
# Known hazard (documented, not worked around): if L8 had to bounce
572+
# node1, the pre-existing rejoin silent-empty-read defect (see L8
573+
# comment) can surface here as count=0 -- that is a REAL defect this
574+
# leg must fail on, not paper over.
575+
# ============================================================
576+
for my $node ($pair->node0, $pair->node1)
577+
{
578+
$node->safe_psql('postgres',
579+
"ALTER SYSTEM SET cluster.tt_status_hint_emit_mode = 'disabled'");
580+
$node->safe_psql('postgres', 'SELECT pg_reload_conf()');
581+
}
582+
ok( wait_for(
583+
sub {
584+
$pair->node0->safe_psql('postgres',
585+
'SHOW cluster.tt_status_hint_emit_mode') eq 'disabled'
586+
&& $pair->node1->safe_psql('postgres',
587+
'SHOW cluster.tt_status_hint_emit_mode') eq 'disabled';
588+
},
589+
15),
590+
'L10a status-hint wire disabled via reload (both nodes)');
591+
592+
# Coinciding-filepath fixture on shared storage (same t/394 recipe as
593+
# L1b; no shared catalog in this rig).
594+
my $l10tbl;
595+
for my $i (1 .. 12)
596+
{
597+
my $t = "t403l10_$i";
598+
$_->safe_psql('postgres', "CREATE TABLE $t (k int, v int)")
599+
for ($pair->node0, $pair->node1);
600+
my $p0 = $pair->node0->safe_psql('postgres', "SELECT pg_relation_filepath('$t')");
601+
my $p1 = $pair->node1->safe_psql('postgres', "SELECT pg_relation_filepath('$t')");
602+
if (($p0 // '') eq ($p1 // '')) { $l10tbl = $t; last; }
603+
}
604+
ok(defined $l10tbl, 'L10b coinciding-filepath fixture table found');
605+
die 'no coinciding filepath found for L10' unless defined $l10tbl;
606+
diag("L10 fixture table=$l10tbl");
607+
608+
# ONE transaction writes all 64 rows: exact expected terminal data is
609+
# count=64, sum(v)=sum(1..64)=2080, and the page ITL stays a single
610+
# fresh entry bound to this xid.
611+
$pair->node0->safe_psql('postgres',
612+
"INSERT INTO $l10tbl SELECT g, g FROM generate_series(1, 64) g");
613+
614+
# Churn phase: lap the TT pool on node0 so the fixture xid's slot is
615+
# rebound (>= 2 laps hard floor, target 3; single background writer on
616+
# a node0-only table so the fixture pages are never revisited).
617+
$pair->node0->safe_psql('postgres', 'CREATE TABLE t403l10_churn (k int)');
618+
my $churnfile = "$tmpdir/churn.sql";
619+
{
620+
open(my $fh, '>', $churnfile) or die "cannot write $churnfile: $!";
621+
print $fh "INSERT INTO t403l10_churn VALUES (1);\n" for (1 .. $WRITER_LINES);
622+
close($fh);
623+
}
624+
my $roll_pre_l10 = state_val($pair->node0, 'undo', 'tt_retention_rollover_count');
625+
my ($c_in, $c_out, $c_err) = ('', '', '');
626+
my $churn = IPC::Run::start(
627+
[
628+
'psql', '-X', '-A', '-t', '-q',
629+
'-d', $pair->node0->connstr('postgres'),
630+
'-f', $churnfile
631+
],
632+
\$c_in, \$c_out, \$c_err);
633+
my $churn_t0 = time();
634+
my $roll_delta_l10 = 0;
635+
while (time() - $churn_t0 < 120)
636+
{
637+
$roll_delta_l10 =
638+
state_val($pair->node0, 'undo', 'tt_retention_rollover_count') - $roll_pre_l10;
639+
last if $roll_delta_l10 >= 3 * $POOL_SEGMENTS;
640+
usleep(400_000);
641+
}
642+
$churn->kill_kill;
643+
diag(sprintf('L10 churn: %d TT rollovers in %.0fs (hard floor %d)',
644+
$roll_delta_l10, time() - $churn_t0, 2 * $POOL_SEGMENTS));
645+
cmp_ok($roll_delta_l10, '>=', 2 * $POOL_SEGMENTS,
646+
'L10c churn lapped the TT pool (fixture slot provably rebound)');
647+
648+
# ------------------------------------------------------------
649+
# RED twin: cluster.crossnode_runtime_visibility is still at its
650+
# default (off). The same read the GREEN twin will heal below MUST
651+
# fail closed here -- 53R97 "cluster TT status unknown"
652+
# (heapam_visibility.c) -- and node1's verdict-wire counter MUST stay
653+
# flat: cluster_runtime_visibility.c refuses before any wire touch
654+
# while the GUC is off, so a moving counter (or a succeeding read)
655+
# would mean a side door carried it. Transient non-53R97 errors
656+
# (post-churn GES tail, the L8 class) are retried; a SUCCEEDING read
657+
# is a hard fail, never retried.
658+
# ------------------------------------------------------------
659+
my $wire_pre_neg = state_val($pair->node1, 'cr', 'rtvis_verdict_wire_count');
660+
my ($neg_hit_53r97, $neg_unexpected) = (0, '');
661+
for my $i (1 .. 10)
662+
{
663+
my ($rc, $stdout, $stderr) =
664+
$pair->node1->psql('postgres', "SELECT count(*), sum(v) FROM $l10tbl");
665+
if ($rc == 0)
666+
{
667+
$neg_unexpected = 'read SUCCEEDED with rtvis off: [' . ($stdout // '') . ']';
668+
last;
669+
}
670+
if (($stderr // '') =~ /cluster TT status unknown for xid/)
671+
{
672+
$neg_hit_53r97 = 1;
673+
last;
674+
}
675+
$neg_unexpected = join(' / ', split(/\n/, $stderr // ''));
676+
diag(sprintf('L10d RED-twin attempt %d: rc=%d non-53R97 err=[%s]',
677+
$i, $rc, $neg_unexpected));
678+
usleep(1_500_000);
679+
}
680+
ok($neg_hit_53r97,
681+
'L10d RED twin HARD ASSERT: rtvis off fails the read closed 53R97 '
682+
. '(cluster TT status unknown)')
683+
or diag("L10d terminal state: $neg_unexpected");
684+
is(state_val($pair->node1, 'cr', 'rtvis_verdict_wire_count') - $wire_pre_neg,
685+
0, 'L10e RED twin: verdict wire provably untouched while the GUC is off');
686+
687+
# GREEN twin arming: flip ONLY the GUC; driver and fixture unchanged.
688+
for my $node ($pair->node0, $pair->node1)
689+
{
690+
$node->safe_psql('postgres',
691+
'ALTER SYSTEM SET cluster.crossnode_runtime_visibility = on');
692+
$node->safe_psql('postgres', 'SELECT pg_reload_conf()');
693+
}
694+
ok( wait_for(
695+
sub {
696+
$pair->node0->safe_psql('postgres',
697+
'SHOW cluster.crossnode_runtime_visibility') eq 'on'
698+
&& $pair->node1->safe_psql('postgres',
699+
'SHOW cluster.crossnode_runtime_visibility') eq 'on';
700+
},
701+
15),
702+
'L10f crossnode_runtime_visibility on via reload (both nodes)');
703+
704+
# Requester-side rtvis verdict counters live on node1 (category 'cr').
705+
my $bh_pre = state_val($pair->node1, 'cr', 'rtvis_verdict_below_horizon_count');
706+
my $ex_pre = state_val($pair->node1, 'cr', 'rtvis_verdict_exact_count');
707+
my $inadm_pre = state_val($pair->node1, 'cr', 'rtvis_verdict_inadmissible_count');
708+
my $wire_pre = state_val($pair->node1, 'cr', 'rtvis_verdict_wire_count');
709+
710+
# node1 cross-node read, bounded retries: the inadmissible
711+
# BELOW_HORIZON arm is fail-closed and the wire observe heals the next
712+
# snapshot (plus post-churn GES tail tolerance, as in L8).
713+
my ($l10_out, $l10_tries) = ('', 0);
714+
for my $i (1 .. 15)
715+
{
716+
$l10_tries = $i;
717+
my ($rc, $stdout, $stderr) =
718+
$pair->node1->psql('postgres', "SELECT count(*), sum(v) FROM $l10tbl");
719+
if ($rc == 0 && $stdout =~ /^64\|2080$/)
720+
{
721+
$l10_out = $stdout;
722+
last;
723+
}
724+
diag(sprintf('L10 attempt %d: rc=%d out=[%s] err=[%s]',
725+
$i, $rc, $stdout // '', join(' / ', split(/\n/, $stderr // ''))));
726+
usleep(1_500_000);
727+
}
728+
is($l10_out, '64|2080',
729+
"L10 HARD ASSERT: node1 healed read returned exact terminal data (tries=$l10_tries)");
730+
731+
my $bh_d = state_val($pair->node1, 'cr', 'rtvis_verdict_below_horizon_count') - $bh_pre;
732+
my $ex_d = state_val($pair->node1, 'cr', 'rtvis_verdict_exact_count') - $ex_pre;
733+
my $inadm_d = state_val($pair->node1, 'cr', 'rtvis_verdict_inadmissible_count') - $inadm_pre;
734+
my $wire_d = state_val($pair->node1, 'cr', 'rtvis_verdict_wire_count') - $wire_pre;
735+
diag("L10 node1 rtvis verdict deltas: wire=$wire_d below_horizon=$bh_d "
736+
. "exact=$ex_d inadmissible=$inadm_d");
737+
cmp_ok($bh_d + $ex_d, '>', 0,
738+
'L10g HARD ASSERT: verdict resolve path really carried the read '
739+
. '(below_horizon + exact advanced)');
740+
cmp_ok($bh_d, '>', 0,
741+
'L10h below-horizon arm served the recycled fixture xid');
742+
503743
# ============================================================
504744
# L9: restart clean.
505745
# ============================================================

0 commit comments

Comments
 (0)