-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathverify.py
More file actions
executable file
·2527 lines (2251 loc) · 114 KB
/
Copy pathverify.py
File metadata and controls
executable file
·2527 lines (2251 loc) · 114 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
#!/usr/bin/env python3
"""
verify.py -- prove the guards are actually live, and actually blocking.
python3 verify.py --target /path/to/your-agent-project
Covers both hooks this repo installs: gate_guard.py (is this action allowed?)
and budget_guard.py (has this session already cost more than you agreed to
spend, and is it still making progress?). A project that installed only the
approval gate is fine -- the budget section reports SKIP, not FAIL.
A PreToolUse hook fails silently. If the path in settings.json is wrong, if
the config didn't resolve, if the harness never reloaded its settings, you
get exactly the same experience as a hook that is working perfectly: nothing
visibly happens. You find out it was never running the first time your agent
does the thing it was supposed to be stopped from doing.
So this script does two separate things:
WIRING -- reads .claude/settings.json, finds the registered hook command,
and checks that every file it depends on exists and parses.
BEHAVIOR -- takes that exact registered command and feeds it real probe
payloads on stdin, the same shape the harness sends, then reads
the exit code. Exit 2 is a block. Exit 0 is an allow. This is
an end-to-end test of the thing your harness will actually run,
not a unit test of an imported function.
There is a third mode, --evidence, which answers a different question from
the other two. WIRING and BEHAVIOR ask "is this correct now?", for the person
who just changed something, and their answer evaporates with the terminal
scrollback. --evidence asks "was this control operating for the whole period,
and not only at the moment somebody checked?" -- and renders the answer as a
dated, self-contained artifact you can hand to someone who was not in the
room: the window covered, how many tool calls the harness routed through the
gate, what it blocked and when, which exact files were running by digest, and
a section of equal weight on what none of it proves. That last section is not
a disclaimer. An evidence artifact that overstates itself is worth less than
no artifact, because the first competent reader who finds the overstatement
stops believing the rest of it.
Both directions are checked. Probes that SHOULD block are the obvious half;
probes that should be ALLOWED matter just as much, because an over-blocking
gate that fights every tool call is a gate you will turn off within a week,
and then you have no gate at all.
A fourth mode, --over-blocks, follows that thought into your own history
instead of a probe set. It reads the blocks the gate actually recorded and
asks, of each one, whether the tool that was blocked could have carried out
the action the rule exists to stop. Writing a file cannot move funds;
fetching a URL cannot open an account. Blocks of that shape prevented
nothing, and their count is the closest thing to an honest price tag on
running these rules unattended. It is deliberately a floor: anything the
model does not recognise is reported as not adjudicable rather than counted,
and the model itself is printed with the result so you can disagree with it.
Probes are derived from the rule ids present in your config. If you replaced
the default rule pack, probes for rules you removed are reported as SKIP
rather than counted as failures -- your rules are yours.
Blocked gate probes append to your configured blocked_log, because that is
what the hook does and suppressing it would mean testing something other than
production. Every probe command contains the string "gate-verify-probe" so
you can filter them back out of the audit trail.
BUDGET PROBES ARE ISOLATED, AND FOR A REASON WORTH STATING. A gate probe's
only side effect is an audit line. A budget probe's is not: budget_guard.py
records each session's cost into a shared daily rollup under its state_dir,
and that rollup decides whether the NEXT call is blocked. Probing a $20
synthetic session against your real state_dir would leave $20 of imaginary
spend sitting in today's total for the rest of the day -- quite possibly
enough to trip your daily ceiling and stop your actual agent. So every budget
probe runs the registered script against a throwaway config in a temp
directory, with only state_dir and blocked_log redirected; the ceilings, the
price table and the loop thresholds are your real ones, read from your real
config. Your spend ledger is never written to. The temp directory is removed
when the run finishes.
Three of the budget probes test arithmetic rather than plumbing, because the
cost ceiling is only as trustworthy as the pricing underneath it:
- Cache reads bill at 0.1x the input rate. Priced at the full input rate --
the obvious shortcut -- a mostly-cached agent session overstates by
roughly 10x and your ceiling stops meaning anything.
- A streamed assistant message is rewritten to the transcript as it grows,
so summing line by line double-counts. Usage is deduplicated by message id.
- An unrecognised model ID -- which is what a newly released model looks
like -- must still be billed, or it sails past the ceiling for free.
Each is probed by constructing a transcript that lands on the safe side of
your ceiling only if that rule is implemented correctly.
A note on how the probe strings below are built. Several are assembled from
fragments at runtime rather than written as single literals -- the payment
domains, the credential words, the fake token. That is not obfuscation, and
the reason is worth knowing before you write your own probes: a strict gate
pointed at a codebase containing this file will block an agent from writing
this file, because the probe payloads match the very rules they exercise.
This exact file was blocked by its own project's hook on the first attempt to
write it (rule PAYMENT_API_WRITE, on a line containing a stripe API URL).
gate_guard.py carries the same note about its key-material pattern. It is a
real, permanent property of content-matching guards, it will happen to you,
and the fix is to split the literal -- never to loosen the rule.
"""
import argparse
import hashlib
import json
import os
import re
import shutil
import subprocess
import sys
import tempfile
from datetime import datetime, timezone
# --- probe payload fragments (see the docstring's note on the splits) ---
_STRIPE = "api." + "stri" + "pe.com"
_REGISTRAR = "name" + "cheap.com"
STRIPE_POST = (
"curl -X POST https://" + _STRIPE + "/v1/charges "
"-d amount=500 # gate-verify-probe"
)
STRIPE_GET = "curl -s https://" + _STRIPE + "/v1/charges # gate-verify-probe"
PKG_INSTALL = "npm" + " install left-pad # gate-verify-probe"
SIGNUP_URL = "curl https://gate-verify.invalid/" + "signup # gate-verify-probe"
_AMOUNT = "5 " + "usd" + "c"
FUND_MOVE = "solana transfer --amount " + _AMOUNT + " --to acct # gate-verify-probe"
CONTRACT = "forge " + "create MyToken --rpc-url x # gate-verify-probe"
REGISTRAR = "curl " + _REGISTRAR + " cart checkout # gate-verify-probe"
KEY_MATERIAL = "cat ~/.config/" + "pri" + "vate_" + "key" + " # gate-verify-probe"
FAKE_TOKEN = "echo " + "sk-" + ("A" * 26) + " # gate-verify-probe"
UNAPPROVED_PUSH = (
"git push https://gate-verify.invalid/unapproved.git main"
" # gate-verify-probe"
)
PASS, FAIL, SKIP, WARN = "PASS", "FAIL", "SKIP", "WARN"
def bash(cmd):
return {"tool_name": "Bash", "tool_input": {"command": cmd}}
def build_probes(config, state_abspath):
"""Probes are (rule_id, description, payload, expect). rule_id None means
the probe isn't tied to a configurable rule and always applies."""
state_base = os.path.basename(config.get("state_path", "STATE.json"))
return [
# --- should block ---
("PAYMENT_API_WRITE", "POST to a payment API",
bash(STRIPE_POST), "block"),
("ACCOUNT_SIGNUP_FLOW", "opening a signup URL",
bash(SIGNUP_URL), "block"),
("FUND_MOVEMENT", "moving funds out of a wallet",
bash(FUND_MOVE), "block"),
("CONTRACT_DEPLOY", "deploying a contract",
bash(CONTRACT), "block"),
("KEY_MATERIAL", "reading offline signing material",
bash(KEY_MATERIAL), "block"),
("DOMAIN_REGISTRAR", "buying a domain",
bash(REGISTRAR), "block"),
("PACKAGE_INSTALL", "installing a package",
bash(PKG_INSTALL), "block"),
(None, "live-credential-shaped string in a command",
bash(FAKE_TOKEN), "block"),
(None, "push to an unapproved remote",
bash(UNAPPROVED_PUSH), "block"),
(None, f"Write tool targeting {state_base} (protected)",
{"tool_name": "Write",
"tool_input": {"file_path": state_abspath, "content": "{}"}},
"block"),
(None, f"shell redirect into {state_base} (protected)",
bash("echo probe > " + state_base + " # gate-verify-probe"),
"block"),
# --- should be allowed: over-blocking is its own failure mode ---
(None, "ordinary shell command",
bash("ls -la # gate-verify-probe"), "allow"),
(None, "GET from a payment API (reading is research)",
bash(STRIPE_GET), "allow"),
(None, "Read tool on an ordinary file",
{"tool_name": "Read", "tool_input": {"file_path": "README.md"}}, "allow"),
(None, "Write tool on an unprotected file",
{"tool_name": "Write",
"tool_input": {"file_path": "notes/scratch.md", "content": "hello"}},
"allow"),
]
def _hook_entry(target, marker="gate_guard.py"):
"""The PreToolUse entry registering `marker`, as (entry, command, err).
Split out of find_hook_command so callers can also read the *matcher*.
Reading the command while discarding the matcher is exactly how a narrowed
matcher gets misdiagnosed as a harness bug: the matcher decides which tools
ever reach the guard at all, and no amount of correct guard code makes a
tool outside it produce a hook call.
"""
settings_path = os.path.join(target, ".claude", "settings.json")
if not os.path.isfile(settings_path):
return None, None, f"{settings_path} does not exist"
try:
with open(settings_path) as f:
settings = json.load(f)
except json.JSONDecodeError as e:
return None, None, f"settings.json is not valid JSON: {e}"
variants = (marker, marker.replace("_", "-"))
pre = (settings.get("hooks") or {}).get("PreToolUse") or []
for entry in pre:
for hook in entry.get("hooks", []) or []:
cmd = str(hook.get("command", ""))
if any(v in cmd for v in variants):
return entry, cmd, None
return None, None, f"no PreToolUse hook referencing {marker} is registered"
def find_hook_command(target, marker="gate_guard.py"):
"""Pull the registered hook command straight out of settings.json. Testing
anything else would be testing a hook your harness isn't running.
`marker` is the guard's script filename; both guards register their own
PreToolUse entry and are recognised by it.
"""
_entry, command, err = _hook_entry(target, marker)
return command, err
def find_hook_matcher(target, marker="gate_guard.py"):
"""The tool-name matcher on the entry that registers the guard.
Returns the matcher string, or None when it cannot be read at all. An
absent matcher key means match-everything, so it reports `"*"` rather than
None -- None is reserved for "could not determine", which callers must not
render as a warning.
"""
entry, _command, err = _hook_entry(target, marker)
if err or not isinstance(entry, dict):
return None
matcher = entry.get("matcher")
return "*" if matcher is None else str(matcher)
def matcher_covers_everything(matcher):
"""True when every tool reaches the guard.
Unknown (None) counts as True on purpose: this gates a warning, and a
warning fired because we could not read the config would be noise on a
healthy install.
"""
return matcher is None or str(matcher).strip() in ("", "*")
def wired_guard_path(command, target):
"""The gate_guard.py the registered hook command actually executes,
absolute. Returns None if the command doesn't name one."""
m = re.search(r"(\S*gate[_-]guard\.py)", command)
if not m:
return None
guard = m.group(1).strip("\"'")
return guard if os.path.isabs(guard) else os.path.join(target, guard)
def resolve_config(command, target, guard_path=None):
"""Resolve the config exactly the way gate_guard.py will: the env var in
the hook command first, then the project root, then next to the script.
The script-dir fallback matches gate_guard.find_config_path() and
resolve_budget_config(); without it this reported "not found -- running on
defaults" for a plugin-style install whose config sits beside the guard,
which is the same class of wrong answer everything else in this file
exists to stop.
"""
m = re.search(r"GATE_GUARD_CONFIG=(\"[^\"]+\"|'[^']+'|\S+)", command)
if m:
path = m.group(1).strip("\"'")
if not os.path.isabs(path):
path = os.path.join(target, path)
return path, "hook command"
root_path = os.path.join(target, "gate-guard.config.json")
if os.path.isfile(root_path) or not guard_path:
return root_path, "project root"
beside = os.path.join(os.path.dirname(os.path.abspath(guard_path)),
"gate-guard.config.json")
if os.path.isfile(beside):
return beside, "next to the script"
return root_path, "project root"
def check_wiring(target):
"""Returns (rows, command, config, state_abspath). command is None if the
wiring is broken badly enough that no probe can run."""
rows = []
command, err = find_hook_command(target)
if not command:
rows.append((FAIL, "hook registered in .claude/settings.json", err))
return rows, None, None, None
rows.append((PASS, "hook registered in .claude/settings.json", command))
guard = wired_guard_path(command, target) or ""
if os.path.isfile(guard):
rows.append((PASS, "hook script exists at the registered path", guard))
else:
rows.append((FAIL, "hook script exists at the registered path",
f"not found: {guard}"))
config_path, origin = resolve_config(command, target, guard)
row, user = config_row(config_path, origin)
rows.append(row)
config = user or {}
# Only meaningful against the guard that is actually installed at
# --target: an older guard has a different key set, and reporting a key
# it does not know about as a typo would be a fabricated finding.
module = load_guard_module(guard, "_gg_keys") if os.path.isfile(guard) \
else None
for extra in (config_keys_row(user, config_path, module),
rule_coverage_row(user, module)):
if extra:
rows.append(extra)
state_rel = config.get("state_path", "STATE.json")
state_abspath = os.path.join(target, state_rel)
rows.append(trust_tier_row(state_abspath, state_rel, config))
rows.append(allowlist_row(target, config))
rows.extend(recording_rows(target, command, config))
return rows, command, config, state_abspath
def recording_rows(target, command, config):
"""Can the guard write down what it did?
The budget half of this file has had these two rows since it shipped. The
gate half never did, and the gate half is the one whose product claim is
evidence. gate_guard.py swallows every bookkeeping failure by design -- so
an unwritable approvals/ directory produces a guard that blocks correctly
and records nothing, and --evidence then reports it as a control that never
ran. This is the check that catches that before an auditor does.
Severity is deliberately split. A heartbeat that cannot be written can
never establish that the control operated, so it is a FAIL. A block log
that cannot be written still leaves enforcement fully intact -- what is
lost is the audit trail -- so it is a WARN, matching check_budget_wiring.
"""
rows = []
hb = heartbeat_path(target, command, config)
if not hb:
rows.append((WARN, "heartbeat directory writable",
"heartbeat_path is disabled in your config -- liveness "
"and evidence cannot be established for this install"))
elif writable(os.path.dirname(hb) or "."):
rows.append((PASS, "heartbeat directory writable", os.path.dirname(hb)))
else:
rows.append((FAIL, "heartbeat directory writable",
f"{os.path.dirname(hb)} is not writable -- the guard "
f"still blocks, but it cannot record that it ran and "
f"--evidence will report it as never having run"))
log = project_path(target, command, config, "blocked_log",
"approvals/blocked.jsonl")
if not log:
rows.append((WARN, "blocked-log directory writable",
"blocked_log is disabled in your config -- blocks are "
"enforced but nothing is written down"))
elif writable(os.path.dirname(log) or "."):
rows.append((PASS, "blocked-log directory writable", os.path.dirname(log)))
else:
rows.append((WARN, "blocked-log directory writable",
f"{os.path.dirname(log)} is not writable -- blocks still "
f"happen and still stop the call, but no audit trail "
f"accumulates and --evidence will show zero of them"))
return rows
def config_row(config_path, origin):
"""One row describing whether the config on disk is the config the guard
will run on. Returns (row, parsed_or_None).
Mirrors gate_guard.read_config()'s states. The three failure states behave
identically -- built-in defaults, every default rule enforced -- and are
fixed completely differently, which is the same reason the tier and
allowlist rows were split.
"""
label = f"config resolves (via {origin})"
if not os.path.isfile(config_path):
if os.path.isdir(config_path):
return (WARN, label,
f"{config_path} is a directory, not a file -- the hook "
f"falls back to its built-in defaults, NOT your rules"), None
return (WARN, label,
f"{config_path} not found -- the hook falls back to its "
f"built-in defaults, NOT your rules"), None
try:
with open(config_path) as f:
parsed = json.load(f)
except ValueError as e:
return (FAIL, label,
f"{config_path}: invalid JSON: {e} -- the hook falls back to "
f"its built-in defaults, so NONE of your rules are in force"), None
except (OSError, UnicodeDecodeError) as e:
return (FAIL, label,
f"{config_path}: {type(e).__name__}: {e} -- the hook falls "
f"back to its built-in defaults, so NONE of your rules are in "
f"force"), None
if not isinstance(parsed, dict):
return (FAIL, label,
f"{config_path}: top level is {type(parsed).__name__}, not an "
f"object -- the hook falls back to its built-in defaults, so "
f"NONE of your rules are in force"), None
return (PASS, label, config_path), parsed
def config_keys_row(user, config_path, module):
"""One row for keys the config sets that nothing in the install reads.
A misspelled key is not an error anywhere: `dict.update()` accepts it,
the guard ignores it, and the setting the operator meant to change stays
at its default until a call they expected to be blocked isn't. This is
the only place that ever says so.
Returns None -- rather than a PASS -- when there is no config to check or
the installed guard is too old to declare its sibling keys, because a
check that cannot run must not print a row that looks like it did.
"""
if not user or module is None:
return None
if not hasattr(module, "unknown_config_keys"):
return None
unknown = module.unknown_config_keys(user)
label = "every config key is read by something"
if not unknown:
readable = [k for k in user if not k.startswith("_")]
return (PASS, label, f"{len(readable)} key(s), all recognised")
siblings = getattr(module, "SIBLING_CONFIG_KEYS", {})
hint = ""
known = sorted(set(getattr(module, "DEFAULT_CONFIG", {})) | set(siblings))
near = [(k, close_key(k, known)) for k in unknown]
near = [f"{k} (did you mean {n}?)" for k, n in near if n]
if near:
hint = " " + "; ".join(near)
return (WARN, label,
f"{config_path} sets {len(unknown)} key(s) nothing reads: "
f"{', '.join(unknown)} -- silently ignored, so whatever you meant "
f"to set is still at its default.{hint}")
def close_key(key, known):
"""The known key a typo most plausibly meant, or None. Deliberately
conservative: a wrong suggestion is worse than none, so it only fires on
a single edit-distance-ish match (a shared prefix and a near-equal
length), not on a general fuzzy score."""
import difflib
matches = difflib.get_close_matches(key, known, n=1, cutoff=0.85)
return matches[0] if matches else None
def rule_coverage_row(user, module):
"""One row for built-in rules a config silently drops.
Setting `absolute_rules` in a config REPLACES the built-in list; it does
not extend it. A project that adds one rule of its own and expects the
other five to still be there is enforcing one rule and believes it is
enforcing six. Nothing anywhere reported that before this row.
Returns None when the config overrides no rule list -- the common case,
and a row saying "you dropped nothing" on every run is noise.
"""
if not user or module is None or not hasattr(module, "replaced_rule_lists"):
return None
replaced = module.replaced_rule_lists(user)
dropping = [r for r in replaced if r[3]]
label = "built-in rules preserved"
if not replaced:
return None
if not dropping:
return (PASS, label,
f"{len(replaced)} list(s) overridden, 0 built-in entries "
f"dropped")
parts = []
for key, builtin_n, user_n, dropped in dropping:
mine = f"{user_n} entry" if user_n == 1 else f"{user_n} entries"
theirs = f"{builtin_n} built-in one" if builtin_n == 1 \
else f"{builtin_n} built-in ones"
parts.append(f"{key}: your config REPLACES the {theirs} with {mine}, "
f"dropping {', '.join(str(d) for d in dropped)}")
return (WARN, label,
"; ".join(parts) + " -- a config REPLACES a rule list, it does "
"not extend it. If that was deliberate this is fine; if it was "
"not, those rules are no longer enforced anywhere.")
# Mirrors gate_guard.read_trust_tier()'s states. verify.py deliberately does
# not import the guard -- it verifies whatever version is installed at
# --target, which may not be this one -- so the states are duplicated here on
# purpose and the two must be kept in step by hand.
TIER_OK = "ok"
TIER_STATE_MISSING = "state_missing"
TIER_STATE_UNREADABLE = "state_unreadable"
TIER_FIELD_MISSING = "field_missing"
TIER_FIELD_INVALID = "field_invalid"
def read_trust_tier(state_abspath, config):
"""Read the tier the way the guard does. Returns (state, tier, detail).
`tier` is 0 for every state except OK, matching the guard's fail-closed
fallback, so callers that only need a number can use it directly. This is
the single place verify.py reads the tier: the wiring row, the behaviour
probes' tier-threshold logic, and the evidence report all go through here,
because three copies of `except: tier = 0` is three chances to disagree
with the guard about what tier the project is actually at."""
field = config.get("trust_tier_field", "trust_tier")
if not state_abspath or not os.path.isfile(state_abspath):
return TIER_STATE_MISSING, 0, ""
try:
with open(state_abspath) as f:
raw = json.load(f)
except ValueError as e:
return TIER_STATE_UNREADABLE, 0, f"invalid JSON ({e})"
except (OSError, UnicodeDecodeError) as e:
return TIER_STATE_UNREADABLE, 0, f"{type(e).__name__}: {e}"
if not isinstance(raw, dict):
return TIER_STATE_UNREADABLE, 0, \
f"top level is {type(raw).__name__}, not an object"
if field not in raw:
return TIER_FIELD_MISSING, 0, f"{len(raw)} key(s), none of them {field!r}"
value = raw[field]
if isinstance(value, bool) or not isinstance(value, int):
return TIER_FIELD_INVALID, 0, \
f"{field} = {value!r} ({type(value).__name__}), not an integer"
return TIER_OK, value, ""
def trust_tier_row(state_abspath, state_rel, config):
"""One row describing where the trust tier came from.
Four of the five states produce tier 0 and identical *behaviour* -- every
tier-gated rule blocked -- but only one of them is a decision. Reporting
`PASS ... trust tier 0` for a state file that has no tier field in it at
all is the same mistake the allowlist row used to make: it tells the
operator the config is fine when the guard is running on a guess."""
label = "trust tier readable"
min_tier = config.get("min_tier_for_tier_gated", 1)
state, tier, detail = read_trust_tier(state_abspath, config)
if state == TIER_OK:
gated = "tier-gated rules are ON" if tier < min_tier else \
"tier-gated rules are OFF"
return (PASS, label,
f"{state_rel}, trust tier {tier} (unlocks at {min_tier}; "
f"{gated})")
assumed = (f"tier is ASSUMED 0, not read -- tier-gated rules stay blocked "
f"and the block message says so")
if state == TIER_STATE_MISSING:
return (WARN, label,
f"{state_rel} not found -- {assumed}. Expected on a fresh "
f"plugin install, which writes no state file")
if state == TIER_STATE_UNREADABLE:
return (WARN, label, f"{state_rel}: {detail} -- {assumed}")
if state == TIER_FIELD_MISSING:
return (WARN, label,
f"{state_rel} parses but has no trust-tier field ({detail}) "
f"-- {assumed}")
return (WARN, label, f"{state_rel}: {detail} -- {assumed}")
def allowlist_row(target, config):
"""One row describing the git-push allowlist.
Reports the three no-push states separately, because they behave
identically (every push blocked) but are fixed differently, and a report
that calls a present-but-empty file PASS tells the operator the opposite
of what they will experience."""
label = "git push allowlist present"
remotes = os.path.join(target, config.get("approved_remotes_file",
"approved-remotes.txt"))
if not os.path.isfile(remotes):
# isfile() is false for a directory too, which is why the detail says
# "no readable file" rather than asserting the path is empty.
if os.path.exists(remotes):
return (WARN, label,
f"{remotes} exists but is not a readable file -- the "
f"guard fails closed and every git push is blocked")
return (WARN, label,
"missing -- every git push is blocked (fail closed)")
try:
with open(remotes) as f:
entries = [l.strip() for l in f
if l.strip() and not l.startswith("#")]
except (OSError, UnicodeDecodeError) as e:
return (WARN, label,
f"{remotes} exists but cannot be read ({e}) -- the guard "
f"fails closed and every git push is blocked")
if not entries:
return (WARN, label,
"file exists but lists 0 remotes -- every git push is "
"blocked, same as if it were missing")
return (PASS, label, f"{len(entries)} approved remote(s)")
def configured_rule_ids(config):
ids = set()
for key in ("absolute_rules", "tier_gated_rules"):
for rule in config.get(key) or []:
ids.add(rule.get("id"))
return ids
# ---------------------------------------------------------------------------
# budget_guard.py
# ---------------------------------------------------------------------------
BUDGET_MARKER = "budget_guard.py"
UNKNOWN_MODEL = "zzz-unreleased-probe-model"
def load_guard_module(guard_path, name):
"""Import an installed guard to read its own declarations -- default
config, price table, key registry.
This is deliberately narrow: nothing here calls its decision functions.
Probes still go through the registered command as a subprocess. The
import exists so this report is sized against the guard that is ACTUALLY
INSTALLED at --target, which may be older than this verify.py; restating
its tables here would drift the first time either one changed, and would
quietly report a newer file's key set against an older guard.
"""
try:
import importlib.util
spec = importlib.util.spec_from_file_location(name, guard_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
except Exception:
return None
def load_budget_module(guard_path):
"""The budget guard, for its default config and price table."""
return load_guard_module(guard_path, "_bg_probe")
def merge_budget_config(module, user_config):
"""Merge a user config over the module's defaults one level deep --
the same shallow merge load_config() documents, so a config that
overrides one loop-detector field doesn't have to restate the rest."""
merged = json.loads(json.dumps(getattr(module, "DEFAULT_CONFIG", {})))
for key, value in (user_config or {}).items():
if key.startswith("_"):
continue
if isinstance(value, dict) and isinstance(merged.get(key), dict):
merged[key].update(value)
else:
merged[key] = value
return merged
def resolve_budget_config(command, target, guard_path):
"""Resolve the config the way budget_guard.py will: the env var in the
hook command, then the working directory, then next to the script."""
m = re.search(r"BUDGET_GUARD_CONFIG=(\"[^\"]+\"|'[^']+'|\S+)", command)
if m:
path = m.group(1).strip("\"'")
if not os.path.isabs(path):
path = os.path.join(target, path)
return path, "hook command"
for candidate, origin in (
(os.path.join(target, "budget-guard.config.json"), "project root"),
(os.path.join(os.path.dirname(guard_path), "budget-guard.config.json"),
"next to the script"),
):
if os.path.isfile(candidate):
return candidate, origin
return None, "built-in defaults"
def writable(directory):
"""Can the hook actually persist state here? If it cannot, check_loop
swallows the OSError by design and the loop window silently never
accumulates -- the detector reports nothing and looks healthy."""
try:
os.makedirs(directory, exist_ok=True)
probe = os.path.join(directory, ".verify-write-probe")
with open(probe, "w") as f:
f.write("")
os.remove(probe)
return True
except OSError:
return False
def check_budget_wiring(target):
"""Returns (rows, command, merged_config). command is None when the
budget guard is not installed, which is a supported choice, not a fault."""
rows = []
command, err = find_hook_command(target, BUDGET_MARKER)
if not command:
rows.append((SKIP, "hook registered in .claude/settings.json",
f"{err} -- skipping (install.py --no-budget-guard is a "
f"supported choice)"))
return rows, None, None
rows.append((PASS, "hook registered in .claude/settings.json", command))
m = re.search(r"(\S*budget[_-]guard\.py)", command)
guard = (m.group(1).strip("\"'") if m else "")
if not os.path.isabs(guard):
guard = os.path.join(target, guard)
if os.path.isfile(guard):
rows.append((PASS, "hook script exists at the registered path", guard))
else:
rows.append((FAIL, "hook script exists at the registered path",
f"not found: {guard}"))
return rows, None, None
config_path, origin = resolve_budget_config(command, target, guard)
user_config = {}
if config_path and os.path.isfile(config_path):
try:
with open(config_path) as f:
user_config = json.load(f)
rows.append((PASS, f"config resolves (via {origin})", config_path))
except json.JSONDecodeError as e:
rows.append((FAIL, f"config resolves (via {origin})",
f"{config_path}: invalid JSON: {e} -- the hook falls "
f"back to built-in defaults, NOT your ceilings"))
elif config_path:
rows.append((WARN, "config resolves",
f"{config_path} not found -- the hook falls back to its "
f"built-in defaults, NOT your ceilings"))
else:
rows.append((WARN, "config resolves", "no config file found anywhere "
"on the resolution path -- built-in defaults are in force"))
module = load_budget_module(guard)
if module is None:
rows.append((FAIL, "hook script imports cleanly",
f"{guard} raised on import -- the hook cannot run"))
return rows, None, None
config = merge_budget_config(module, user_config)
session_ceiling = config.get("session_cost_ceiling_usd")
day_ceiling = config.get("daily_cost_ceiling_usd")
detail = (f"session {'off' if session_ceiling is None else f'${session_ceiling:.2f}'}, "
f"daily {'off' if day_ceiling is None else f'${day_ceiling:.2f}'}")
if session_ceiling is None and day_ceiling is None:
rows.append((WARN, "a cost ceiling is configured",
"both ceilings are null -- the loop detector still runs, "
"but nothing stops a slow expensive session"))
else:
rows.append((PASS, "a cost ceiling is configured", detail))
cache_read = (config.get("cache_multipliers") or {}).get("cache_read")
if not config.get("pricing_usd_per_mtok"):
rows.append((FAIL, "price table is populated",
"empty -- every model prices at $0 and no ceiling can trip"))
elif cache_read is None or cache_read >= 1:
rows.append((WARN, "cache reads priced below the input rate",
f"cache_read multiplier is {cache_read} -- agent sessions "
f"are mostly cache reads, so this overstates spend and "
f"trips ceilings early"))
else:
rows.append((PASS, "price table is populated",
f"{len(config['pricing_usd_per_mtok'])} models, cache "
f"reads at {cache_read}x input"))
loop = config.get("loop_detector") or {}
if loop.get("enabled", True):
rows.append((PASS, "loop detector enabled",
f"{loop.get('consecutive_repeats')} consecutive, "
f"{loop.get('max_repeats')} in a window of "
f"{loop.get('window')}"))
else:
rows.append((WARN, "loop detector enabled",
"disabled in config -- a stuck agent will not be stopped"))
root = os.path.dirname(config_path) if config_path else target
config["_project_root"] = root
state = os.path.join(root, config.get("state_dir", ".budget-guard"))
if writable(state):
rows.append((PASS, "state directory writable", state))
else:
rows.append((FAIL, "state directory writable",
f"{state} is not writable -- the loop window cannot "
f"persist, so the detector silently never fires"))
log_dir = os.path.dirname(os.path.join(
root, config.get("blocked_log", "approvals/budget-blocked.jsonl")))
if writable(log_dir):
rows.append((PASS, "blocked-log directory writable", log_dir))
else:
rows.append((WARN, "blocked-log directory writable",
f"{log_dir} is not writable -- blocks still happen but "
f"go unrecorded"))
return rows, command, config
def transcript(path, messages):
"""Write a synthetic transcript in the harness's own JSONL shape.
`messages` is a list of (message_id, model, usage)."""
with open(path, "w") as f:
for message_id, model, usage in messages:
f.write(json.dumps({
"type": "assistant",
"message": {"id": message_id, "model": model, "usage": usage},
}) + "\n")
return path
def tokens_for(usd, rate_per_mtok, multiplier=1.0):
"""How many tokens cost `usd` at a per-million rate. Rounded up so a
probe aimed above a ceiling lands above it rather than on the boundary."""
if rate_per_mtok <= 0:
return 0
return int(usd * 1_000_000 / (rate_per_mtok * multiplier)) + 1
def budget_probe_config(root, config, **overrides):
"""Write a throwaway config into its own directory.
A fresh directory per probe is not tidiness. budget_guard keys the daily
rollup by session id inside state_dir, so probes sharing one would
accumulate each other's synthetic spend and the third probe would fail
because of the second. Each probe gets its own world.
"""
os.makedirs(root, exist_ok=True)
probe = json.loads(json.dumps(
{k: v for k, v in config.items() if not k.startswith("_")}))
probe.update(overrides)
probe["state_dir"] = ".budget-guard"
probe["blocked_log"] = "budget-blocked.jsonl"
path = os.path.join(root, "budget-guard.config.json")
with open(path, "w") as f:
json.dump(probe, f)
return path
def with_config(command, config_path):
"""Point the registered command at a probe config. Rewrites the env
assignment install.py writes, or adds one if the command has none."""
quoted = json.dumps(config_path)
if re.search(r"BUDGET_GUARD_CONFIG=(\"[^\"]+\"|'[^']+'|\S+)", command):
return re.sub(r"BUDGET_GUARD_CONFIG=(\"[^\"]+\"|'[^']+'|\S+)",
f"BUDGET_GUARD_CONFIG={quoted}", command)
return f"env BUDGET_GUARD_CONFIG={quoted} {command}"
def budget_payload(session_id, transcript_path="", tool="Bash", command="ls"):
return {
"session_id": session_id,
"transcript_path": transcript_path,
"tool_name": tool,
"tool_input": {"command": command},
}
def run_budget_probes(command, config, target, workdir):
"""Yields (status, description, detail). Each probe is self-contained:
its own config, its own state directory, its own synthetic transcript."""
model = ("claude-opus-5" if "claude-opus-5" in config["pricing_usd_per_mtok"]
else next(iter(config["pricing_usd_per_mtok"]), ""))
row = config["pricing_usd_per_mtok"].get(model) or {"input": 0, "output": 0}
in_rate = row["input"]
cache_mult = (config.get("cache_multipliers") or {}).get("cache_read", 0.1)
session_ceiling = config.get("session_cost_ceiling_usd")
day_ceiling = config.get("daily_cost_ceiling_usd")
counter = [0]
def run(desc, payload, expect, expect_rule=None, **overrides):
counter[0] += 1
root = os.path.join(workdir, f"probe{counter[0]}")
cfg = budget_probe_config(root, config, **overrides)
code, stderr = run_probe(with_config(command, cfg), payload, target)
blocked = code == 2
got = "blocked" if blocked else (
"allowed" if code == 0 else f"exit {code}")
m = re.search(r"BLOCKED BY BUDGET GUARD \[([A-Z_]+)\]", stderr)
rule = m.group(1) if m else None
if rule:
got = f"blocked [{rule}]"
ok = ((blocked and expect == "block") or (code == 0 and expect == "allow"))
if ok and expect_rule and rule != expect_rule:
ok, got = False, f"{got} <-- expected rule {expect_rule}"
elif not ok:
got += f" <-- expected {expect}"
return (PASS if ok else FAIL), desc, got
# --- cost ceiling ---
if in_rate <= 0:
yield SKIP, "cost ceiling probes", f"no usable input rate for {model}"
elif session_ceiling is None and day_ceiling is None:
yield SKIP, "cost ceiling probes", "no ceiling configured"
else:
ceiling = session_ceiling if session_ceiling is not None else day_ceiling
rule = "SESSION_BUDGET" if session_ceiling is not None else "DAILY_BUDGET"
# Isolate whichever ceiling we are not probing, so a block can only
# have come from the one under test.
other = ({"daily_cost_ceiling_usd": None} if session_ceiling is not None
else {"session_cost_ceiling_usd": None})
under = transcript(os.path.join(workdir, "under.jsonl"), [
("msg_under", model,
{"input_tokens": tokens_for(ceiling * 0.1, in_rate)})])
yield run("spend under the ceiling is allowed",
budget_payload("verify-under", under), "allow", **other)
over = transcript(os.path.join(workdir, "over.jsonl"), [
("msg_over", model,
{"input_tokens": tokens_for(ceiling * 2, in_rate)})])
yield run("spend over the ceiling blocks",
budget_payload("verify-over", over), "block", rule, **other)
# Sized to 0.3x the ceiling at YOUR configured cache-read multiplier,
# not at a hardcoded 0.1 -- the probe tests that the multiplier is
# applied at all, whatever you set it to. If it is ignored and cache
# reads bill as fresh input, the same transcript costs 0.3/mult times
# the ceiling (30x at the default 0.1) and blocks.
cached = transcript(os.path.join(workdir, "cached.jsonl"), [
("msg_cache", model,
{"cache_read_input_tokens":
tokens_for(ceiling * 0.3, in_rate, cache_mult)})])
yield run(f"cache reads priced at the configured {cache_mult}x rate",
budget_payload("verify-cache", cached), "allow", **other)
# One streamed message rewritten five times as it grew. Deduplicated
# by id it is 0.5x the ceiling; summed line by line it is 2.5x.
streamed = transcript(os.path.join(workdir, "streamed.jsonl"), [
("msg_stream", model,
{"input_tokens": tokens_for(ceiling * 0.5, in_rate)})] * 5)
yield run("a streamed message is counted once, not once per line",
budget_payload("verify-stream", streamed), "allow", **other)
# An unknown model ID is what a newly released model looks like from
# here. Under the default "priciest" policy it must still be billed.
if config.get("unknown_model_policy") == "ignore":
yield SKIP, "an unrecognised model is still billed", \
"unknown_model_policy is 'ignore' -- off by your config"
else:
priciest = max(config["pricing_usd_per_mtok"].values(),
key=lambda r: r["output"])["input"]
unknown = transcript(os.path.join(workdir, "unknown.jsonl"), [
("msg_unknown", UNKNOWN_MODEL,
{"input_tokens": tokens_for(ceiling * 2, priciest)})])
yield run("an unrecognised model is still billed",
budget_payload("verify-unknown", unknown), "block", rule,
**other)
if session_ceiling is not None and day_ceiling is not None:
day = transcript(os.path.join(workdir, "day.jsonl"), [
("msg_day", model,
{"input_tokens": tokens_for(day_ceiling * 1.5, in_rate)})])
yield run("spend over the daily ceiling blocks",
budget_payload("verify-day", day), "block",
"DAILY_BUDGET", session_cost_ceiling_usd=None)
# --- loop detector ---
loop = config.get("loop_detector") or {}
if not loop.get("enabled", True):
yield SKIP, "loop detector probes", "disabled in your config"
return
consecutive = int(loop.get("consecutive_repeats", 4) or 0)
max_repeats = int(loop.get("max_repeats", 6) or 0)
window = max(1, int(loop.get("window", 20) or 1))
if "Bash" in (loop.get("ignore_tools") or []):
yield SKIP, "loop detector probes", "Bash is in your ignore_tools"
return
limits = [n for n in (consecutive, max_repeats) if n > 0]
if not limits:
yield SKIP, "loop detector probes", "both loop limits are disabled"
return
trip_at = min(limits)
def loop_sequence(desc, commands, expect_block_at, expect_rule=None):
"""Replay a call sequence through one shared config, since the
window is stateful across calls by design."""
counter[0] += 1
root = os.path.join(workdir, f"probe{counter[0]}")
# transcript_path is empty so the cost check returns early and cannot
# confound the result: this probe is about repetition only.
cfg = budget_probe_config(root, config,
session_cost_ceiling_usd=None,
daily_cost_ceiling_usd=None)
cmd = with_config(command, cfg)
for i, call in enumerate(commands, start=1):
code, stderr = run_probe(
cmd, budget_payload("verify-loop", command=call), target)
m = re.search(r"BLOCKED BY BUDGET GUARD \[([A-Z_]+)\]", stderr)
rule = m.group(1) if m else None
if expect_block_at is None:
if code != 0:
return FAIL, desc, f"call {i} of {len(commands)} was blocked [{rule}]"
continue
if i < expect_block_at and code != 0:
return FAIL, desc, f"blocked early, on call {i} of {expect_block_at}"