Skip to content

Commit 78250d2

Browse files
regenrekregenrek
authored
Merge pull request #39 from instructa/codex/eval-security-evidence
feat: add evidence-backed evaluation workflow
2 parents 0c68a4f + efce2ba commit 78250d2

52 files changed

Lines changed: 21081 additions & 4504 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.betterleaks.toml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
title = "Planr BetterLeaks configuration"
2+
3+
prefilter = '''
4+
matchesAny(attributes[?"path"].orValue(""), [
5+
r"""(?:^|/)node_modules(?:/.*)?$""",
6+
r"""(?:^|/)dist(?:/.*)?$""",
7+
r"""(?:^|/)target(?:/.*)?$"""
8+
])
9+
'''
10+
11+
filter = '''
12+
containsAny(finding["secret"], [
13+
"EXAMPLE",
14+
"CHANGEME",
15+
"YOUR_API_KEY_HERE"
16+
])
17+
'''
18+
19+
[[rules]]
20+
id = "ssh-test-headers"
21+
description = "SSH private-key headers outside test fixtures"
22+
regex = '''-----BEGIN (OPENSSH |RSA )?PRIVATE KEY-----'''
23+
keywords = ["PRIVATE KEY"]
24+
filter = '''
25+
matchesAny(attributes[?"path"].orValue(""), [
26+
r""".*\.test\.[cm]?[jt]s$""",
27+
r""".*\.spec\.[cm]?[jt]s$""",
28+
r""".*fixtures.*"""
29+
])
30+
'''

‎.forbidden-paths.regex‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# Local runtime state
2+
(^|/)\.planr/
3+
(^|/)\.codex/
4+
(^|/)\.claude/
5+
(^|/)\.claude\.json$
6+
7+
# Environment files
8+
(^|/)\.env$
9+
(^|/)\.env\.
10+
(^|/)\.envrc$
11+
12+
# Credential files
13+
(^|/)\.aws/credentials$
14+
(^|/)\.npmrc$
15+
(^|/)\.netrc$
16+
(^|/)\.git-credentials$
17+
(^|/)\.docker/config\.json$
18+
19+
# Private keys
20+
\.pem$
21+
\.key$
22+
\.p12$
23+
\.pfx$
24+
\.jks$
25+
\.keystore$
26+
\.agekey$
27+
(^|/)id_rsa$
28+
(^|/)id_ed25519$
29+
(^|/)id_ecdsa$
30+
(^|/)id_dsa$
31+
32+
# Infrastructure state
33+
\.tfstate($|\.|/)
34+
\.tfplan$
35+
\.tfvars$
36+
37+
# Secrets and tokens
38+
(^|/)secrets\.json$
39+
(^|/)tokens\.json$
40+
(^|/)credentials\.json$
41+
(^|/)service-account.*\.json$
42+
43+
# IDE/editor local files
44+
(^|/)\.idea/
45+
(^|/)\.vscode/settings\.json$
46+
(^|/)\.vscode/launch\.json$
47+
(^|/)\.cursor/(settings|launch|mcp)\.json$

‎.github/workflows/security.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,9 @@ jobs:
2323
persist-credentials: false
2424
fetch-depth: 0
2525

26+
- name: Check repository privacy boundaries
27+
run: sh scripts/check-repository-privacy.sh
28+
2629
- name: TruffleHog verified secrets
2730
uses: trufflesecurity/trufflehog@d411fff7b8879a62509f3fa98c07f247ac089a51 # v3.95.5
2831
with:

‎.gitignore‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,12 +33,49 @@ apps/web/
3333
tmp/
3434
temp/
3535

36+
# Local credentials, private keys, and machine-specific auth
37+
.aws/
38+
.npmrc
39+
.netrc
40+
.git-credentials
41+
.docker/config.json
42+
*.pem
43+
*.key
44+
*.p12
45+
*.pfx
46+
*.jks
47+
*.keystore
48+
*.agekey
49+
id_rsa
50+
id_ed25519
51+
id_ecdsa
52+
id_dsa
53+
credentials.json
54+
secrets.json
55+
tokens.json
56+
service-account*.json
57+
58+
# Local infrastructure state and databases
59+
*.tfstate
60+
*.tfstate.*
61+
*.tfplan
62+
*.tfvars
63+
!*.tfvars.example
64+
*.sqlite
65+
*.sqlite3
66+
*.db
67+
3668
# Local agent/runtime state
3769
.alchemy/
3870
.codex/
71+
.claude/
72+
.claude.json
3973
.planr/
4074
docs/planr-spec.zip
4175

4276
# provisioned local copies (single source: plugins/planr); hooks stay committed
4377
.cursor/skills/
4478
.cursor/agents/
79+
.cursor/settings.json
80+
.cursor/launch.json
81+
.cursor/mcp.json

‎apps/docs/content/docs/reference/cli-generated.mdx‎

Lines changed: 173 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
title: Generated CLI Reference
3-
description: Complete compiled Planr 1.5.2 command and option help, checked against the repository binary.
3+
description: Complete compiled Planr 1.6.0 command and option help, checked against the repository binary.
44
---
55

66
<Callout type="info" title="Generated executable contract">
@@ -30,6 +30,7 @@ Commands:
3030
close
3131
done Log evidence and finish the step in one command: completion log, then review request (--review) or close, optionally pick the next item
3232
review
33+
eval Evaluate stored run evidence: suite check, run, show, compare, gate, invalidate, and rescore through the shared eval services
3334
context
3435
note
3536
search
@@ -1324,6 +1325,177 @@ Options:
13241325
-h, --help Print help
13251326
```
13261327

1328+
### `planr eval`
1329+
1330+
```text
1331+
Evaluate stored run evidence: suite check, run, show, compare, gate, invalidate, and rescore through the shared eval services
1332+
1333+
Usage: planr eval [OPTIONS] <COMMAND>
1334+
1335+
Commands:
1336+
suite-check Store or verify an immutable suite snapshot from a normalized manifest
1337+
run Start a run, optionally record case evidence, and optionally finish it
1338+
show Show a stored suite, run, comparison, or invalidation
1339+
compare Compare two stored eval runs and persist the comparison
1340+
gate Gate on a stored comparison verdict
1341+
invalidate Invalidate a run or comparison
1342+
rescore Start a rescore run from an existing run
1343+
evidence-ref Attach an eval run/comparison to an existing Planr log, review, or artifact
1344+
help Print this message or the help of the given subcommand(s)
1345+
1346+
Options:
1347+
--db <DB> Path to Planr SQLite database
1348+
--json Emit JSON output
1349+
--no-color Disable color in human output
1350+
-h, --help Print help
1351+
```
1352+
1353+
#### `planr eval suite-check`
1354+
1355+
```text
1356+
Store or verify an immutable suite snapshot from a normalized manifest
1357+
1358+
Usage: planr eval suite-check [OPTIONS] --input <INPUT>
1359+
1360+
Options:
1361+
--db <DB> Path to Planr SQLite database
1362+
--input <INPUT> JSON file containing suite_id, suite_version, digest, and normalized_manifest
1363+
--json Emit JSON output
1364+
--no-color Disable color in human output
1365+
-h, --help Print help
1366+
```
1367+
1368+
#### `planr eval run`
1369+
1370+
```text
1371+
Start a run, optionally record case evidence, and optionally finish it
1372+
1373+
Usage: planr eval run [OPTIONS] --input <INPUT>
1374+
1375+
Options:
1376+
--db <DB> Path to Planr SQLite database
1377+
--input <INPUT> JSON file containing suite_digest, subject, testbed_fingerprint, and optional cases/status
1378+
--json Emit JSON output
1379+
--no-color Disable color in human output
1380+
-h, --help Print help
1381+
```
1382+
1383+
#### `planr eval show`
1384+
1385+
```text
1386+
Show a stored suite, run, comparison, or invalidation
1387+
1388+
Usage: planr eval show [OPTIONS] <KIND> <ID>
1389+
1390+
Arguments:
1391+
<KIND> [possible values: suite, run, comparison, invalidation]
1392+
<ID>
1393+
1394+
Options:
1395+
--db <DB> Path to Planr SQLite database
1396+
--json Emit JSON output
1397+
--no-color Disable color in human output
1398+
-h, --help Print help
1399+
```
1400+
1401+
#### `planr eval compare`
1402+
1403+
```text
1404+
Compare two stored eval runs and persist the comparison
1405+
1406+
Usage: planr eval compare [OPTIONS] <BASELINE_RUN_ID> <CANDIDATE_RUN_ID>
1407+
1408+
Arguments:
1409+
<BASELINE_RUN_ID>
1410+
<CANDIDATE_RUN_ID>
1411+
1412+
Options:
1413+
--db <DB> Path to Planr SQLite database
1414+
--policy-digest <POLICY_DIGEST> [default: default]
1415+
--json Emit JSON output
1416+
--recompute-of <RECOMPUTE_OF> Existing comparison id this comparison recomputes
1417+
--no-color Disable color in human output
1418+
--rescore-of <RESCORE_OF> Existing run id whose completed rescore this comparison evaluates
1419+
-h, --help Print help
1420+
```
1421+
1422+
#### `planr eval gate`
1423+
1424+
```text
1425+
Gate on a stored comparison verdict
1426+
1427+
Usage: planr eval gate [OPTIONS] <COMPARISON_ID>
1428+
1429+
Arguments:
1430+
<COMPARISON_ID>
1431+
1432+
Options:
1433+
--db <DB> Path to Planr SQLite database
1434+
--json Emit JSON output
1435+
--no-color Disable color in human output
1436+
-h, --help Print help
1437+
```
1438+
1439+
#### `planr eval invalidate`
1440+
1441+
```text
1442+
Invalidate a run or comparison
1443+
1444+
Usage: planr eval invalidate [OPTIONS] --reason <REASON> <TARGET_KIND> <TARGET_ID>
1445+
1446+
Arguments:
1447+
<TARGET_KIND> [possible values: run, comparison]
1448+
<TARGET_ID>
1449+
1450+
Options:
1451+
--db <DB> Path to Planr SQLite database
1452+
--reason <REASON>
1453+
--json Emit JSON output
1454+
--reason-code <REASON_CODES>
1455+
--no-color Disable color in human output
1456+
--replacement-hint <REPLACEMENT_HINT>
1457+
-h, --help Print help
1458+
```
1459+
1460+
#### `planr eval rescore`
1461+
1462+
```text
1463+
Start a rescore run from an existing run
1464+
1465+
Usage: planr eval rescore [OPTIONS] <RUN_ID>
1466+
1467+
Arguments:
1468+
<RUN_ID>
1469+
1470+
Options:
1471+
--db <DB> Path to Planr SQLite database
1472+
--id <ID>
1473+
--json Emit JSON output
1474+
--no-color Disable color in human output
1475+
-h, --help Print help
1476+
```
1477+
1478+
#### `planr eval evidence-ref`
1479+
1480+
```text
1481+
Attach an eval run/comparison to an existing Planr log, review, or artifact
1482+
1483+
Usage: planr eval evidence-ref [OPTIONS] --item <ITEM> <TARGET_KIND> <TARGET_ID> <ATTACHMENT_KIND> <ATTACHMENT_ID>
1484+
1485+
Arguments:
1486+
<TARGET_KIND> [possible values: run, comparison]
1487+
<TARGET_ID>
1488+
<ATTACHMENT_KIND> [possible values: log, review, artifact]
1489+
<ATTACHMENT_ID>
1490+
1491+
Options:
1492+
--db <DB> Path to Planr SQLite database
1493+
--item <ITEM>
1494+
--json Emit JSON output
1495+
--no-color Disable color in human output
1496+
-h, --help Print help
1497+
```
1498+
13271499
### `planr context`
13281500

13291501
```text

0 commit comments

Comments
 (0)