Skip to content

Commit 9eb418c

Browse files
committed
Address public evidence review feedback
Harden evidence validation against blank claims, mutable pinned container tags, absolute catalog path mistakes, and symlink escapes. Pin manual workflow examples to the matching release definition and align evidence navigation and fallback wording with the catalog.
1 parent 3f15bf0 commit 9eb418c

8 files changed

Lines changed: 129 additions & 18 deletions

File tree

docs/EVIDENCE.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ the case.
2727
| --- | --- |
2828
| Deterministic | Uses checked-in inputs or local services and compares a stable normalized result |
2929
| Pinned input | Uses a public repository at a recorded commit; local tools or artifact registries can still affect build-tool-backed resolution |
30+
| Snapshot | Records the normalized result of an input that can move, such as a container tag |
3031
| Live service | Uses a pinned project with current advisory data; the result is a dated observation |
3132
| Manual assurance | Runs a separately started GitHub Actions workflow and saves its detailed report as an artifact |
3233

@@ -37,8 +38,8 @@ expected results include SHA-256 checksums.
3738
## Case studies
3839

3940
- [Dependency graph evidence](evidence/DEPENDENCY_GRAPHS.md) covers npm, pnpm,
40-
Yarn, Bun, Go, Python, and Maven graphs, plus visible degraded fallback
41-
behavior.
41+
Yarn, Bun, Go, Python, and Maven graphs. A separate deterministic
42+
`degraded-detector-fallback` case covers visible fallback orchestration.
4243
- [Policy and vulnerability-guidance evidence](evidence/POLICY_AND_GUIDANCE.md)
4344
covers vulnerability and SPDX policy, baselines, source changes, persisted
4445
findings, reachability tiers, and read-only remediation suggestions.

docs/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,6 @@ Task-oriented walkthroughs.
99
- [Installation](INSTALLATION.md) — install methods, `bomly` vs `bomly-lite`, checksum verification, uninstall
1010
- [Getting Started](GETTING_STARTED.md) — first scan, enrich, audit, diff
1111
- [Use Cases](USE_CASES.md) — recipes for PR gates, SBOMs, triage, license and offline scans
12-
- [Reproducible Evidence](EVIDENCE.md) — public inputs, commands, results, and limitations behind important behavior claims
1312
- [Scan Targets](SCAN_TARGETS.md) — directories, Git repos, containers, SBOMs
1413
- [Output Formats](OUTPUT_FORMATS.md) — text, JSON, SARIF, SBOM
1514
- [SBOM Formats](SBOM.md) — SPDX vs. CycloneDX, write and ingest
@@ -32,6 +31,7 @@ How Bomly thinks about your project.
3231
- [Reachability](REACHABILITY.md) — narrowing findings to code your app actually calls
3332
- [MCP Server](MCP.md) — connect Bomly to Claude Code, Cursor, VS Code, or another MCP client
3433
- [Plugins](PLUGINS.md) — install, trust, configure, and package external plugins
34+
- [Reproducible Evidence](EVIDENCE.md) — public inputs, commands, results, and limitations behind important behavior claims
3535
- Plugin implementation guides: [detector](plugins/how-to-implement-detector.md), [matcher](plugins/how-to-implement-matcher.md), [auditor](plugins/how-to-implement-auditor.md)
3636
- Example plugin repos: [Bun detector](https://github.com/bomly-dev/bomly-plugin-bun-lock-detector), [ClearlyDefined matcher](https://github.com/bomly-dev/bomly-plugin-clearlydefined-matcher), [EOL lifecycle matcher](https://github.com/bomly-dev/bomly-plugin-eol-matcher), [Meme auditor](https://github.com/bomly-dev/bomly-plugin-meme-auditor)
3737
- [Glossary](GLOSSARY.md) — every term, one sentence each

docs/evidence/TARGETS_AND_OPERATIONS.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ The deterministic ingestion cases check Bomly's internal graph. The manually
4444
started interoperability workflow adds an external check:
4545

4646
```sh
47-
gh workflow run sbom-interoperability.yml
47+
gh workflow run sbom-interoperability.yml --ref v0.20.0
4848
```
4949

5050
It generates canonical SPDX 2.3 and CycloneDX 1.6 files, verifies the
@@ -56,13 +56,15 @@ for the workflow summary and failure-investigation steps.
5656
The public catalog records the workflow checksum under
5757
`sbom-interoperability`. Validator versions and download checksums stay in the
5858
workflow so changing either requires an intentional evidence update.
59+
The `v0.20.0` release tag contains the recorded workflow definition, so this
60+
command does not silently switch to a later default-branch workflow.
5961

6062
## Supported-system checks
6163

6264
The `portable-platforms` case starts:
6365

6466
```sh
65-
gh workflow run portable-assurance.yml
67+
gh workflow run portable-assurance.yml --ref v0.20.0
6668
```
6769

6870
This workflow:

docs/manifest.json

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,8 @@
4949
"slug": "evidence",
5050
"title": "Reproducible evidence",
5151
"description": "Public inputs, commands, results, and limitations behind important behavior claims.",
52-
"group": "start"
52+
"group": "concepts",
53+
"hasChildren": true
5354
},
5455
{
5556
"slug": "scan-targets",

internal/tools/publicevidence/main.go

Lines changed: 47 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -23,9 +23,10 @@ const (
2323
)
2424

2525
var (
26-
caseIDPattern = regexp.MustCompile(`^[a-z0-9]+(?:-[a-z0-9]+)*$`)
27-
revisionPattern = regexp.MustCompile(`^[0-9a-f]{40}$`)
28-
hashPattern = regexp.MustCompile(`^[0-9a-f]{64}$`)
26+
caseIDPattern = regexp.MustCompile(`^[a-z0-9]+(?:-[a-z0-9]+)*$`)
27+
revisionPattern = regexp.MustCompile(`^[0-9a-f]{40}$`)
28+
hashPattern = regexp.MustCompile(`^[0-9a-f]{64}$`)
29+
containerDigestPattern = regexp.MustCompile(`@sha256:[0-9a-f]{64}$`)
2930
)
3031

3132
type catalog struct {
@@ -68,7 +69,8 @@ func main() {
6869
if err != nil {
6970
exitError(err)
7071
}
71-
loaded, err := loadCatalog(filepath.Join(root, filepath.FromSlash(*catalogPath)))
72+
resolvedCatalog := resolveCatalogPath(root, *catalogPath)
73+
loaded, err := loadCatalog(resolvedCatalog)
7274
if err != nil {
7375
exitError(err)
7476
}
@@ -109,6 +111,14 @@ func repositoryRoot() (string, error) {
109111
}
110112
}
111113

114+
func resolveCatalogPath(root, catalogPath string) string {
115+
resolved := filepath.FromSlash(catalogPath)
116+
if filepath.IsAbs(resolved) {
117+
return resolved
118+
}
119+
return filepath.Join(root, resolved)
120+
}
121+
112122
func loadCatalog(path string) (catalog, error) {
113123
file, err := os.Open(path)
114124
if err != nil {
@@ -175,15 +185,15 @@ func validateCase(root string, current evidenceCase) error {
175185
return errors.New("title and area are required")
176186
}
177187
switch current.EvidenceLevel {
178-
case "deterministic", "pinned-input", "live-service", "manual-assurance":
188+
case "deterministic", "pinned-input", "live-service", "manual-assurance", "snapshot":
179189
default:
180190
return fmt.Errorf("unsupported evidence level %q", current.EvidenceLevel)
181191
}
182192
if len(current.Inputs) == 0 {
183193
return errors.New("at least one input is required")
184194
}
185195
for _, item := range current.Inputs {
186-
if err := validateInput(root, item); err != nil {
196+
if err := validateInput(root, current.EvidenceLevel, item); err != nil {
187197
return err
188198
}
189199
}
@@ -211,10 +221,20 @@ func validateCase(root string, current evidenceCase) error {
211221
if len(current.Proves) == 0 || len(current.Limitations) == 0 {
212222
return errors.New("proves and limitations must both be explicit")
213223
}
224+
for _, claim := range current.Proves {
225+
if strings.TrimSpace(claim) == "" {
226+
return errors.New("proves and limitations cannot contain blank entries")
227+
}
228+
}
229+
for _, limitation := range current.Limitations {
230+
if strings.TrimSpace(limitation) == "" {
231+
return errors.New("proves and limitations cannot contain blank entries")
232+
}
233+
}
214234
return nil
215235
}
216236

217-
func validateInput(root string, current input) error {
237+
func validateInput(root, evidenceLevel string, current input) error {
218238
if strings.TrimSpace(current.Location) == "" {
219239
return errors.New("input location is required")
220240
}
@@ -232,6 +252,9 @@ func validateInput(root string, current input) error {
232252
if current.Ref == "" {
233253
return errors.New("container input requires an image reference")
234254
}
255+
if evidenceLevel == "pinned-input" && !containerDigestPattern.MatchString(current.Ref) {
256+
return errors.New("pinned container input requires an immutable sha256 digest")
257+
}
235258
case "workflow":
236259
if !hashPattern.MatchString(current.SHA256) {
237260
return errors.New("workflow input requires a SHA-256 hash")
@@ -251,15 +274,30 @@ func validateArtifact(root string, item artifact) error {
251274
if filepath.IsAbs(clean) || clean == ".." || strings.HasPrefix(clean, ".."+string(filepath.Separator)) {
252275
return fmt.Errorf("artifact path %q must stay inside the repository", item.Path)
253276
}
277+
resolvedRoot, err := filepath.EvalSymlinks(root)
278+
if err != nil {
279+
return fmt.Errorf("resolve repository root: %w", err)
280+
}
254281
path := filepath.Join(root, clean)
255-
info, err := os.Stat(path)
282+
resolvedPath, err := filepath.EvalSymlinks(path)
283+
if err != nil {
284+
return fmt.Errorf("resolve artifact %q: %w", item.Path, err)
285+
}
286+
relative, err := filepath.Rel(resolvedRoot, resolvedPath)
287+
if err != nil {
288+
return fmt.Errorf("resolve artifact %q relative to repository: %w", item.Path, err)
289+
}
290+
if relative == ".." || strings.HasPrefix(relative, ".."+string(filepath.Separator)) {
291+
return fmt.Errorf("artifact path %q resolves outside the repository", item.Path)
292+
}
293+
info, err := os.Stat(resolvedPath)
256294
if err != nil {
257295
return fmt.Errorf("inspect artifact %q: %w", item.Path, err)
258296
}
259297
if !info.Mode().IsRegular() {
260298
return fmt.Errorf("artifact %q is not a regular file", item.Path)
261299
}
262-
data, err := os.ReadFile(path)
300+
data, err := os.ReadFile(resolvedPath)
263301
if err != nil {
264302
return fmt.Errorf("read artifact %q: %w", item.Path, err)
265303
}

internal/tools/publicevidence/main_test.go

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import (
55
"encoding/hex"
66
"os"
77
"path/filepath"
8+
"runtime"
89
"strings"
910
"testing"
1011
)
@@ -44,6 +45,16 @@ func TestLoadAndValidateCatalog(t *testing.T) {
4445
if err := validateCatalog(root, loaded); err != nil {
4546
t.Fatal(err)
4647
}
48+
49+
loaded.Cases[0].Proves = []string{" "}
50+
if err := validateCatalog(root, loaded); err == nil || !strings.Contains(err.Error(), "blank entries") {
51+
t.Fatalf("validateCatalog() blank proof error = %v", err)
52+
}
53+
loaded.Cases[0].Proves = []string{"The example succeeds."}
54+
loaded.Cases[0].Limitations = []string{"\t"}
55+
if err := validateCatalog(root, loaded); err == nil || !strings.Contains(err.Error(), "blank entries") {
56+
t.Fatalf("validateCatalog() blank limitation error = %v", err)
57+
}
4758
}
4859

4960
func TestValidateCatalogRejectsUnpinnedGitAndChangedArtifact(t *testing.T) {
@@ -101,6 +112,58 @@ func TestValidateCatalogRejectsUnsortedAndUnknownFields(t *testing.T) {
101112
}
102113
}
103114

115+
func TestValidateInputRequiresDigestForPinnedContainer(t *testing.T) {
116+
tagged := input{Kind: "container", Location: "Docker Hub", Ref: "alpine:3.20"}
117+
if err := validateInput(t.TempDir(), "pinned-input", tagged); err == nil ||
118+
!strings.Contains(err.Error(), "immutable sha256 digest") {
119+
t.Fatalf("validateInput() tagged pinned container error = %v", err)
120+
}
121+
if err := validateInput(t.TempDir(), "snapshot", tagged); err != nil {
122+
t.Fatalf("validateInput() snapshot tag error = %v", err)
123+
}
124+
digested := tagged
125+
digested.Ref = "alpine@sha256:" + strings.Repeat("a", 64)
126+
if err := validateInput(t.TempDir(), "pinned-input", digested); err != nil {
127+
t.Fatalf("validateInput() digest error = %v", err)
128+
}
129+
}
130+
131+
func TestValidateArtifactRejectsSymlinkOutsideRepository(t *testing.T) {
132+
if runtime.GOOS == "windows" {
133+
t.Skip("symlink creation requires elevated privileges on Windows")
134+
}
135+
root := t.TempDir()
136+
data := []byte("outside")
137+
outside := filepath.Join(t.TempDir(), "result.json")
138+
if err := os.WriteFile(outside, data, 0o600); err != nil {
139+
t.Fatal(err)
140+
}
141+
link := filepath.Join(root, "result.json")
142+
if err := os.Symlink(outside, link); err != nil {
143+
t.Fatal(err)
144+
}
145+
sum := sha256.Sum256(data)
146+
err := validateArtifact(root, artifact{
147+
Path: "result.json",
148+
SHA256: hex.EncodeToString(sum[:]),
149+
})
150+
if err == nil || !strings.Contains(err.Error(), "resolves outside the repository") {
151+
t.Fatalf("validateArtifact() error = %v", err)
152+
}
153+
}
154+
155+
func TestResolveCatalogPathHonorsAbsolutePath(t *testing.T) {
156+
root := t.TempDir()
157+
absolute := filepath.Join(t.TempDir(), "cases.json")
158+
if got := resolveCatalogPath(root, absolute); got != absolute {
159+
t.Fatalf("resolveCatalogPath() absolute = %q, want %q", got, absolute)
160+
}
161+
wantRelative := filepath.Join(root, "test", "evidence", "cases.json")
162+
if got := resolveCatalogPath(root, "test/evidence/cases.json"); got != wantRelative {
163+
t.Fatalf("resolveCatalogPath() relative = %q, want %q", got, wantRelative)
164+
}
165+
}
166+
104167
func TestShellCommandQuotesPatterns(t *testing.T) {
105168
got := shellCommand([]string{"go", "test", "-run", "TestScan$/scan-npm$"})
106169
want := "go test -run 'TestScan$/scan-npm$'"

test/evidence/README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,8 @@ The checker verifies that:
3030
the same normalized result.
3131
- `pinned-input` uses a public repository revision. Package-manager tools or
3232
registries can still affect build-tool-backed resolution.
33+
- `snapshot` records the normalized result of an input that can move, such as
34+
a container tag.
3335
- `live-service` combines a pinned project with current advisory data. It is a
3436
dated observation because advisory services change.
3537
- `manual-assurance` starts a GitHub Actions workflow and stores the detailed

test/evidence/cases.json

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@
5050
"id": "container-inventory",
5151
"title": "Container package inventory",
5252
"area": "targets",
53-
"evidence_level": "pinned-input",
53+
"evidence_level": "snapshot",
5454
"inputs": [
5555
{
5656
"kind": "container",
@@ -568,7 +568,9 @@
568568
"gh",
569569
"workflow",
570570
"run",
571-
"portable-assurance.yml"
571+
"portable-assurance.yml",
572+
"--ref",
573+
"v0.20.0"
572574
]
573575
],
574576
"evidence": [
@@ -865,7 +867,9 @@
865867
"gh",
866868
"workflow",
867869
"run",
868-
"sbom-interoperability.yml"
870+
"sbom-interoperability.yml",
871+
"--ref",
872+
"v0.20.0"
869873
]
870874
],
871875
"evidence": [

0 commit comments

Comments
 (0)