Skip to content

Commit 3f15bf0

Browse files
bomly-guybomly-guy
committed
Document target and operational assurance evidence
Add the public evidence guide and explain local, Git, container, and SBOM target cases. Record dated validator and portable workflow evidence, clarify that portable assurance runs unit tests and release builds, and document repeatable cold and warm performance measurements with their limits.
1 parent 92bcea3 commit 3f15bf0

5 files changed

Lines changed: 232 additions & 1 deletion

File tree

‎README.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -78,7 +78,10 @@ Bomly reads manifests, lockfiles, package-manager output, container layers, or e
7878
| Can CI fail on high-severity findings? | `bomly scan --enrich --audit --fail-on high --format sarif` |
7979
| Can I triage reachable findings first? | `bomly scan --enrich --audit --analyze --fail-on high --fail-on reachable` |
8080

81-
For more recipes, see [Getting Started](docs/GETTING_STARTED.md) and [Use Cases](docs/USE_CASES.md).
81+
For more recipes, see [Getting Started](docs/GETTING_STARTED.md) and
82+
[Use Cases](docs/USE_CASES.md). To review the public inputs, commands, expected
83+
results, and limitations behind important behavior claims, see
84+
[Reproducible Evidence](docs/EVIDENCE.md).
8285

8386
## Explore Interactively
8487

‎docs/EVIDENCE.md‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
# Reproducible evidence
2+
3+
Bomly publishes the inputs, commands, expected artifacts, and limitations
4+
behind important behavior claims. The goal is to make a claim reviewable
5+
without private infrastructure or a one-off demonstration.
6+
7+
The machine-readable catalog is
8+
[`test/evidence/cases.json`](../test/evidence/cases.json). Check it from a
9+
repository checkout:
10+
11+
```sh
12+
make evidence
13+
```
14+
15+
To inspect one case:
16+
17+
```sh
18+
make evidence CASE=graph-npm
19+
```
20+
21+
This verifies the recorded checksums and prints the exact command to reproduce
22+
the case.
23+
24+
## Evidence levels
25+
26+
| Level | Meaning |
27+
| --- | --- |
28+
| Deterministic | Uses checked-in inputs or local services and compares a stable normalized result |
29+
| Pinned input | Uses a public repository at a recorded commit; local tools or artifact registries can still affect build-tool-backed resolution |
30+
| Live service | Uses a pinned project with current advisory data; the result is a dated observation |
31+
| Manual assurance | Runs a separately started GitHub Actions workflow and saves its detailed report as an artifact |
32+
33+
Each catalog case must state both what it proves and what it does not prove.
34+
Remote Git inputs include a full commit revision. Fixtures, workflows, and
35+
expected results include SHA-256 checksums.
36+
37+
## Case studies
38+
39+
- [Dependency graph evidence](evidence/DEPENDENCY_GRAPHS.md) covers npm, pnpm,
40+
Yarn, Bun, Go, Python, and Maven graphs, plus visible degraded fallback
41+
behavior.
42+
- [Policy and vulnerability-guidance evidence](evidence/POLICY_AND_GUIDANCE.md)
43+
covers vulnerability and SPDX policy, baselines, source changes, persisted
44+
findings, reachability tiers, and read-only remediation suggestions.
45+
- [Targets and operational assurance](evidence/TARGETS_AND_OPERATIONS.md)
46+
covers local and Git projects, containers, SBOM ingestion and validation,
47+
repeated unit tests across supported systems, release builds, and repeatable
48+
performance measurements.
49+
50+
## Dated workflow evidence
51+
52+
The
53+
[SBOM interoperability run from July 24, 2026](https://github.com/bomly-dev/bomly-cli/actions/runs/30057587653)
54+
completed successfully at commit
55+
`9530b9f3bfcb3fe1d2748fa2bcfadb5e53e3346c`. The workflow generated canonical
56+
SPDX 2.3 and CycloneDX 1.6 documents and checked them with its recorded
57+
checksum-pinned validators.
58+
59+
The
60+
[portable stability run from July 24, 2026](https://github.com/bomly-dev/bomly-cli/actions/runs/30065452505)
61+
completed successfully at commit
62+
`e6bc5235f85dc909b6bc73f6ba9eb82c22c44ac4`. It repeated Go unit tests on
63+
Linux, macOS, and Windows, repeated the Java-related and full Linux unit-test
64+
suites, and built every release target. It did not run remote smoke tests.
65+
66+
Workflow summaries explain what ran, the result, and where to inspect failures.
67+
Their downloadable artifacts retain detailed commands, versions, diagnostics,
68+
and hashes where the workflow produces a run manifest.
69+
70+
## How to read the results
71+
72+
- A checked golden proves the normalized result for its recorded input. It is
73+
not a promise that every project in that ecosystem has the same fidelity.
74+
- A live enrichment golden can change when advisory services add, correct, or
75+
withdraw records. Bomly does not claim an immutable offline advisory view.
76+
- A package count alone is not graph proof. Review identities, versions,
77+
relationships, scopes, sources, and occurrence paths.
78+
- A fallback warning means useful evidence may still exist, but coverage can
79+
be lower than the preferred detector.
80+
- `unreachable` does not mean safe. The analyzer and tier define what was
81+
checked.
82+
- Remediation output is read-only guidance. It does not apply or validate a
83+
package change.
84+
- SBOM validation proves acceptance by the named validator versions for the
85+
canonical fixtures, not lossless conversion for every producer or consumer.
86+
87+
## Provenance
88+
89+
The tracked cases use Bomly-owned fixtures, Bomly-owned example repositories,
90+
or complete public input repositories. The descriptions and test structure
91+
are written for Bomly's own graph, pipeline, and output contracts.

‎docs/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ Task-oriented walkthroughs.
99
- [Installation](INSTALLATION.md) — install methods, `bomly` vs `bomly-lite`, checksum verification, uninstall
1010
- [Getting Started](GETTING_STARTED.md) — first scan, enrich, audit, diff
1111
- [Use Cases](USE_CASES.md) — recipes for PR gates, SBOMs, triage, license and offline scans
12+
- [Reproducible Evidence](EVIDENCE.md) — public inputs, commands, results, and limitations behind important behavior claims
1213
- [Scan Targets](SCAN_TARGETS.md) — directories, Git repos, containers, SBOMs
1314
- [Output Formats](OUTPUT_FORMATS.md) — text, JSON, SARIF, SBOM
1415
- [SBOM Formats](SBOM.md) — SPDX vs. CycloneDX, write and ingest

‎docs/evidence/TARGETS_AND_OPERATIONS.md‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# Targets and operational assurance
2+
3+
Bomly accepts local projects, Git repositories, container images, and existing
4+
SBOMs. The public evidence uses the same engine paths as the CLI and states
5+
where the input itself is not immutable.
6+
7+
## Target cases
8+
9+
| Target | Public case | What is checked |
10+
| --- | --- | --- |
11+
| Local project | `baseline-policy` | A public Git fixture is materialized locally, then scanned and audited through `--path` |
12+
| Git repository | `graph-npm`, `graph-go`, and the other graph cases | The CLI clones a recorded commit and runs the selected detector |
13+
| Container image | `container-inventory` | Built-in inventory reads packages from the checked Alpine image |
14+
| SPDX SBOM | `sbom-spdx-ingest` | The checked SPDX 2.3 graph is ingested through the SBOM detector |
15+
| CycloneDX SBOM | `sbom-cyclonedx-ingest` | The checked CycloneDX 1.6 graph is ingested through the SBOM detector |
16+
17+
Inspect or reproduce one:
18+
19+
```sh
20+
make evidence CASE=container-inventory
21+
make evidence CASE=sbom-spdx-ingest
22+
```
23+
24+
The container smoke case currently uses `alpine:3.20`. The tag can move, so
25+
the checked-in golden is explicitly a snapshot rather than an immutable image
26+
claim. The Git cases separately record the full commit behind their readable
27+
tag or ref.
28+
29+
## Example SBOM workflow
30+
31+
A release engineer receives a supplier SBOM and wants to apply the same policy
32+
used for source scans:
33+
34+
```sh
35+
bomly scan \
36+
--sbom \
37+
--path supplier.spdx.json \
38+
--enrich \
39+
--audit \
40+
--fail-on high
41+
```
42+
43+
The deterministic ingestion cases check Bomly's internal graph. The manually
44+
started interoperability workflow adds an external check:
45+
46+
```sh
47+
gh workflow run sbom-interoperability.yml
48+
```
49+
50+
It generates canonical SPDX 2.3 and CycloneDX 1.6 files, verifies the
51+
downloaded validator checksums, runs the named official validators, and saves
52+
the generated files plus a `bomly.sbom-assurance-run/v1` report. See
53+
[`test/assurance/SBOM_INTEROPERABILITY.md`](../../test/assurance/SBOM_INTEROPERABILITY.md)
54+
for the workflow summary and failure-investigation steps.
55+
56+
The public catalog records the workflow checksum under
57+
`sbom-interoperability`. Validator versions and download checksums stay in the
58+
workflow so changing either requires an intentional evidence update.
59+
60+
## Supported-system checks
61+
62+
The `portable-platforms` case starts:
63+
64+
```sh
65+
gh workflow run portable-assurance.yml
66+
```
67+
68+
This workflow:
69+
70+
- runs the Go unit tests twice on Linux, macOS, and Windows;
71+
- repeats Java-related unit tests ten times;
72+
- repeats all Go unit tests five more times on Linux;
73+
- builds full and lightweight binaries for every release target.
74+
75+
It is important to be precise: this is unit-test and build assurance. It does
76+
not run smoke tests against public repositories, container registries, or
77+
advisory services. The workflow summary identifies each group and explains
78+
how to open failed logs. See
79+
[`test/assurance/BENCHMARK_RUNS.md`](../../test/assurance/BENCHMARK_RUNS.md)
80+
for the full description.
81+
82+
## Repeatable performance measurements
83+
84+
Run:
85+
86+
```sh
87+
make benchmark-samples
88+
```
89+
90+
The `performance-stability` case uses the checked SPDX fixture and the
91+
lightweight Bomly binary. It records five isolated cold-cache scans and five
92+
shared-cache warm scans under `.benchmark-runs/performance`.
93+
94+
The resulting `bomly.benchmark-run/v1` report includes:
95+
96+
- repository and executable revisions and hashes;
97+
- host and Go runtime details;
98+
- exact command, working directory, cache mode, and network state;
99+
- exit status, output size and hashes, timing, and peak memory for every run;
100+
- median, variation, and an approximate 95% confidence interval.
101+
102+
The stable gates are successful exit status, normalized output consistency,
103+
and an optional explicit output-size cap. Wall time and memory remain
104+
machine-specific measurements for review rather than universal limits.
105+
106+
## Example release-confidence workflow
107+
108+
Before a broad release:
109+
110+
1. Run `make test` and the relevant pinned smoke slices.
111+
2. Run `make benchmark-samples` and compare the report with the previous run
112+
from a comparable host.
113+
3. Start the portable workflow and read its Summary page.
114+
4. If SBOM output changed, start the interoperability workflow and inspect its
115+
generated artifact hashes and validator results.
116+
5. Record the repository commit with every retained workflow or benchmark
117+
report.
118+
119+
## Limits
120+
121+
- Remote services and package registries can be temporarily unavailable.
122+
- Build-tool-backed graph resolution can vary with tool versions; the catalog
123+
states required tools and checked source revisions.
124+
- Local repository scans can contain any number of individually bounded
125+
files. Per-file parser limits do not create a total project-size limit.
126+
- Portable Git options bound time and checkout validation but cannot reliably
127+
cap transfer bytes or `.git` object storage before checkout completes.
128+
- Workflow artifacts have retention periods. The workflow file, pinned tools,
129+
reproduction command, and result link remain public after an artifact
130+
expires, but the raw artifact may need to be regenerated.

‎docs/manifest.json‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,12 @@
4545
"description": "Recipes for PR gates, SBOMs, triage, and license and offline scans.",
4646
"group": "start"
4747
},
48+
{
49+
"slug": "evidence",
50+
"title": "Reproducible evidence",
51+
"description": "Public inputs, commands, results, and limitations behind important behavior claims.",
52+
"group": "start"
53+
},
4854
{
4955
"slug": "scan-targets",
5056
"title": "Scan targets",

0 commit comments

Comments
 (0)