Integrate Anthropic's Defending Code Harness

[azchin]

Based on https://github.com/anthropics/defending-code-reference-harness and https://claude.com/blog/using-llms-to-secure-source-code

The pipeline is similar to ours, with bug-finding, triage, patching. Threat modelling is essentially harness generation in our scope, but can still be used for other steps.

We need to be more flexible with our artifact types. Something like registering a new artifact type with libCRS would be good, along with some metadata describing it. This is to accommodate things like TRIAGE.json.

We do not need to directly port the harness/ reference since we have our own sandbox. Looking into the parallelized arch is still meaningful.

If someone is interested in picking this up, I'll follow up with more concrete ideas. Otherwise I'll assign myself when I find bandwidth.

[tusharshah21]

Hey so any flow how you want to trigger the harness?

[azchin]

Which harness are you referring to?

For the fuzzing harness? That's given to the (bug-finding and bug-fixing) CRSs so they can focus on one particular entrypoint. Please check docs for more info.

As for AI harness, we have reference CRSs (e.g. crs-bug-finding-claude-code) that can be followed.

[goutamadwant]

I would like to take a current-main-aware, non-overlapping first slice here. I rechecked the repository: #323 already added report artifacts, generic host inputs, and cross-run forwarding, while #356 is adding source-only/auditing runs. The remaining gap from the Anthropic reference pipeline appears to be preserving semantic artifact identity and schema across stages (for example, distinguishing a TRIAGE.json result from a generic report), rather than porting its sandbox or orchestration.

Proposed first boundary:

• declare logical artifact descriptors in crs.yaml (name, compatible physical channel, media type, and optional schema URI/version);
• expose those descriptors to libCRS and attach a small manifest when artifacts are submitted/fetched;
• keep the existing physical directories and CLI flags unchanged for backward compatibility;
• leave source-only execution, harness handling, and parallel scheduling to Add support for running run, artifacts, and archive subcommands without a fuzzing harness #356/current orchestration.

Would you prefer this compatibility-first logical layer, or truly dynamic exchange directories and generated CLI flags? I have prepared an exploratory design with tests and migration-free defaults pending that choice.

[azchin]

Could you concretize what sort of artifacts Anthropic's pipeline uses? My original comment was before we added the report as artifact as an active contribution, the TRIAGE.json seems like it can fit there if nothing else is needed.

FYI, the ultimate goal here is to port Anthropic's prototype as multiple new CRSs.

[goutamadwant]

Thanks @azchin that clarification helps. I reviewed the current reference harness in detail, and I agree that TRIAGE.json itself can fit within the existing report artifact.

The Anthropic workflow produces two related artifact sets:

Interactive/static workflow

• THREAT_MODEL.md
• VULN-FINDINGS.json and its Markdown rendering
• TRIAGE.json and its Markdown rendering
• PATCHES/, containing candidate diffs and patch verification results

Autonomous pipeline

• a proof-of-concept input plus reproduction command, crash type, sanitizer output, and exit code
• a grader verdict for the candidate
• a structured exploitability report with severity, reachability, and evidence
• patch.diff
• patch_result.json, containing build, reproduction, regression, and re-attack results

My current mapping to OSS-CRS would be:

• proof-of-concept bytes -> pov
• unverified crash/finding metadata -> bug-candidate
• threat model, triage output, exploitability analysis, and verification evidence -> bundled report
• accepted candidate diff -> patch

That suggests the current artifact channels are sufficient, and my earlier logical-artifact proposal would be broader than necessary.

For the port itself, I would split the prototype into independently composable CRSs:

1. a recon/bug-finding CRS that emits PoVs, bug candidates, and its threat-model/finding report bundle;
2. a verification/triage CRS that consumes those artifacts and emits ranked, deduplicated reports;
3. a patching CRS that consumes verified findings and PoVs, then emits patches plus verification reports.

I propose starting with the first CRS as the smallest end-to-end contribution, reusing OSS-CRS sandboxing and libCRS rather than porting Anthropic's orchestration layer. Does that match the decomposition you have in mind? let me know your thoughts on this to proceed. thanks!

[azchin]

a proof-of-concept input

Yes, maps directly to pov.

plus reproduction command

This one we don't have yet, if we can standardize the way this is outputted (e.g. reproduce.sh which accepts the crashing input + optionally builds the environment for the repro) then that would be a great contribution to the framework. But I don't have enough examples / intuition on what that looks like yet.

a grader verdict for the candidate

That fits into our triage CRS type, which currently just acts like a PoV filter. And current implementations are purely algorithmic. An agentic one would be appreciated.

If the triager needs to output more, like TRIAGE.json, should we still output it as a report? Or semantically should it be separated?

threat model, triage output, exploitability analysis, and verification evidence

report fits here, but again, would we benefit from separating threat model and triage into its own artifact? I'd like to see some examples of a threat model and triage output from the agent.

containing build, reproduction, regression, and re-attack results

The build is standardize so we don't need to output anything there. I'd like to know more about regression and re-attack results if possible.