Extend /scanner/benchmark comparator to score against facade's merged /scanner/dast + /scanner/sast

[preetkaran20]

Comparator currently only scores against VulnerableApp's own ground truth. Extend it to pull
ground truth from facade's new merged endpoints (/scanner/dast, /scanner/sast — see
VulnerableApp-facade ticket) instead of hitting each app individually, then score submitted
findings against the combined set.

Scope:

• Fetch merged ground truth from facade's /scanner/dast + /scanner/sast (single call each,
  no per-app discovery needed on this side — facade already did the fan-out/merge)
• Run existing matching rules (any-axis match for DAST; filePath+line for SAST) against the
  merged ground truth
• Coverage/missed/unmatched report labeled per-app (facade's merged response should retain
  which app each entry came from, so scoring can attribute results correctly)
• Bump benchmarks/<tool>-results.json schema/version to reflect multi-app scope
• Fallback: if facade is unreachable, fall back to VulnerableApp's own ground truth only
  (degrade gracefully rather than failing the whole benchmark run)

[vianbas]

Not claiming this yet. Here is the state of what the scope list asks for, and one decision I would rather not make alone.

The DAST half looks like configuration. It is not. The comparator already fetches ground truth over HTTP from benchmark.dast.ground-truth.url, and both benchmarks/README.md and the comparator's own javadoc tell you to point that at the facade when running composed, the javadoc adding that coverage then reflects the full attack surface. Pointing it there fails today. The facade merges into an object keyed by app name, {"VulnerableApp": [...], "VulnerableApp-jsp": [...]}, while DastBenchmarkStrategy asks for an array. Jackson stops on the first token:

Cannot deserialize value of type `[Lorg.sasanlabs.beans.ScannerResponseBean;`
from Object value (token `JsonToken.START_OBJECT`)

which surfaces as IOException: Failed to fetch DAST ground truth from .... The run fails outright rather than degrading.

I ran that rather than reasoning about it, twice and two ways: a real RestTemplate with the converters Boot builds, driven against a mock HTTP server, and the ObjectMapper on its own. The array shape was the control in both, and it passed in both, scoring 100 percent coverage through the comparator and parsing cleanly through the mapper. The merged body I fed in is the one merge_vulnerability_information builds in the facade. I did not stand up the composed stack to capture a live response, so that shape comes from the facade's own merge code rather than from the wire.

Nothing in the suite covers it. The only test that touches the fetch, DastBenchmarkStrategyTest, mocks RestTemplate and hands back beans directly, so nothing exercises deserialising a ground-truth response.

The SAST half is not configuration at all. SastBenchmarkStrategy goes through IExpectedIssuesProvider, whose only implementation loads from the classpath or a file. There is no HTTP path in it, so pulling merged ground truth from the facade is unimplemented on that side.

Two smaller items from the list. Per-app labelling has its data available already, since the facade keys each app's entries by name, but Finding and BenchmarkResult carry no app field to put it in. The fallback does not exist either: fetchGroundTruth throws.

The decision. Does the comparator learn to read the keyed object, or does the facade flatten to an array? Both score correctly. Only the first keeps per-app attribution for free, which is what this ticket asks for, so I lean that way, but it is your call and it lands in two repositories.

One dependency worth naming: for its merged DAST response the facade still captures the Java app at the bare /scanner. SasanLabs/VulnerableApp-facade#123 changes that and is open.

#730 sits behind this. Its ZAP modern leg scans the facade, so any finding from the jsp or php apps can only land as unmatched until the comparator scores against merged ground truth.

[vianbas]

Thanks for the assign. One thing to confirm before I change the comparator.

DastBenchmarkStrategy asks for ScannerResponseBean[], a JSON array. The facade's /scanner/dast returns an object keyed by application name. The two shapes do not meet, so one side has to move.

I think the scope already decides it: "facade's merged response should retain which app each entry came from". Flattening on the facade side would drop exactly that. So the comparator is the side that learns to read the keyed object, and the change stays here.

Confirming rather than assuming, because the alternative lands in the facade instead.

Two things that shape the work either way

The SAST half is not configuration. SastBenchmarkStrategy takes an IExpectedIssuesProvider, and the only implementation in src/main is CsvExpectedIssuesProvider, which reads a classpath resource or a file. Nothing on that side speaks HTTP, so merged SAST ground truth needs writing rather than pointing.

Finding and BenchmarkResult carry no application field, so the per-app labelling in the scope needs one added.

Worth knowing for scope: /scanner/dast and /scanner/sast capture VulnerableApp, VulnerableApp-jsp and VulnerableApp-php. Only /VulnerabilityDefinitions captures llmforge.

Read on VulnerableApp deedb7f and facade c7efe62.