From d449b115124c042f689704e4050125aa41397c11 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Sun, 5 Jul 2026 21:47:01 +0200 Subject: [PATCH 01/35] docs: design spec for call-stack AST-detach heap reduction MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- ...-callstack-detach-heap-reduction-design.md | 126 ++++++++++++++++++ 1 file changed, 126 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-05-callstack-detach-heap-reduction-design.md diff --git a/docs/superpowers/specs/2026-07-05-callstack-detach-heap-reduction-design.md b/docs/superpowers/specs/2026-07-05-callstack-detach-heap-reduction-design.md new file mode 100644 index 00000000..900c78f4 --- /dev/null +++ b/docs/superpowers/specs/2026-07-05-callstack-detach-heap-reduction-design.md @@ -0,0 +1,126 @@ +# Call-stack Detach — Heap Reduction Design Spec + +**Status:** Draft for review +**Date:** 2026-07-05 +**Supersedes / refines:** `docs/superpowers/plans/2026-07-05-callstack-hooks-heap-reduction.md` (Phase 2 section) +**Decision this spec records:** go straight to the structural fix (detach recorded calls from the AST), skipping the plan's incremental Phase 1 filter/caps. + +## Goal + +Cut the project-lifetime heap held by `com.ibm.engine.callstack` on large scans — measured at ~179k retained `CallContext`s driving ~7 GB and climbing on a near-full keycloak scan, growing linearly and unbounded with project size — **without regressing cross-file detection**. Cross-file detection has no CI coverage today (every `CheckVerifier` test is single-file), so correctness preservation is the dominant constraint. + +## Root cause (verified against current code) + +One static `CallStackAgent` per language accumulates a `CallContext(tree, scanContext)` for **every recorded method invocation / enum access** across the whole scan and is released only at end-of-scan. `CallStackAgent.invokedCallStack` (`CallStackAgent.java:42`) is the holder. + +The heap is dominated not by the *count* of `CallContext`s but by **per-file AST pinning**. Each `CallContext`: +- holds a `Tree` whose `parent()` chain reaches the file's `CompilationUnitTree`, and +- holds `JavaScanContext`, a `record(JavaFileScannerContext)` (`JavaScanContext.java:29`) that pins the whole file's AST a second way. + +So a *single* surviving `CallContext` pins its entire file AST for the whole scan. Retained heap ≈ (distinct files with ≥1 recorded call) × (avg file AST size). Reducing the *count* (the plan's Phase 1 filter/caps) does not unpin a file that has even one surviving call — which is why this spec goes straight to detaching trees. + +### The tree is load-bearing in three distinct places + +A recorded call's tree is read at three points. All three must be handled to hold zero trees: + +1. **Match** — `MethodMatcher.match(callContext.tree(), …)` in `onNewHookSubscription` (`CallStackAgent.java:107`) and via `HookRepository.update` → `hook.isInvocationOn(callContext, …)` (`HookRepository.java:117`). Reads invoked-object type, method name, parameter types off the tree. + - *Snapshot-able at record time* — these three keys are all derivable while the file is live. + +2. **Resolution input** — at fire time, `DetectionStoreWithHook.handleMethodInvocationHookWithParameterResolvement` (`DetectionStoreWithHook.java:124`) runs `extractArgumentFromMethodCaller` (`JavaDetectionEngine.java:121`) + `resolveValuesInInnerScope` (`:169`) on the **live** call-site argument tree and file symbol table. + - *Verified narrowing (Java):* the parameter hook is always built with the 4-arg constructor (`JavaDetectionEngine.java:477`), so `expressionToResolve` is always `null` (`MethodInvocationHookWithParameterResolvement.java:43`). The cross-boundary branch never fires for Java. Fire-time resolution is therefore always "resolve the call-site argument directly," which is **fully reproducible at record time** because the file is live then. + - *Verified narrowing (factory influence):* `valueFactory` steers traversal in exactly one place — `if (valueFactory instanceof SizeFactory)` for `NEW_ARRAY` args (`JavaDetectionEngine.java:275`). Everywhere else the factory is threaded but not inspected; the raw `ResolvedValue` list is factory-independent, and the factory is applied *afterward* (`DetectionStoreWithHook.java:185`, `ValueDetection.toValue`). + +3. **Detection output (location)** — the produced `IValue.getLocation()` returns the tree (`IValue.java:26`; each of 27 model value classes stores its own `T location`). The sole production consumer is the translator: `JavaTranslator.getDetectionContextFrom` (`JavaTranslator.java:170`) reduces the tree to `DetectionLocation(filePath, lineNumber, columnOffset, keywords, bundle)` — every field derivable at record time. + - *Consequence:* to emit a cross-file detection without holding the AST, the produced `IValue`'s location must be a tree-free snapshot. This forces a location-model change (see Scope Revision). + +## Scope revision vs. the original plan + +The plan constrained changes to "strictly callstack/hooks." **That constraint is not achievable for a faithful detach.** Point 3 above proves that every cross-file detection's produced value must carry a tree-free location, which touches the value-location model. This spec therefore expands scope to include a location abstraction across the model/factory/translator seam. This is the single largest cost of the work and the main review surface. + +## Chosen architecture: Hybrid detach with tree-fallback + +Detach (store a tree-free record) **only** for calls whose fire-time behavior is provably reproducible from a record-time snapshot; **keep the live tree** (today's behavior, that file stays pinned) for the residual. This guarantees no Java detection loss while unpinning the common case. + +### 1. Detachability predicate (record time, Java) + +At `addCall`, run the existing resolution over each argument with a `null` factory and record whether resolution descended into a `NEW_ARRAY`. A recorded call is **detachable** iff: +- it is a method invocation whose argument resolution did **not** touch a `NEW_ARRAY` (the only factory-steered case; a future `SizeFactory` hook would want the array size, not its elements), and +- (Java) always — since `expressionToResolve` is always null, there is no cross-boundary case. + +Otherwise the call is **non-detachable** → keep the live `Tree` (fallback path, unchanged behavior). Python/Go are **always non-detachable** in this iteration (they use `expressionToResolve`; their semantics are out of scope) → they keep the tree exactly as today. The predicate lives in the language layer so the generic engine stays language-agnostic. + +### 2. Detached record shape + +Replace `CallContext(tree, scanContext)` with a sealed shape: +- `RetainedCall(T tree, IScanContext scanContext)` — the fallback (non-detachable) case, identical to today. +- `DetachedCall` — tree-free, holding: + - **match keys:** invoked-object `IType`, method name, parameter `IType`s (for `MethodMatcher` without a tree); + - **pre-resolved arguments:** for each argument index, the raw resolved value(s) (`Object`) plus a `LocationSnapshot` (see §4). If *any* argument fails to pre-resolve at record time, the call is treated as non-detachable and keeps its tree (fallback) — never a silent per-argument drop; + - for enum accesses: the enum class name plus a snapshot map `{constantName → resolvedValue + LocationSnapshot}` (fire-time `EnumHook` selection picks by name). + +Match uses the keys; the argument/enum snapshots feed fire-time replay. No `Tree` and no `JavaFileScannerContext` are retained → the file AST becomes GC-eligible after `leaveFile`. + +### 3. Fire-time replay + +`onNewHookSubscription` and `HookRepository.update` match against the record's keys instead of a tree. On a match of a `DetachedCall`, `DetectionStoreWithHook` skips `extractArgumentFromMethodCaller`/`resolveValuesInInnerScope` and instead takes the pre-resolved snapshot for the hook's parameter index, applies the hook's factory to it, and proceeds through `handleNextRulesForMethodHooks` as today (that path visits `hook.methodDefinition()`, from the hook's own file, and is unaffected by detachment). `RetainedCall`s replay exactly as today. + +### 4. Location abstraction (the model change) + +Introduce an engine-level location carrier (no dependency on `mapper`): + +``` +sealed interface Location permits TreeLocation, SnapshotLocation + TreeLocation(T tree) // normal / fallback path + SnapshotLocation(String filePath, int line, int columnOffset, List keywords) +``` + +- `ResolvedValue` carries a `Location` instead of a raw `T tree`. Normal resolution wraps the leaf tree in `TreeLocation`; record-time pre-resolution captures a `SnapshotLocation` (computed with the same logic `JavaTranslator.getDetectionContextFrom` uses today). +- The 27 model value classes store `Location` instead of `T`; the 27 factories pass `resolvedValue.location()`. +- Translators handle both: `SnapshotLocation` → build `DetectionLocation` directly; `TreeLocation` → existing derivation. Python/Go always produce `TreeLocation`, so their behavior is unchanged. + +This is mechanical but wide (~27 model + ~27 factory + 3 translator edits, plus `ResolvedValue` construction sites in the Java/Python/Go engines and tests that call `getLocation()`/`reportIssue`). It is exercised on the **main** cross-file path, not just edge cases. + +### 5. Lossless cleanups folded in + +- Delete the redundant `visitedTreeObjects` set (`CallStackAgent.java:45`); dedup within the per-name bucket (buckets are small once keys index them). Measured 100% redundant with `invokedCallStack`. +- Key-indexed subscription lookup in `onNewHookSubscription` (`CallStackAgent.java:101`): derive the name key from the hook value and scan only that bucket, with a fallback scan for multi-name/`ANY` matchers. + +## Bounded, documented loss + +- **Java:** zero detection loss by construction — the only non-reproducible cases (`NEW_ARRAY`/`SizeFactory`, cross-boundary `expressionToResolve`) are kept on the tree-fallback path. If measurement later shows the array-fallback set is large enough to matter for heap, a follow-up can pre-resolve both interpretations; not in this iteration. +- **Python/Go:** unchanged (always tree-fallback). No detach, no loss, no heap win for those languages yet. +- Any record-time resolution failure logs at DEBUG and falls back to keeping the tree, never to silent loss. + +## Blast radius / files + +- `engine/.../callstack/CallStackAgent.java` — key-indexed lookup; drop `visitedTreeObjects`; record `DetachedCall` vs `RetainedCall`; match against keys. +- `engine/.../callstack/CallContext.java` — becomes the sealed record family (`RetainedCall`/`DetachedCall`). +- `engine/.../detection/ResolvedValue.java` — carry `Location`. +- `engine/.../model/Location.java` (+ `TreeLocation`, `SnapshotLocation`) — NEW. +- `engine/.../model/*.java` (27) + `engine/.../model/factory/*.java` (27) — location field/parameter type. +- `engine/.../detection/DetectionStoreWithHook.java` — detached replay branch. +- `engine/.../language/java/JavaDetectionEngine.java` — record-time pre-resolution + `NEW_ARRAY` detection; `ResolvedValue` construction. +- `engine/.../language/{python,go}/*DetectionEngine.java` — `ResolvedValue`/`TreeLocation` construction (behavior-preserving). +- `engine/.../language/ILanguageSupport.java` (or `ILanguageTranslation.java`) — `isDetachableCall` predicate (default: not detachable). +- `{java,python,go}/.../translation/translator/*Translator.java` — handle `SnapshotLocation`. +- `java/src/test/files/...` + a new multi-file `CheckVerifier` test — NEW (see Verification). + +## Verification + +- **Cross-file regression guard (new, does not exist today):** a multi-file `CheckVerifier` test where a hook created in file B resolves a call recorded in file A. Must cover **both** a detached argument (literal / constant / field / nested-call) **and** a tree-fallback argument (`NEW_ARRAY`/size), asserting both still detect. +- `mvn test -pl engine` and `mvn test -pl java` stay green throughout (run `mvn spotless:apply` before commits; restore `JsonCipherSuites` if Spotless truncates it). +- **Heap profile:** scan a large project (e.g. keycloak via `mvn sonar:sonar`, or supply `sonar.java.libraries`) under a constrained `MAVEN_OPTS="-Xmx"`; capture `jmap -histo` before/after and record the drop in retained `CallContext`/AST and peak heap. Types must resolve (source-only scans fire `addCall` zero times). + +## Risks & open implementation questions + +- **Location-model diff size** is the top risk — wide but mechanical. Mitigate by landing it as an isolated, behavior-preserving commit (`TreeLocation` everywhere) before any detach logic, so it can be reviewed/tested independently. +- **Record-time pre-resolution fidelity:** must produce exactly the raw `ResolvedValue`s fire-time would, minus the `SizeFactory` case. Guard with a test that resolves the same argument both ways and asserts equality. +- **Enum snapshot completeness:** pre-resolving all enum constants at record time must match `resolveEnumValue`'s selection semantics; verify against existing enum-hook tests. +- **`SnapshotLocation` fidelity:** the `keywords`/line/offset must match what `getDetectionContextFrom` produces from the live tree, so CBOM occurrences are identical for detached vs. non-detached detections. Verify via an output-level test. +- **`filePath` origin for cross-file values:** confirm the detached value's `SnapshotLocation.filePath` is file A's (the call site), matching today's tree-derived behavior. + +## Explicitly out of scope + +- Python/Go detachment (kept on tree-fallback). +- The rule-graph construction OOM (`MethodMatcher.`, tracked in #476/#477). +- Retention caps / lossy bounding from the original plan's Phase 1 (Task 5) — the detach makes them unnecessary; revisit only if post-detach record *count* itself proves a problem. From 675e5c31d45bb10f70285d28b15e878d35187fad Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Sun, 5 Jul 2026 21:53:56 +0200 Subject: [PATCH 02/35] docs: use synthetic detached SyntaxToken instead of value-model change MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- ...-callstack-detach-heap-reduction-design.md | 56 +++++++++---------- 1 file changed, 27 insertions(+), 29 deletions(-) diff --git a/docs/superpowers/specs/2026-07-05-callstack-detach-heap-reduction-design.md b/docs/superpowers/specs/2026-07-05-callstack-detach-heap-reduction-design.md index 900c78f4..7d9fc47e 100644 --- a/docs/superpowers/specs/2026-07-05-callstack-detach-heap-reduction-design.md +++ b/docs/superpowers/specs/2026-07-05-callstack-detach-heap-reduction-design.md @@ -31,11 +31,15 @@ A recorded call's tree is read at three points. All three must be handled to hol - *Verified narrowing (factory influence):* `valueFactory` steers traversal in exactly one place — `if (valueFactory instanceof SizeFactory)` for `NEW_ARRAY` args (`JavaDetectionEngine.java:275`). Everywhere else the factory is threaded but not inspected; the raw `ResolvedValue` list is factory-independent, and the factory is applied *afterward* (`DetectionStoreWithHook.java:185`, `ValueDetection.toValue`). 3. **Detection output (location)** — the produced `IValue.getLocation()` returns the tree (`IValue.java:26`; each of 27 model value classes stores its own `T location`). The sole production consumer is the translator: `JavaTranslator.getDetectionContextFrom` (`JavaTranslator.java:170`) reduces the tree to `DetectionLocation(filePath, lineNumber, columnOffset, keywords, bundle)` — every field derivable at record time. - - *Consequence:* to emit a cross-file detection without holding the AST, the produced `IValue`'s location must be a tree-free snapshot. This forces a location-model change (see Scope Revision). + - *Consequence:* the produced `IValue`'s location must not pin the AST. We satisfy this **without changing the value model** by using a synthetic detached `SyntaxToken` (see §4). -## Scope revision vs. the original plan +## Keeping `T = Tree` (no value-model change) -The plan constrained changes to "strictly callstack/hooks." **That constraint is not achievable for a faithful detach.** Point 3 above proves that every cross-file detection's produced value must carry a tree-free location, which touches the value-location model. This spec therefore expands scope to include a location abstraction across the model/factory/translator seam. This is the single largest cost of the work and the main review surface. +An earlier reading of point 3 suggested every model value would need a `Location` abstraction (re-typing 27 model classes + 27 factories). That is **not necessary**. The value model's location `T` stays `Tree`; we supply a **synthetic detached `Tree`** for cross-file detections: + +- `org.sonar.plugins.java.api.location.Position.at(int,int)` and `Range.at(...)` are public static factories producing AST-free coordinate objects; `SyntaxToken` is a small interface (`text/trivias/line/column/range` + `Tree`'s `is/accept/parent/firstToken/lastToken/kind`). +- A `DetachedSyntaxToken implements SyntaxToken` backed by captured primitives (line, column offsets, and the record-time keywords) returns `parent() == null` and `firstToken()/lastToken() == this`, so it pins nothing. +- The existing factories build `IValue` with this token as the location, unchanged. Scope stays essentially within callstack/hooks + one new token class + a single translator branch. The plan's "strictly callstack/hooks" spirit holds. ## Chosen architecture: Hybrid detach with tree-fallback @@ -55,8 +59,8 @@ Replace `CallContext(tree, scanContext)` with a sealed shape: - `RetainedCall(T tree, IScanContext scanContext)` — the fallback (non-detachable) case, identical to today. - `DetachedCall` — tree-free, holding: - **match keys:** invoked-object `IType`, method name, parameter `IType`s (for `MethodMatcher` without a tree); - - **pre-resolved arguments:** for each argument index, the raw resolved value(s) (`Object`) plus a `LocationSnapshot` (see §4). If *any* argument fails to pre-resolve at record time, the call is treated as non-detachable and keeps its tree (fallback) — never a silent per-argument drop; - - for enum accesses: the enum class name plus a snapshot map `{constantName → resolvedValue + LocationSnapshot}` (fire-time `EnumHook` selection picks by name). + - **pre-resolved arguments:** for each argument index, the raw resolved value(s) (`Object`) plus captured location primitives (line, offset, keywords) for a `DetachedSyntaxToken` (see §4). If *any* argument fails to pre-resolve at record time, the call is treated as non-detachable and keeps its tree (fallback) — never a silent per-argument drop; + - for enum accesses: the enum class name plus a snapshot map `{constantName → resolvedValue + location primitives}` (fire-time `EnumHook` selection picks by name). Match uses the keys; the argument/enum snapshots feed fire-time replay. No `Tree` and no `JavaFileScannerContext` are retained → the file AST becomes GC-eligible after `leaveFile`. @@ -64,21 +68,16 @@ Match uses the keys; the argument/enum snapshots feed fire-time replay. No `Tree `onNewHookSubscription` and `HookRepository.update` match against the record's keys instead of a tree. On a match of a `DetachedCall`, `DetectionStoreWithHook` skips `extractArgumentFromMethodCaller`/`resolveValuesInInnerScope` and instead takes the pre-resolved snapshot for the hook's parameter index, applies the hook's factory to it, and proceeds through `handleNextRulesForMethodHooks` as today (that path visits `hook.methodDefinition()`, from the hook's own file, and is unaffected by detachment). `RetainedCall`s replay exactly as today. -### 4. Location abstraction (the model change) - -Introduce an engine-level location carrier (no dependency on `mapper`): +### 4. Detached location via synthetic `SyntaxToken` (no model change) -``` -sealed interface Location permits TreeLocation, SnapshotLocation - TreeLocation(T tree) // normal / fallback path - SnapshotLocation(String filePath, int line, int columnOffset, List keywords) -``` +`getLocation()` keeps returning `Tree`; for a detached call we hand the factory a `DetachedSyntaxToken` instead of a real leaf tree: -- `ResolvedValue` carries a `Location` instead of a raw `T tree`. Normal resolution wraps the leaf tree in `TreeLocation`; record-time pre-resolution captures a `SnapshotLocation` (computed with the same logic `JavaTranslator.getDetectionContextFrom` uses today). -- The 27 model value classes store `Location` instead of `T`; the 27 factories pass `resolvedValue.location()`. -- Translators handle both: `SnapshotLocation` → build `DetectionLocation` directly; `TreeLocation` → existing derivation. Python/Go always produce `TreeLocation`, so their behavior is unchanged. +- **New class** `DetachedSyntaxToken implements SyntaxToken` (engine, `com.ibm.engine…`): fields = start/end line + column offsets and the record-time `keywords`. `range()` → `Range.at(Position.at(line, col), Position.at(endLine, endCol))`; `firstToken()/lastToken()` → `this`; `kind()` → `Tree.Kind.TOKEN`; `parent()` → `null`; `accept()` → no-op; `text()` → the resolved value string; `trivias()` → empty. Value `equals/hashCode` over the fields (model `equals` compares `getLocation()`). +- **Record time:** while the argument's leaf tree is live, capture (line, columnOffset, end position, keywords) — computed by the *same* logic `JavaTranslator.getDetectionContextFrom` uses (`JavaTranslator.java:170`) — and store the primitives in the `DetachedCall`. No `Tree`/`Range`/`Position` object is retained (store ints), so nothing pins the AST. +- **Fire time:** build `new ResolvedValue<>(rawValue, new DetachedSyntaxToken(...))` and run the existing factory unchanged → `IValue` whose location pins nothing. +- **One translator branch:** `JavaTranslator.getDetectionContextFrom` gets a leading `if (location instanceof DetachedSyntaxToken d) return new DetectionLocation(filePath, d.line(), d.offset(), d.keywords(), bundle);`. This preserves `keywords`/`additionalContext` fidelity exactly (they were computed from the live tree at record time). Python/Go never produce a `DetachedSyntaxToken`, so their translators are untouched. -This is mechanical but wide (~27 model + ~27 factory + 3 translator edits, plus `ResolvedValue` construction sites in the Java/Python/Go engines and tests that call `getLocation()`/`reportIssue`). It is exercised on the **main** cross-file path, not just edge cases. +Zero edits to the 27 model classes, 27 factories, and `ResolvedValue`. The change is one new class + one translator branch, plus the record/replay wiring. ### 5. Lossless cleanups folded in @@ -95,16 +94,15 @@ This is mechanical but wide (~27 model + ~27 factory + 3 translator edits, plus - `engine/.../callstack/CallStackAgent.java` — key-indexed lookup; drop `visitedTreeObjects`; record `DetachedCall` vs `RetainedCall`; match against keys. - `engine/.../callstack/CallContext.java` — becomes the sealed record family (`RetainedCall`/`DetachedCall`). -- `engine/.../detection/ResolvedValue.java` — carry `Location`. -- `engine/.../model/Location.java` (+ `TreeLocation`, `SnapshotLocation`) — NEW. -- `engine/.../model/*.java` (27) + `engine/.../model/factory/*.java` (27) — location field/parameter type. -- `engine/.../detection/DetectionStoreWithHook.java` — detached replay branch. -- `engine/.../language/java/JavaDetectionEngine.java` — record-time pre-resolution + `NEW_ARRAY` detection; `ResolvedValue` construction. -- `engine/.../language/{python,go}/*DetectionEngine.java` — `ResolvedValue`/`TreeLocation` construction (behavior-preserving). -- `engine/.../language/ILanguageSupport.java` (or `ILanguageTranslation.java`) — `isDetachableCall` predicate (default: not detachable). -- `{java,python,go}/.../translation/translator/*Translator.java` — handle `SnapshotLocation`. +- `engine/.../callstack/DetachedSyntaxToken.java` — NEW (synthetic `SyntaxToken`, §4). +- `engine/.../detection/DetectionStoreWithHook.java` — detached replay branch (build `ResolvedValue` from snapshot + `DetachedSyntaxToken`, skip `extractArgumentFromMethodCaller`/`resolveValuesInInnerScope`). +- `engine/.../language/java/JavaDetectionEngine.java` — record-time pre-resolution + `NEW_ARRAY` detection + capture location primitives. +- `engine/.../language/ILanguageSupport.java` (or `ILanguageTranslation.java`) — `isDetachableCall` predicate (default: not detachable → Python/Go keep trees). +- `java/.../translation/translator/JavaTranslator.java` — leading `DetachedSyntaxToken` branch in `getDetectionContextFrom` (Python/Go translators untouched). - `java/src/test/files/...` + a new multi-file `CheckVerifier` test — NEW (see Verification). +No edits to the 27 `engine/.../model/*.java`, the 27 `engine/.../model/factory/*.java`, `ResolvedValue.java`, or the Python/Go engines/translators. + ## Verification - **Cross-file regression guard (new, does not exist today):** a multi-file `CheckVerifier` test where a hook created in file B resolves a call recorded in file A. Must cover **both** a detached argument (literal / constant / field / nested-call) **and** a tree-fallback argument (`NEW_ARRAY`/size), asserting both still detect. @@ -113,11 +111,11 @@ This is mechanical but wide (~27 model + ~27 factory + 3 translator edits, plus ## Risks & open implementation questions -- **Location-model diff size** is the top risk — wide but mechanical. Mitigate by landing it as an isolated, behavior-preserving commit (`TreeLocation` everywhere) before any detach logic, so it can be reviewed/tested independently. -- **Record-time pre-resolution fidelity:** must produce exactly the raw `ResolvedValue`s fire-time would, minus the `SizeFactory` case. Guard with a test that resolves the same argument both ways and asserts equality. +- **`DetachedSyntaxToken` consumer safety (top risk):** the synthetic token returns `parent() == null` and a no-op `accept()`. This is safe only if every consumer of a value's `getLocation()` reads at most `firstToken()/lastToken()/kind()/range()`. Verified today: the sole *production* consumer is `getDetectionContextFrom`; production CBOM output does not `reportIssue` on value locations (that path is test-only). **Task:** audit all `IValue.getLocation()` consumers to confirm none navigate `parent()`/`accept()`/children, and make the new multi-file test's `asserts()` tolerant of detached locations. +- **Record-time pre-resolution fidelity:** must produce exactly the raw `ResolvedValue`s fire-time would, minus the `NEW_ARRAY`/`SizeFactory` case (kept on tree-fallback). Guard with a test that resolves the same argument both ways and asserts equality. - **Enum snapshot completeness:** pre-resolving all enum constants at record time must match `resolveEnumValue`'s selection semantics; verify against existing enum-hook tests. -- **`SnapshotLocation` fidelity:** the `keywords`/line/offset must match what `getDetectionContextFrom` produces from the live tree, so CBOM occurrences are identical for detached vs. non-detached detections. Verify via an output-level test. -- **`filePath` origin for cross-file values:** confirm the detached value's `SnapshotLocation.filePath` is file A's (the call site), matching today's tree-derived behavior. +- **Location fidelity:** `keywords`/line/offset captured at record time must match what `getDetectionContextFrom` derives from the live tree, so CBOM occurrences (incl. `additionalContext`) are identical for detached vs. non-detached detections. Verify via an output-level test. +- **`filePath` origin for cross-file values:** confirm the translator receives file A's (call-site) path for a detached value, matching today's tree-derived behavior — this is independent of the token but must be checked on the cross-file path. ## Explicitly out of scope From c3ca731cd99c6edefd9fe5d9ac9142082fd62720 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Sun, 5 Jul 2026 22:06:36 +0200 Subject: [PATCH 03/35] docs: task-by-task implementation plan for call-stack AST-detach MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- ...6-07-05-callstack-detach-implementation.md | 1211 +++++++++++++++++ 1 file changed, 1211 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-05-callstack-detach-implementation.md diff --git a/docs/superpowers/plans/2026-07-05-callstack-detach-implementation.md b/docs/superpowers/plans/2026-07-05-callstack-detach-implementation.md new file mode 100644 index 00000000..93f9a63c --- /dev/null +++ b/docs/superpowers/plans/2026-07-05-callstack-detach-implementation.md @@ -0,0 +1,1211 @@ +# Call-stack AST-Detach Heap Reduction — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Stop the per-language `CallStackAgent` from pinning whole-file ASTs across a scan by storing tree-free "detached" records for eligible Java method calls, cutting the measured ~7 GB / ~179k-`CallContext` runtime heap without regressing cross-file detection. + +**Architecture:** `CallContext` becomes a sealed interface with two variants: `RetainedCall` (today's `tree + scanContext`, unchanged behavior) and `DetachedCall` (match keys + per-argument pre-resolved value snapshots + an AST-free `DetachedScanContext`). At record time the Java engine pre-resolves each argument while the file is live and, if faithfully reproducible, stores a `DetachedCall`; otherwise it keeps a `RetainedCall`. At hook-fire time a detached record replays from its snapshot, producing detection values whose location is a synthetic AST-free `DetachedSyntaxToken`. Because a `DetachedCall` holds no `Tree` and no `JavaFileScannerContext`, the file's AST becomes GC-eligible after `leaveFile`. + +**Tech Stack:** Java 17, Maven multi-module, sonar-java 8.0.1 `CheckVerifier`, JUnit 5 + AssertJ. + +## Global Constraints + +- Java 17; changes live in `engine` (+ Java language support) and `java` (translator + tests). Do not modify Python/Go behavior. +- Apache 2.0 license header in every new `.java` file — copy the 19-line header verbatim from any neighbor (e.g. `engine/src/main/java/com/ibm/engine/callstack/CallContext.java:1-19`). +- Run `mvn spotless:apply` before every commit. If Spotless truncates `mapper/.../JsonCipherSuites.java`, restore it (`git checkout -- `) before committing. +- `mvn test -pl engine` and `mvn test -pl java` must stay green after every task. +- Keep the generic engine language-agnostic: the detachability decision and pre-resolution live in the Java layer; the generic `CallStackAgent` only stores/keys/replays `CallContext` variants. +- **Scope for this iteration:** only Java **method invocations** are detachable. Enum accesses (`Tree.Kind.ENUM`), Python, and Go always use `RetainedCall` (unchanged). + +--- + +## File Structure + +- `engine/.../callstack/CallContext.java` — MODIFY → `sealed interface CallContext permits RetainedCall, DetachedCall`. +- `engine/.../callstack/RetainedCall.java` — CREATE — record `(T tree, IScanContext scanContext)`; today's shape. +- `engine/.../callstack/DetachedCall.java` — CREATE — record with match keys, `List`, `DetachedScanContext`. +- `engine/.../callstack/ArgSnapshot.java` — CREATE — one argument's resolved raw values + location primitives. +- `engine/.../callstack/DetachedSyntaxToken.java` — CREATE — AST-free `SyntaxToken` for value locations. +- `engine/.../callstack/DetachedScanContext.java` — CREATE — AST-free `IScanContext` (InputFile + filePath). +- `engine/.../callstack/CallStackAgent.java` — MODIFY — variant-aware add/key/match; drop `visitedTreeObjects`; key-indexed lookup. +- `engine/.../detection/MethodMatcher.java` — MODIFY — add `matchKeys(...)` overload. +- `engine/.../detection/DetectionStoreWithHook.java` — MODIFY — detached replay branch. +- `engine/.../hooks/{HookRepository,IHook,EnumHook,MethodInvocationHookWithParameterResolvement,MethodInvocationHookWithReturnResolvement,HookDetectionObservable,IHookDetectionObserver}.java` — MODIFY — consume sealed `CallContext` / carry the record through fire. +- `engine/.../detection/DetectionStore.java` — MODIFY — `onHookInvocation` carries the `CallContext`. +- `engine/.../language/ILanguageSupport.java` — MODIFY — `isDetachableCall` default `false`. +- `engine/.../language/java/JavaLanguageSupport.java` — MODIFY — Java `isDetachableCall`. +- `engine/.../language/java/JavaDetectionEngine.java` — MODIFY — record-time pre-resolution + build variant; expose a location-primitive capture helper. +- `java/.../plugin/translation/translator/JavaTranslator.java` — MODIFY — leading `DetachedSyntaxToken` branch in `getDetectionContextFrom`. +- `java/src/test/files/rules/detection/crossfile/CrossFileDefinition.java` + `CrossFileUsage.java` — CREATE — multi-file fixtures. +- `java/src/test/java/com/ibm/plugin/rules/detection/crossfile/CrossFileHookDetachTest.java` — CREATE — cross-file regression guard. + +--- + +## Task 1: Characterization test — cross-file hook detection baseline + +Locks current cross-file behavior BEFORE any refactor. This is the regression guard; it must pass on unmodified code and after every later task. + +**Files:** +- Create: `java/src/test/files/rules/detection/crossfile/CrossFileDefinition.java` +- Create: `java/src/test/files/rules/detection/crossfile/CrossFileUsage.java` +- Create: `java/src/test/java/com/ibm/plugin/rules/detection/crossfile/CrossFileHookDetachTest.java` + +**Interfaces:** +- Consumes: `TestBase` (`java/src/test/java/com/ibm/plugin/TestBase.java`), `CheckVerifier` (sonar-java). +- Produces: nothing code-facing; establishes the green baseline all later tasks preserve. + +- [ ] **Step 1: Write the definition fixture** (a wrapper method whose algorithm arg is a plain `String` parameter — forces a parameter hook). NOT a test source; no license header needed (matches other `src/test/files`). + +`java/src/test/files/rules/detection/crossfile/CrossFileDefinition.java`: +```java +package rules.detection.crossfile; + +import javax.crypto.Cipher; +import java.security.GeneralSecurityException; + +public class CrossFileDefinition { + public Cipher make(String transformation) throws GeneralSecurityException { + return Cipher.getInstance(transformation); // hook: resolve 'transformation' from caller + } +} +``` + +- [ ] **Step 2: Write the usage fixture** (calls the wrapper with a literal — this is the call recorded in "file A"): + +`java/src/test/files/rules/detection/crossfile/CrossFileUsage.java`: +```java +package rules.detection.crossfile; + +import javax.crypto.Cipher; +import java.security.GeneralSecurityException; + +public class CrossFileUsage { + public Cipher use() throws GeneralSecurityException { + return new CrossFileDefinition().make("AES/GCM/NoPadding"); + } +} +``` + +- [ ] **Step 3: Write the multi-file test.** Uses `CheckVerifier.newVerifier().onFiles(usage, definition)` so the usage (call site) is analyzed before the definition (hook site), exercising the `onNewHookSubscription` retro-scan across files. Assert that the resolved algorithm value `"AES"` reaches a detection value. + +`java/src/test/java/com/ibm/plugin/rules/detection/crossfile/CrossFileHookDetachTest.java`: +```java +/* */ +package com.ibm.plugin.rules.detection.crossfile; + +import static org.assertj.core.api.Assertions.assertThat; + +import com.ibm.engine.detection.DetectionStore; +import com.ibm.engine.model.IValue; +import com.ibm.engine.model.ValueAction; +import com.ibm.mapper.model.INode; +import com.ibm.plugin.TestBase; +import java.util.List; +import javax.annotation.Nonnull; +import org.junit.jupiter.api.Test; +import org.sonar.java.checks.verifier.CheckVerifier; +import org.sonar.plugins.java.api.JavaCheck; +import org.sonar.plugins.java.api.JavaFileScannerContext; +import org.sonar.plugins.java.api.semantic.Symbol; +import org.sonar.plugins.java.api.tree.Tree; + +class CrossFileHookDetachTest extends TestBase { + + private static boolean sawAes = false; + + @Test + void crossFileLiteralResolves() { + sawAes = false; + CheckVerifier.newVerifier() + .onFiles( + "src/test/files/rules/detection/crossfile/CrossFileUsage.java", + "src/test/files/rules/detection/crossfile/CrossFileDefinition.java") + .withCheck(this) + .verifyNoIssues(); + assertThat(sawAes).as("cross-file resolved algorithm value 'AES'").isTrue(); + } + + @Override + public void asserts( + int findingId, + @Nonnull DetectionStore detectionStore, + @Nonnull List nodes) { + collectValues(detectionStore); + } + + private void collectValues( + @Nonnull DetectionStore store) { + for (IValue value : store.getDetectionValues()) { + if (value.asString().contains("AES")) { + sawAes = true; + } + } + store.getChildren().forEach(this::collectValues); + } +} +``` + +- [ ] **Step 4: Run the test — verify it PASSES on unmodified code.** + +Run: `mvn test -pl java -Dtest=CrossFileHookDetachTest` +Expected: PASS, `sawAes` true. (If it fails, the fixtures don't trigger a parameter hook — adjust the wrapper so the algorithm is a method parameter, not a literal inside the wrapper, and re-run. Do not proceed until green.) + +- [ ] **Step 5: Commit.** +```bash +mvn spotless:apply +git add java/src/test/files/rules/detection/crossfile java/src/test/java/com/ibm/plugin/rules/detection/crossfile +git commit -m "test: cross-file hook detection characterization baseline" +``` + +--- + +## Task 2: `DetachedSyntaxToken` — AST-free value location + +**Files:** +- Create: `engine/src/main/java/com/ibm/engine/callstack/DetachedSyntaxToken.java` +- Test: `engine/src/test/java/com/ibm/engine/callstack/DetachedSyntaxTokenTest.java` + +**Interfaces:** +- Produces: `DetachedSyntaxToken(int line, int columnOffset, int endLine, int endColumnOffset, String text, List keywords) implements SyntaxToken`, plus getter `List keywords()`. `parent()` returns `null`; `kind()` returns `Tree.Kind.TOKEN`; `firstToken()`/`lastToken()` return `this`; `range()` returns `Range.at(Position.at(line, columnOffset), Position.at(endLine, endColumnOffset))`. + +- [ ] **Step 1: Write the failing test.** +```java +/* */ +package com.ibm.engine.callstack; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.util.List; +import org.junit.jupiter.api.Test; +import org.sonar.plugins.java.api.tree.Tree; + +class DetachedSyntaxTokenTest { + @Test + void pinsNothingAndReportsRange() { + DetachedSyntaxToken t = + new DetachedSyntaxToken(12, 4, 12, 20, "AES", List.of("javax.crypto.Cipher#getInstance")); + assertThat(t.parent()).isNull(); + assertThat(t.kind()).isEqualTo(Tree.Kind.TOKEN); + assertThat(t.firstToken()).isSameAs(t); + assertThat(t.range().start().line()).isEqualTo(12); + assertThat(t.range().start().columnOffset()).isEqualTo(4); + assertThat(t.keywords()).containsExactly("javax.crypto.Cipher#getInstance"); + assertThat(t.text()).isEqualTo("AES"); + } +} +``` + +- [ ] **Step 2: Run — verify it fails to compile (class missing).** +Run: `mvn test -pl engine -Dtest=DetachedSyntaxTokenTest` +Expected: FAIL — `cannot find symbol DetachedSyntaxToken`. + +- [ ] **Step 3: Implement the class.** +```java +/* */ +package com.ibm.engine.callstack; + +import java.util.List; +import javax.annotation.Nonnull; +import javax.annotation.Nullable; +import org.sonar.plugins.java.api.location.Position; +import org.sonar.plugins.java.api.location.Range; +import org.sonar.plugins.java.api.tree.SyntaxToken; +import org.sonar.plugins.java.api.tree.SyntaxTrivia; +import org.sonar.plugins.java.api.tree.Tree; +import org.sonar.plugins.java.api.tree.TreeVisitor; + +/** AST-free {@link SyntaxToken} used as the location of a detached cross-file detection value. + * Holds only primitives, so it never pins a compilation unit ({@link #parent()} is null). */ +public final class DetachedSyntaxToken implements SyntaxToken { + private final int line; + private final int columnOffset; + private final int endLine; + private final int endColumnOffset; + @Nonnull private final String text; + @Nonnull private final List keywords; + + public DetachedSyntaxToken(int line, int columnOffset, int endLine, int endColumnOffset, + @Nonnull String text, @Nonnull List keywords) { + this.line = line; + this.columnOffset = columnOffset; + this.endLine = endLine; + this.endColumnOffset = endColumnOffset; + this.text = text; + this.keywords = List.copyOf(keywords); + } + + @Nonnull public List keywords() { return keywords; } + public int offset() { return columnOffset; } + + @Override public String text() { return text; } + @Override public List trivias() { return List.of(); } + @Override public int line() { return line; } + @Override public int column() { return columnOffset; } + @Override public Range range() { + return Range.at(Position.at(line, columnOffset + 1), Position.at(endLine, endColumnOffset + 1)); + } + @Override public boolean is(Tree.Kind... kinds) { + for (Tree.Kind k : kinds) { if (k == Tree.Kind.TOKEN) return true; } + return false; + } + @Override public void accept(TreeVisitor visitor) { /* detached: no traversal */ } + @Nullable @Override public Tree parent() { return null; } + @Override public SyntaxToken firstToken() { return this; } + @Override public SyntaxToken lastToken() { return this; } + @Override public Tree.Kind kind() { return Tree.Kind.TOKEN; } + + @Override public boolean equals(Object o) { + if (this == o) return true; + if (!(o instanceof DetachedSyntaxToken t)) return false; + return line == t.line && columnOffset == t.columnOffset && endLine == t.endLine + && endColumnOffset == t.endColumnOffset && text.equals(t.text) && keywords.equals(t.keywords); + } + @Override public int hashCode() { + int r = line; + r = 31 * r + columnOffset; + r = 31 * r + endLine; + r = 31 * r + endColumnOffset; + r = 31 * r + text.hashCode(); + r = 31 * r + keywords.hashCode(); + return r; + } +} +``` +Note: `Position.at` is 1-based on column in sonar-java; capture `columnOffset` 0-based and add 1 here so `range().start().columnOffset()` returns the original 0-based value. Verify against Step-1 assertion; adjust the `+1`/`-1` if the assertion fails. + +- [ ] **Step 4: Run — verify PASS** (fix the column offset convention if the range assertion fails). +Run: `mvn test -pl engine -Dtest=DetachedSyntaxTokenTest` +Expected: PASS. + +- [ ] **Step 5: Commit.** +```bash +mvn spotless:apply +git add engine/src/main/java/com/ibm/engine/callstack/DetachedSyntaxToken.java engine/src/test/java/com/ibm/engine/callstack/DetachedSyntaxTokenTest.java +git commit -m "feat(engine): AST-free DetachedSyntaxToken for detached value locations" +``` + +--- + +## Task 3: `DetachedScanContext` — AST-free scan context + +**Files:** +- Create: `engine/src/main/java/com/ibm/engine/callstack/DetachedScanContext.java` +- Test: `engine/src/test/java/com/ibm/engine/callstack/DetachedScanContextTest.java` + +**Interfaces:** +- Consumes: `IScanContext` (`engine/.../language/IScanContext.java`). +- Produces: `DetachedScanContext(InputFile inputFile, String filePath)` implementing `IScanContext`. `getFilePath()`/`getInputFile()` return the captured values; `reportIssue(...)` throws `UnsupportedOperationException` (never called on the production CBOM path — see spec risk audit in Task 12). + +- [ ] **Step 1: Write the failing test.** +```java +/* */ +package com.ibm.engine.callstack; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.Mockito.mock; + +import org.junit.jupiter.api.Test; +import org.sonar.api.batch.fs.InputFile; + +class DetachedScanContextTest { + @Test + void exposesFilePathWithoutPinningAst() { + InputFile inputFile = mock(InputFile.class); + DetachedScanContext ctx = + new DetachedScanContext<>(inputFile, "/abs/path/CrossFileUsage.java"); + assertThat(ctx.getFilePath()).isEqualTo("/abs/path/CrossFileUsage.java"); + assertThat(ctx.getInputFile()).isSameAs(inputFile); + } +} +``` + +- [ ] **Step 2: Run — verify it fails to compile.** +Run: `mvn test -pl engine -Dtest=DetachedScanContextTest` +Expected: FAIL — class missing. + +- [ ] **Step 3: Implement.** +```java +/* */ +package com.ibm.engine.callstack; + +import com.ibm.engine.language.IScanContext; +import javax.annotation.Nonnull; +import org.sonar.api.batch.fs.InputFile; + +/** AST-free {@link IScanContext}: retains only the file handle + path captured at record time, + * never the {@code JavaFileScannerContext}, so a detached record does not pin the file's AST. */ +public record DetachedScanContext(@Nonnull InputFile inputFile, @Nonnull String filePath) + implements IScanContext { + @Override public void reportIssue(@Nonnull R currentRule, @Nonnull T tree, @Nonnull String message) { + throw new UnsupportedOperationException("reportIssue is not available on a detached scan context"); + } + @Nonnull @Override public InputFile getInputFile() { return inputFile; } + @Nonnull @Override public String getFilePath() { return filePath; } +} +``` + +- [ ] **Step 4: Run — verify PASS.** +Run: `mvn test -pl engine -Dtest=DetachedScanContextTest` +Expected: PASS. + +- [ ] **Step 5: Commit.** +```bash +mvn spotless:apply +git add engine/src/main/java/com/ibm/engine/callstack/DetachedScanContext.java engine/src/test/java/com/ibm/engine/callstack/DetachedScanContextTest.java +git commit -m "feat(engine): AST-free DetachedScanContext" +``` + +--- + +## Task 4: `ArgSnapshot` + `MethodMatcher.matchKeys` overload + +**Files:** +- Create: `engine/src/main/java/com/ibm/engine/callstack/ArgSnapshot.java` +- Modify: `engine/src/main/java/com/ibm/engine/detection/MethodMatcher.java` +- Test: `engine/src/test/java/com/ibm/engine/detection/MethodMatcherKeysTest.java` + +**Interfaces:** +- Produces: `ArgSnapshot(int index, List values)` and `ResolvedSnapshotValue(Object value, DetachedSyntaxToken location)`. +- Produces: `MethodMatcher.matchKeys(@Nonnull IType invokedObjectType, @Nonnull String methodName, @Nonnull List parameterTypes)` returning `boolean`, applying the same predicates as `match(...)` (`MethodMatcher.java:159`) without a tree. + +- [ ] **Step 1: Write `ArgSnapshot` and `ResolvedSnapshotValue`** (plain records; no test needed — exercised via later tasks). +```java +/* */ +package com.ibm.engine.callstack; + +import java.util.List; +import javax.annotation.Nonnull; + +public record ArgSnapshot(int index, @Nonnull List values) { + public record ResolvedSnapshotValue(@Nonnull Object value, @Nonnull DetachedSyntaxToken location) {} +} +``` + +- [ ] **Step 2: Write the failing `matchKeys` test.** Read `MethodMatcher.java:31-160` first to reuse the exact predicate fields (`invokedObjectTypeString`, `methodName`, `parameterTypes`). +```java +/* */ +package com.ibm.engine.detection; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.util.LinkedList; +import java.util.List; +import org.junit.jupiter.api.Test; + +class MethodMatcherKeysTest { + @Test + void matchesByKeysSameAsTree() { + MethodMatcher matcher = + new MethodMatcher<>("javax.crypto.Cipher", "getInstance", + new LinkedList<>(List.of("java.lang.String"))); + IType type = new SimpleTypeForTest("javax.crypto.Cipher"); + assertThat(matcher.matchKeys(type, "getInstance", List.of(new SimpleTypeForTest("java.lang.String")))) + .isTrue(); + assertThat(matcher.matchKeys(type, "doFinal", List.of(new SimpleTypeForTest("java.lang.String")))) + .isFalse(); + } +} +``` +Add a tiny `SimpleTypeForTest implements IType` in the same test package if no test `IType` fake exists (check `engine/src/test/java/com/ibm/engine/detection/` first for an existing one and reuse it). + +- [ ] **Step 3: Run — verify it fails.** +Run: `mvn test -pl engine -Dtest=MethodMatcherKeysTest` +Expected: FAIL — `matchKeys` missing. + +- [ ] **Step 4: Implement `matchKeys`** by extracting the predicate tail of `match(...)`. In `MethodMatcher.java`, refactor `match(...)` (`:159`) to delegate: +```java +public boolean match(@Nonnull T expression, @Nonnull ILanguageTranslation translation, + @Nonnull MatchContext matchContext) { + Optional invokedObjectType = translation.getInvokedObjectTypeString(matchContext, expression); + Optional invokedMethodName = translation.getMethodName(matchContext, expression); + List param = translation.getMethodParameterTypes(matchContext, expression); + if (invokedObjectType.isEmpty() || invokedMethodName.isEmpty()) { + return false; + } + return matchKeysInternal(invokedObjectType.get(), invokedMethodName.get(), param, + translation.supportsSubsetParameterMatching()); +} + +public boolean matchKeys(@Nonnull IType invokedObjectType, @Nonnull String methodName, + @Nonnull List parameterTypes) { + return matchKeysInternal(invokedObjectType, methodName, parameterTypes, false); +} + +private boolean matchKeysInternal(@Nonnull IType invokedObjectType, @Nonnull String methodName, + @Nonnull List param, boolean subsetParameterMatching) { + boolean typeMatches = this.invokedObjectTypeString.test(invokedObjectType); + boolean nameMatches = this.methodName.test(methodName); + if (!typeMatches || !nameMatches) { + return false; + } + if (methodName.equals("") && !parameterTypesSerializable.isEmpty() && subsetParameterMatching) { + return anyParameterMatches(param); + } + return this.parameterTypes.test(param); +} +``` +(Detached calls are never constructors with subset matching, so `matchKeys` passes `false`.) + +- [ ] **Step 5: Run — verify PASS and no regressions.** +Run: `mvn test -pl engine -Dtest=MethodMatcherKeysTest,MethodMatcherTest` +Expected: PASS (run the existing `MethodMatcher*` tests too; if none exist, run `mvn test -pl engine`). + +- [ ] **Step 6: Commit.** +```bash +mvn spotless:apply +git add engine/src/main/java/com/ibm/engine/callstack/ArgSnapshot.java engine/src/main/java/com/ibm/engine/detection/MethodMatcher.java engine/src/test/java/com/ibm/engine/detection/MethodMatcherKeysTest.java +git commit -m "feat(engine): ArgSnapshot + MethodMatcher.matchKeys (tree-free match)" +``` + +--- + +## Task 5: `isDetachableCall` predicate in the language layer + +**Files:** +- Modify: `engine/src/main/java/com/ibm/engine/language/ILanguageSupport.java` +- Modify: `engine/src/main/java/com/ibm/engine/language/java/JavaLanguageSupport.java` +- Test: `engine/src/test/java/com/ibm/engine/language/java/JavaIsDetachableCallTest.java` + +**Interfaces:** +- Produces: `default boolean isDetachableCall(@Nonnull T tree) { return false; }` on `ILanguageSupport`. +- Produces: Java override — `true` iff `tree` is a `MethodInvocationTree` whose `methodSymbol().declaration() != null` (user method → could be hooked) AND no argument expression subtree contains a `NEW_ARRAY` node (the only factory-steered resolution case, `JavaDetectionEngine.java:275`); else `false`. + +- [ ] **Step 1: Add the default to `ILanguageSupport`.** Find the interface method list and add: +```java +/** Whether a recorded call may be stored tree-free (detached) instead of retaining its AST. + * Default: conservative false (retain the tree). Only Java method invocations override. */ +default boolean isDetachableCall(@Nonnull T tree) { + return false; +} +``` + +- [ ] **Step 2: Write the failing Java test.** +```java +/* */ +package com.ibm.engine.language.java; + +import static org.assertj.core.api.Assertions.assertThat; + +import org.junit.jupiter.api.Test; +import org.sonar.java.model.JParserTestUtils; +import org.sonar.plugins.java.api.tree.ClassTree; +import org.sonar.plugins.java.api.tree.CompilationUnitTree; +import org.sonar.plugins.java.api.tree.ExpressionStatementTree; +import org.sonar.plugins.java.api.tree.MethodTree; +import org.sonar.plugins.java.api.tree.Tree; + +class JavaIsDetachableCallTest { + private Tree firstStatementExpression(String body) { + CompilationUnitTree cut = JParserTestUtils.parse( + "class A { void m(String s){ " + body + " } byte[] b(){return null;} }"); + MethodTree m = (MethodTree) ((ClassTree) cut.types().get(0)).members().get(0); + ExpressionStatementTree stmt = (ExpressionStatementTree) m.block().body().get(0); + return stmt.expression(); + } + + @Test + void literalArgIsDetachable() { + JavaLanguageSupport support = new JavaLanguageSupport(); + // user method call: A.b() has a source declaration + Tree call = firstStatementExpression("b();"); + assertThat(support.isDetachableCall(call)).isTrue(); + } + + @Test + void arrayArgIsNotDetachable() { + JavaLanguageSupport support = new JavaLanguageSupport(); + Tree call = firstStatementExpression("java.util.Arrays.toString(new byte[]{1,2});"); + assertThat(support.isDetachableCall(call)).isFalse(); + } +} +``` +(If `JParserTestUtils` is not on the test classpath, mirror the parsing helper used by an existing `engine` Java test — check `engine/src/test/java/com/ibm/engine/language/java/` for the established parse utility and reuse it.) + +- [ ] **Step 3: Run — verify it fails.** +Run: `mvn test -pl engine -Dtest=JavaIsDetachableCallTest` +Expected: FAIL — `isDetachableCall` not overridden / returns false for the literal case. + +- [ ] **Step 4: Implement in `JavaLanguageSupport`.** +```java +@Override +public boolean isDetachableCall(@Nonnull Tree tree) { + if (!(tree instanceof MethodInvocationTree invocation)) { + return false; + } + if (invocation.methodSymbol().declaration() == null) { + return false; // library method: cannot match any source-declared method hook anyway + } + for (ExpressionTree arg : invocation.arguments()) { + if (containsNewArray(arg)) { + return false; // SizeFactory-steered resolution — keep the tree (fallback) + } + } + return true; +} + +private static boolean containsNewArray(@Nonnull Tree tree) { + if (tree.is(Tree.Kind.NEW_ARRAY)) { + return true; + } + boolean[] found = {false}; + tree.accept(new org.sonar.plugins.java.api.tree.BaseTreeVisitor() { + @Override public void visitNewArray(org.sonar.plugins.java.api.tree.NewArrayTree t) { + found[0] = true; + } + }); + return found[0]; +} +``` +Add imports: `MethodInvocationTree`, `ExpressionTree`, `Tree`. + +- [ ] **Step 5: Run — verify PASS.** +Run: `mvn test -pl engine -Dtest=JavaIsDetachableCallTest` +Expected: PASS. + +- [ ] **Step 6: Commit.** +```bash +mvn spotless:apply +git add engine/src/main/java/com/ibm/engine/language/ILanguageSupport.java engine/src/main/java/com/ibm/engine/language/java/JavaLanguageSupport.java engine/src/test/java/com/ibm/engine/language/java/JavaIsDetachableCallTest.java +git commit -m "feat(engine): isDetachableCall predicate (Java method invocations, no NEW_ARRAY)" +``` + +--- + +## Task 6: Convert `CallContext` to a sealed interface with `RetainedCall` (pure refactor, no behavior change) + +Make `CallContext` sealed with a single variant `RetainedCall`, updating every consumer of `.tree()`/`.publisher()`. Behavior is identical; all existing tests (incl. Task 1) stay green. This isolates the compile-breaking type change from any logic change. + +**Files:** +- Modify: `engine/src/main/java/com/ibm/engine/callstack/CallContext.java` +- Create: `engine/src/main/java/com/ibm/engine/callstack/RetainedCall.java` +- Modify: `CallStackAgent.java`, `HookRepository.java`, `IHook.java`, `EnumHook.java`, `MethodInvocationHookWithParameterResolvement.java`, `MethodInvocationHookWithReturnResolvement.java` + +**Interfaces:** +- Produces: `sealed interface CallContext permits RetainedCall, DetachedCall` with `@Nonnull IScanContext publisher()` and `@Nullable T tree()` (default-null; `RetainedCall` overrides with its tree). `DetachedCall` is added in Task 7 — declare it in `permits` now and create a temporary empty stub, or add `permits RetainedCall` here and widen in Task 7. +- Produces: `RetainedCall(T tree, IScanContext publisher) implements CallContext` — `tree()` returns the tree. + +- [ ] **Step 1: Turn `CallContext` into the sealed interface** (start with a single permit; Task 7 widens it): +```java +/* */ +package com.ibm.engine.callstack; + +import com.ibm.engine.language.IScanContext; +import javax.annotation.Nonnull; +import javax.annotation.Nullable; + +public sealed interface CallContext permits RetainedCall { + @Nullable T tree(); + @Nonnull IScanContext publisher(); +} +``` + +- [ ] **Step 2: Create `RetainedCall`.** +```java +/* */ +package com.ibm.engine.callstack; + +import com.ibm.engine.language.IScanContext; +import javax.annotation.Nonnull; + +public record RetainedCall(@Nonnull T tree, @Nonnull IScanContext publisher) + implements CallContext {} +``` + +- [ ] **Step 3: Update `CallStackAgent.addCall`** to build a `RetainedCall` and update the `visitedTreeObjects` guard (unchanged logic, new type). At `CallStackAgent.java:60`: +```java +final CallContext callContext = new RetainedCall<>(tree, scanContext); +``` +Everywhere `callContext.tree()` is used inside `CallStackAgent`, it remains valid (RetainedCall returns non-null). + +- [ ] **Step 4: Update the remaining consumers** to construct/expect the interface. The only constructions are in `CallStackAgent`. `HookRepository.update` (`:115-121`) uses `callContext.tree()` and `callContext.publisher()` — both still compile (interface methods). `IHook.isInvocationOn(CallContext,...)` and the three hook impls call `callContext.tree()` — still compile. No signature changes needed; only the `new CallContext<>(...)` construction (now `new RetainedCall<>(...)`) changes. Grep to confirm no other `new CallContext<>` exists: +Run: `grep -rn "new CallContext" engine/src/main/java` +Expected: no results after the edit. + +- [ ] **Step 5: Build + full engine/java tests — verify green (no behavior change).** +Run: `mvn test -pl engine && mvn test -pl java -Dtest=CrossFileHookDetachTest` +Expected: PASS (all engine tests + the Task 1 baseline). + +- [ ] **Step 6: Commit.** +```bash +mvn spotless:apply +git add engine/src/main/java/com/ibm/engine/callstack/CallContext.java engine/src/main/java/com/ibm/engine/callstack/RetainedCall.java engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java +git commit -m "refactor(engine): CallContext -> sealed interface with RetainedCall variant" +``` + +--- + +## Task 7: Add the `DetachedCall` variant + +**Files:** +- Create: `engine/src/main/java/com/ibm/engine/callstack/DetachedCall.java` +- Modify: `engine/src/main/java/com/ibm/engine/callstack/CallContext.java` (widen `permits`) +- Test: `engine/src/test/java/com/ibm/engine/callstack/DetachedCallTest.java` + +**Interfaces:** +- Produces: `DetachedCall(IType invokedObjectType, String methodName, List parameterTypes, List arguments, DetachedScanContext publisher) implements CallContext`. `tree()` returns `null`. Getter `arguments()`, `invokedObjectType()`, `methodName()`, `parameterTypes()`. + +- [ ] **Step 1: Widen the sealed permit** in `CallContext.java`: `permits RetainedCall, DetachedCall`. + +- [ ] **Step 2: Write the failing test.** +```java +/* */ +package com.ibm.engine.callstack; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.Mockito.mock; + +import com.ibm.engine.detection.IType; +import java.util.List; +import org.junit.jupiter.api.Test; +import org.sonar.api.batch.fs.InputFile; + +class DetachedCallTest { + @Test + void holdsNoTree() { + DetachedScanContext ctx = + new DetachedScanContext<>(mock(InputFile.class), "/p/CrossFileUsage.java"); + DetachedCall call = + new DetachedCall<>(mock(IType.class), "make", List.of(), List.of(), ctx); + assertThat(call.tree()).isNull(); + assertThat(call.publisher()).isSameAs(ctx); + assertThat(call.methodName()).isEqualTo("make"); + } +} +``` + +- [ ] **Step 3: Run — verify it fails to compile.** +Run: `mvn test -pl engine -Dtest=DetachedCallTest` +Expected: FAIL — class missing. + +- [ ] **Step 4: Implement `DetachedCall`.** +```java +/* */ +package com.ibm.engine.callstack; + +import com.ibm.engine.detection.IType; +import java.util.List; +import javax.annotation.Nonnull; +import javax.annotation.Nullable; + +public record DetachedCall( + @Nonnull IType invokedObjectType, + @Nonnull String methodName, + @Nonnull List parameterTypes, + @Nonnull List arguments, + @Nonnull DetachedScanContext detachedPublisher) + implements CallContext { + @Nullable @Override public T tree() { return null; } + @Nonnull @Override public com.ibm.engine.language.IScanContext publisher() { + return detachedPublisher; + } +} +``` + +- [ ] **Step 5: Run — verify PASS and engine stays green.** +Run: `mvn test -pl engine -Dtest=DetachedCallTest && mvn test -pl engine` +Expected: PASS. + +- [ ] **Step 6: Commit.** +```bash +mvn spotless:apply +git add engine/src/main/java/com/ibm/engine/callstack/DetachedCall.java engine/src/main/java/com/ibm/engine/callstack/CallContext.java engine/src/test/java/com/ibm/engine/callstack/DetachedCallTest.java +git commit -m "feat(engine): DetachedCall variant (tree-free recorded call)" +``` + +--- + +## Task 8: Record-time production of `DetachedCall` in the Java engine + +Decide detach at record time, pre-resolve arguments while the file is live, and store a `DetachedCall`; else a `RetainedCall`. Keeps matching/replay unchanged for now (Task 9 wires replay), so detached calls are recorded but a hook firing on one would still take the retained path — to keep tests green this task also makes `CallStackAgent` skip detached calls during match until Task 9. To avoid a green-but-lossy interim, **Tasks 8 and 9 are committed together** (see Task 9 Step 6); do Task 8 without an intermediate "all green" claim beyond compilation. + +**Files:** +- Modify: `engine/src/main/java/com/ibm/engine/language/java/JavaDetectionEngine.java` +- Modify: `engine/src/main/java/com/ibm/engine/detection/Handler.java` (add `addRecordedCall`) +- Modify: `engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java` (accept a prebuilt `CallContext`) + +**Interfaces:** +- Produces: `Handler.addRecordedCall(@Nonnull CallContext recordedCall)` → delegates to `callStackAgent.add(recordedCall)`. +- Produces: `CallStackAgent.add(@Nonnull CallContext callContext)` — keys off `methodName` for `DetachedCall`, off `getKeyFormT(tree)` for `RetainedCall`; same dedup/notify contract. +- Produces (Java engine): a `DetachedCall` whose `arguments` are built by pre-resolving each invocation argument with `resolveValuesInInnerScope(Object.class, arg, null)` and capturing each `ResolvedValue`'s value + a `DetachedSyntaxToken` computed via a new `captureLocation(Tree)` helper mirroring `JavaTranslator.getDetectionContextFrom`. + +- [ ] **Step 1: Add `CallStackAgent.add(CallContext)`** alongside the existing `addCall(tree, scanContext)`. Extract a key helper that works for both variants: +```java +public void add(@Nonnull CallContext callContext) { + final Optional keyOptional = keyOf(callContext); + if (keyOptional.isEmpty()) { + return; + } + if (addedToCallContext(keyOptional.get(), callContext)) { + this.notify(callContext); + } +} + +private Optional keyOf(@Nonnull CallContext callContext) { + if (callContext instanceof DetachedCall detached) { + return Optional.of(detached.methodName().hashCode()); + } + final T tree = callContext.tree(); + return tree == null ? Optional.empty() : getKeyFormT(tree); +} +``` +Change the dedup in `addedToCallContext` to guard on the tree only when present (detached calls dedup by identity of the record — always add): +```java +private boolean addedToCallContext(int key, @Nonnull CallContext callContext) { + final T tree = callContext.tree(); + if (tree != null) { + if (visitedTreeObjects.contains(tree)) { + return false; + } + visitedTreeObjects.add(tree); + } + invokedCallStack.compute( + key, + (k, v) -> { + if (v == null) { + final ArrayList> list = new ArrayList<>(); + list.add(callContext); + return list; + } + v.add(callContext); + return v; + }); + return true; +} +``` +Keep `addCall(tree, scanContext)` delegating: `add(new RetainedCall<>(tree, scanContext));`. (Task 12 later removes `visitedTreeObjects`; keep it here so this task is behavior-preserving.) + +- [ ] **Step 2: Add `Handler.addRecordedCall`** next to `addCallToCallStack` (`Handler.java:51`): +```java +public void addRecordedCall(@Nonnull CallContext recordedCall) { + this.callStackAgent.add(recordedCall); +} +``` + +- [ ] **Step 3: Add the `captureLocation` helper to `JavaDetectionEngine`** (mirrors `JavaTranslator.getDetectionContextFrom`, `JavaTranslator.java:170-224`, but returns a `DetachedSyntaxToken`). Read that method first and copy its token/keyword logic exactly: +```java +@Nullable +private DetachedSyntaxToken captureLocation(@Nonnull Tree location, @Nonnull String text) { + SyntaxToken firstToken = location.firstToken(); + SyntaxToken lastToken = location.lastToken(); + if (firstToken == null || lastToken == null) { + return null; + } + SyntaxToken locationToken = firstToken; + List keywords = List.of(); + // Keyword/additionalContext fidelity: for the common detached case the location is a literal + // (default branch -> empty keywords), so implementing only `default` is correct for literals. + // To match today's output when a resolved value's location is a NEW_CLASS / METHOD_INVOCATION / + // ENUM_CONSTANT, replicate the three case bodies from JavaTranslator.getDetectionContextFrom + // (JavaTranslator.java:179-216) verbatim here — they only read the live tree and produce the + // `keywords` list + a more specific `locationToken`. Those Java-tree casts are available in the + // engine module. Example for METHOD_INVOCATION: + // MethodInvocationTree mit = (MethodInvocationTree) location; + // SyntaxToken sel = mit.methodSelect().firstToken(); + // if (sel != null) locationToken = sel; + // keywords = List.of(mit.methodSymbol().signature(), mit.methodSymbol().name()); + Position start = locationToken.range().start(); + Position end = lastToken.range().end(); + return new DetachedSyntaxToken(start.line(), start.columnOffset(), end.line(), end.columnOffset(), + text, keywords); +} +``` +Note the column convention: `Position.columnOffset()` is 0-based, matching `DetachedSyntaxToken`'s stored `columnOffset`. `DetectionLocation`'s `offset` is fed from `columnOffset()` today (`JavaTranslator.java:223`), so capture and store `columnOffset()` (0-based) consistently end to end. + +- [ ] **Step 4: Build the variant in `JavaDetectionEngine.run`** at the method-invocation site (`JavaDetectionEngine.java:98-100`). Replace the unconditional `handler.addCallToCallStack(...)` with: +```java +MethodInvocationTree methodInvocationTree = (MethodInvocationTree) tree; +final IScanContext scanContext = detectionStore.getScanContext(); +if (handler.getLanguageSupport().isDetachableCall(methodInvocationTree)) { + final DetachedCall detached = buildDetachedCall(methodInvocationTree, scanContext); + if (detached != null) { + handler.addRecordedCall(detached); + } else { + handler.addCallToCallStack(methodInvocationTree, scanContext); // pre-resolution failed -> keep tree + } +} else { + handler.addCallToCallStack(methodInvocationTree, scanContext); +} +``` +Leave the enum site (`:113-115`) calling `addCallToCallStack` unchanged (enums are never detached this iteration). + +- [ ] **Step 5: Implement `buildDetachedCall`.** Uses the translation for match keys and `resolveValuesInInnerScope` for arguments; returns `null` if any argument fails to resolve or a location can't be captured (→ caller keeps the tree): +```java +@Nullable +private DetachedCall buildDetachedCall( + @Nonnull MethodInvocationTree invocation, @Nonnull IScanContext scanContext) { + final MatchContext matchContext = MatchContext.build(false, detectionStore.getDetectionRule()); + final ILanguageTranslation translation = handler.getLanguageSupport().translation(); + final Optional invokedType = translation.getInvokedObjectTypeString(matchContext, invocation); + final Optional name = translation.getMethodName(matchContext, invocation); + if (invokedType.isEmpty() || name.isEmpty()) { + return null; + } + final List paramTypes = translation.getMethodParameterTypes(matchContext, invocation); + + final List args = new ArrayList<>(); + final List arguments = invocation.arguments(); + for (int i = 0; i < arguments.size(); i++) { + final List> resolved = + resolveValuesInInnerScope(Object.class, arguments.get(i), null); + final List snapshots = new ArrayList<>(); + for (ResolvedValue rv : resolved) { + final DetachedSyntaxToken loc = captureLocation(rv.tree(), rv.value().toString()); + if (loc == null) { + return null; // cannot faithfully snapshot -> fall back to retained tree + } + snapshots.add(new ArgSnapshot.ResolvedSnapshotValue(rv.value(), loc)); + } + args.add(new ArgSnapshot(i, snapshots)); + } + + final DetachedScanContext detachedCtx = + new DetachedScanContext<>(scanContext.getInputFile(), scanContext.getFilePath()); + return new DetachedCall<>(invokedType.get(), name.get(), paramTypes, args, detachedCtx); +} +``` +Add imports: `ArgSnapshot`, `DetachedCall`, `DetachedScanContext`, `DetachedSyntaxToken`, `ILanguageTranslation`, `IType`, `MatchContext`, `Position`, `SyntaxToken`, `ArrayList`, `List`, `Optional`. + +- [ ] **Step 6: Compile only (replay lands in Task 9).** +Run: `mvn -q -pl engine test-compile` +Expected: compiles. (Do not run full tests here; matching still ignores detached records — Task 9 wires replay. Proceed directly to Task 9; commit is shared.) + +--- + +## Task 9: Fire-time replay for `DetachedCall` + +Make matching and hook-fire handle detached records so a `DetachedCall` produces the same detection value a `RetainedCall` would, then verify the Task 1 baseline passes with detachment live. + +**Files:** +- Modify: `engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java` (`onNewHookSubscription` match) +- Modify: `engine/src/main/java/com/ibm/engine/hooks/HookRepository.java` (`update`) +- Modify: `engine/.../hooks/MethodInvocationHookWithParameterResolvement.java`, `EnumHook.java`, `MethodInvocationHookWithReturnResolvement.java`, `IHook.java` (`isInvocationOn(CallContext,...)`) +- Modify: `engine/.../hooks/HookDetectionObservable.java`, `IHookDetectionObserver.java`, `DetectionStore.java`, `DetectionStoreWithHook.java` (carry the `CallContext` through fire instead of a raw `T` tree) + +**Interfaces:** +- Produces: `IHook.isInvocationOn(CallContext,...)` matches `DetachedCall` via `MethodMatcher.matchKeys(invokedObjectType, methodName, parameterTypes)` and `RetainedCall` via the existing tree `match(...)`. +- Produces: the fire path carries `CallContext` (not `T invocationTree`) from `HookRepository.update` → `HookDetectionObservable.notify` → `IHookDetectionObserver.onHookInvocation` → `DetectionStoreWithHook`. Retained records behave exactly as before (`callContext.tree()`); detached records take the snapshot branch. + +- [ ] **Step 1: Match keys in the hooks.** In `MethodInvocationHookWithParameterResolvement.isInvocationOn(CallContext,...)` (`:59-63`) and the return-resolvement equivalent, branch on the variant: +```java +@Override +public boolean isInvocationOn(@Nonnull CallContext callContext, + @Nonnull ILanguageSupport languageSupport) { + if (callContext instanceof DetachedCall detached) { + MethodMatcher matcher = languageSupport.createMethodMatcherBasedOn(methodDefinition); + return matcher != null + && matcher.matchKeys(detached.invokedObjectType(), detached.methodName(), + detached.parameterTypes()); + } + return isInvocationOn(callContext.tree(), languageSupport); +} +``` +For `EnumHook.isInvocationOn(CallContext,...)`: detached calls are never enums, so `if (callContext instanceof DetachedCall) return false;` then fall through to the tree path. + +- [ ] **Step 2: Match in `CallStackAgent.onNewHookSubscription`** (`:100-112`). Replace `methodMatcher.match(callContext.tree(), translation, matchContext)` with a variant-aware check: +```java +final boolean matches; +if (callContext instanceof DetachedCall detached) { + matches = methodMatcher.matchKeys(detached.invokedObjectType(), detached.methodName(), + detached.parameterTypes()); +} else { + matches = methodMatcher.match(callContext.tree(), languageSupport.translation(), hook.matchContext()); +} +if (matches) { stackCalls.add(callContext); } +``` + +- [ ] **Step 3: Carry the `CallContext` through the fire path.** Change signatures from `T invocationTree` to `CallContext callContext`: + - `HookRepository.update` (`:120-121`): `handler.notifyAllHookDetectionObservers(callContext, hook)` (drop the separate tree/publisher args). + - `HookDetectionObservable.notify` (`:57-75`) + `IHookDetectionObservable`: take `CallContext callContext`, call `subscribers.get(i).onHookInvocation(callContext, hook)`. + - `IHookDetectionObserver.onHookInvocation` + `DetectionStore.onHookInvocation` (`:399-416`): take `CallContext callContext`; build the child store with `callContext.publisher()` and pass `callContext` into `DetectionStoreWithHook`. + - Update `Handler.notifyAllHookDetectionObservers` accordingly. + - `DetectionStoreWithHook` stores `CallContext callContext` instead of `T invocationTree`; `invocationTree` usages become `callContext.tree()` (retained) or the snapshot branch (detached). + +- [ ] **Step 4: Detached branch in `DetectionStoreWithHook.handleMethodInvocationHookWithParameterResolvement`** (`:124-203`). When `callContext instanceof DetachedCall`, skip `extractArgumentFromMethodCaller`/`resolveValuesInInnerScope` and use the snapshot at the hook's parameter index: +```java +if (callContext instanceof DetachedCall detached) { + int idx = parameterIndex(hook.methodDefinition(), hook.methodParameter()); + if (idx < 0 || idx >= detached.arguments().size()) { return; } + ArgSnapshot snap = detached.arguments().get(idx); + if (hook.getParameter() instanceof DetectableParameter detectableParameter) { + for (ArgSnapshot.ResolvedSnapshotValue rv : snap.values()) { + @SuppressWarnings("unchecked") + T loc = (T) rv.location(); + ResolvedValue resolved = new ResolvedValue<>(rv.value(), loc); + new ValueDetection<>(resolved, detectableParameter, loc, null) + .toValue(detectableParameter.getiValueFactory()) + .ifPresent(iValue -> addValue(detectableParameter.getIndex(), iValue)); + } + } + handleNextRulesForMethodHooks(hook, /* traceSymbol */ null, isSuccessive); + return; +} +// ... existing retained-tree logic unchanged ... +``` +Add helper (index of the hook's parameter identifier within the method definition): +```java +private int parameterIndex(@Nonnull T methodDefinition, @Nonnull T methodParameter) { + return handler.getLanguageSupport().parameterIndexOf(methodDefinition, methodParameter); +} +``` +Add `parameterIndexOf(T methodDefinition, T methodParameter)` to `ILanguageSupport` (default `-1`) and implement for Java in `JavaLanguageSupport` by matching `methodParameter`'s name against `((MethodTree) methodDefinition).parameters()` simple names (reuse `translation().resolveIdentifierAsString`). Because `T loc` is a `DetachedSyntaxToken` (a `Tree`), the cast is safe for Java (`T = Tree`). + +- [ ] **Step 5: Run the full engine suite + the cross-file baseline (now exercising the detached path).** +Run: `mvn test -pl engine && mvn test -pl java -Dtest=CrossFileHookDetachTest` +Expected: PASS. The Task 1 test now flows through `DetachedCall` (the `"AES/GCM/NoPadding"` literal resolves at record time). If it fails, log the recorded variant in `CallStackAgent.add` at DEBUG and confirm the usage call became a `DetachedCall` and the snapshot index maps to the `transformation` parameter. + +- [ ] **Step 6: Commit Tasks 8 + 9 together.** +```bash +mvn spotless:apply +git add engine/src/main/java/com/ibm/engine java/ # staged engine + any java touch +git commit -m "feat(engine): produce and replay DetachedCall for Java method invocations" +``` + +--- + +## Task 10: `JavaTranslator` DetachedSyntaxToken branch (location fidelity) + +**Files:** +- Modify: `java/src/main/java/com/ibm/plugin/translation/translator/JavaTranslator.java` +- Test: `java/src/test/java/com/ibm/plugin/translation/translator/JavaTranslatorDetachedLocationTest.java` + +**Interfaces:** +- Consumes: `DetachedSyntaxToken` (engine). +- Produces: `getDetectionContextFrom` returns a `DetectionLocation` built directly from a `DetachedSyntaxToken` when the location is one. + +- [ ] **Step 1: Write the failing test.** +```java +/* */ +package com.ibm.plugin.translation.translator; + +import static org.assertj.core.api.Assertions.assertThat; + +import com.ibm.engine.callstack.DetachedSyntaxToken; +import com.ibm.mapper.model.IBundle; +import com.ibm.mapper.utils.DetectionLocation; +import java.util.List; +import org.junit.jupiter.api.Test; +import org.mockito.Mockito; + +class JavaTranslatorDetachedLocationTest { + @Test + void buildsLocationFromDetachedToken() { + DetachedSyntaxToken token = + new DetachedSyntaxToken(7, 2, 7, 9, "AES", List.of("kw")); + DetectionLocation loc = + new JavaTranslator().getDetectionContextFrom(token, Mockito.mock(IBundle.class), "F.java"); + assertThat(loc.lineNumber()).isEqualTo(7); + assertThat(loc.keywords()).containsExactly("kw"); + } +} +``` +(Confirm `DetectionLocation`'s accessor names via `mapper/.../DetectionLocation.java:26-31`; adjust `lineNumber()`/`keywords()` to the real record component names.) + +- [ ] **Step 2: Run — verify it fails.** +Run: `mvn test -pl java -Dtest=JavaTranslatorDetachedLocationTest` +Expected: FAIL — falls through to token logic / empty keywords. + +- [ ] **Step 3: Add the leading branch** to `getDetectionContextFrom` (`JavaTranslator.java:170`): +```java +@Nullable public DetectionLocation getDetectionContextFrom( + @Nonnull Tree location, @Nonnull IBundle bundle, @Nonnull String filePath) { + if (location instanceof DetachedSyntaxToken d) { + return new DetectionLocation(filePath, d.line(), d.offset(), d.keywords(), bundle); + } + // ... existing logic unchanged ... +} +``` +Add import `com.ibm.engine.callstack.DetachedSyntaxToken`. + +- [ ] **Step 4: Run — verify PASS.** +Run: `mvn test -pl java -Dtest=JavaTranslatorDetachedLocationTest` +Expected: PASS. + +- [ ] **Step 5: Commit.** +```bash +mvn spotless:apply +git add java/src/main/java/com/ibm/plugin/translation/translator/JavaTranslator.java java/src/test/java/com/ibm/plugin/translation/translator/JavaTranslatorDetachedLocationTest.java +git commit -m "feat(java): translate DetachedSyntaxToken locations directly" +``` + +--- + +## Task 11: End-to-end verification + `getLocation()` consumer audit + +**Files:** +- Modify (if audit finds a gap): the offending consumer only. +- Test: extend `CrossFileHookDetachTest` with a NEW_ARRAY (tree-fallback) case. + +- [ ] **Step 1: Add a fallback-path case to the cross-file test.** This exercises the `isDetachableCall == false` retained path across files (a `NEW_ARRAY` argument keeps the tree). + +`java/src/test/files/rules/detection/crossfile/CrossFileIvDefinition.java`: +```java +package rules.detection.crossfile; + +import javax.crypto.spec.IvParameterSpec; + +public class CrossFileIvDefinition { + public IvParameterSpec make(byte[] iv) { + return new IvParameterSpec(iv); // NEW_ARRAY flows here -> retained (non-detachable) call + } +} +``` +`java/src/test/files/rules/detection/crossfile/CrossFileIvUsage.java`: +```java +package rules.detection.crossfile; + +import javax.crypto.spec.IvParameterSpec; + +public class CrossFileIvUsage { + public IvParameterSpec use() { + return new CrossFileIvDefinition().make(new byte[16]); + } +} +``` +Add a test method to `CrossFileHookDetachTest` (reuse its `collectValues`/`sawAes` pattern with a fresh flag `sawIv`, asserting an `IvParameterSpec`/nonce-related detection value is produced across the two files): +```java +private static boolean sawIv = false; + +@Test +void crossFileArrayStillDetects() { + sawIv = false; + CheckVerifier.newVerifier() + .onFiles( + "src/test/files/rules/detection/crossfile/CrossFileIvUsage.java", + "src/test/files/rules/detection/crossfile/CrossFileIvDefinition.java") + .withCheck(this) + .verifyNoIssues(); + assertThat(sawIv).as("cross-file retained (array) path still detects").isTrue(); +} +``` +In `collectValues`, also set `sawIv = true` when a value's class/asString indicates the IV/nonce detection (adjust the marker to whatever `JavaDetectionRules` emit for `IvParameterSpec`; run once and inspect the logged detection store to pick the exact value type). + +- [ ] **Step 2: Run both cross-file cases.** +Run: `mvn test -pl java -Dtest=CrossFileHookDetachTest` +Expected: PASS (both literal-detached and array-retained resolve). + +- [ ] **Step 3: Audit `getLocation()` consumers** (spec top risk). Confirm no consumer navigates `parent()`/`accept()`/children on a value location: +Run: `grep -rn "getLocation()" --include=*.java engine java mapper output enricher | grep -v /target/ | grep -v test` +Expected: only translator `getDetectionContextFrom` calls; if any consumer calls `.parent()`/`.accept()`/child accessors on a location, add a `DetachedSyntaxToken` guard there. Record the audit result in the commit message. + +- [ ] **Step 4: Full suite green.** +Run: `mvn test -pl engine && mvn test -pl java` +Expected: PASS. + +- [ ] **Step 5: Commit.** +```bash +mvn spotless:apply +git add java/src/test engine java # only files actually changed +git commit -m "test: cross-file detach + retained-array cases; getLocation consumer audit" +``` + +--- + +## Task 12: Cleanup — remove redundant `visitedTreeObjects` + +Measured 100% redundant with `invokedCallStack` retention. Dedup within the per-name bucket instead. + +**Files:** +- Modify: `engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java` + +- [ ] **Step 1: Replace the field-based dedup** (`CallStackAgent.java:45`, `:123-127`) with a per-bucket check. In `addedToCallContext`, for retained calls test membership in the bucket list by tree identity before appending; detached calls always append (distinct records): +```java +private boolean addedToCallContext(int key, @Nonnull CallContext callContext) { + final T tree = callContext.tree(); + final boolean[] added = {true}; + invokedCallStack.compute(key, (k, v) -> { + List> list = (v == null) ? new ArrayList<>() : v; + if (tree != null) { + for (CallContext existing : list) { + if (tree.equals(existing.tree())) { added[0] = false; return list; } + } + } + list.add(callContext); + return list; + }); + return added[0]; +} +``` +Delete the `visitedTreeObjects` field and its imports. + +- [ ] **Step 2: Full engine + cross-file green.** +Run: `mvn test -pl engine && mvn test -pl java -Dtest=CrossFileHookDetachTest` +Expected: PASS. + +- [ ] **Step 3: Commit.** +```bash +mvn spotless:apply +git add engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java +git commit -m "perf(engine): drop redundant visitedTreeObjects; dedup within bucket" +``` + +--- + +## Task 13: Cleanup — key-indexed subscription lookup + +`onNewHookSubscription` currently scans all buckets (`CallStackAgent.java:101`). Look up only the bucket for the hook's method name. + +**Files:** +- Modify: `engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java` + +- [ ] **Step 1: Derive the name key from the hook and scan one bucket.** In `onNewHookSubscription`, after building `methodMatcher`, compute the hook's method-name key the same way records are keyed (`methodName.hashCode()` for detached, `getKeyFormT` for retained) and iterate only `invokedCallStack.get(key)`. If the hook's method name isn't statically known (multi-name/`ANY` matcher), fall back to scanning `invokedCallStack.values()` as today. Obtain the name from `languageSupport.translation().getMethodName(hook.matchContext(), hook.hookValue())`; if empty, use the fallback scan. +```java +final Optional hookName = + languageSupport.translation().getMethodName(hook.matchContext(), hook.hookValue()); +final Collection>> buckets = + hookName.map(n -> invokedCallStack.getOrDefault(n.hashCode(), List.of())) + .map(List::of) + .map(l -> (Collection>>) l) + .orElseGet(invokedCallStack::values); +``` +Then iterate `buckets` with the existing match logic (Task 9 Step 2). Keep the fallback correct for hooks whose name resolves differently than the record key. + +- [ ] **Step 2: Full engine + cross-file green** (the Task 1 test registers the hook in the definition file and must still find the usage-file record via the keyed bucket). +Run: `mvn test -pl engine && mvn test -pl java -Dtest=CrossFileHookDetachTest` +Expected: PASS. + +- [ ] **Step 3: Commit.** +```bash +mvn spotless:apply +git add engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java +git commit -m "perf(engine): key-indexed subscription lookup in CallStackAgent" +``` + +--- + +## Task 14: Heap measurement + docs + +**Files:** +- Modify: `docs/TROUBLESHOOTING.md` (append a short note referencing the measurement) + +- [ ] **Step 1: Full build + all module tests.** +Run: `mvn clean test` +Expected: PASS across all modules. + +- [ ] **Step 2: Measure heap on a large compiled project.** Use the keycloak repro from the memory note (`mvn sonar:sonar` so types resolve; source-only scans fire `addCall` zero times). Constrain the scanner heap and capture before/after: +```bash +MAVEN_OPTS="-Xmx4g" mvn -o sonar:sonar -Dsonar.host.url=... 2>&1 | tee scan.log +# in another shell, sample the scanner JVM: +jmap -histo | grep -E "CallContext|DetachedCall|RetainedCall|Tree" | head +``` +Record retained `RetainedCall` vs `DetachedCall` counts and peak used heap; compare to the ~179k `CallContext` / ~7 GB baseline in `docs/superpowers/plans/2026-07-05-callstack-hooks-heap-reduction.md`. + +- [ ] **Step 3: Write the result** as a short paragraph in `docs/TROUBLESHOOTING.md` (retained vs detached split, peak-heap delta). If detached share is low, note it (candidate follow-ups: enum detachment, array pre-resolution). + +- [ ] **Step 4: Commit.** +```bash +git add docs/TROUBLESHOOTING.md +git commit -m "docs: record call-stack detach heap measurement" +``` + +--- + +## Self-review notes (coverage map) + +- Spec §"Match" → Task 4 (`matchKeys`), Task 9 Steps 1–2. +- Spec §"Resolution input" (record-time pre-resolution, NEW_ARRAY fallback, expressionToResolve n/a for Java) → Task 5 (predicate), Task 8 (`buildDetachedCall`), Task 9 Step 4 (snapshot replay). +- Spec §"Detection output / synthetic token" → Task 2, Task 10. +- Spec §"scanContext gap" → Task 3, Task 8 Step 5, Task 9 Step 3. +- Spec §"Hybrid detach with tree-fallback" → Task 5 + Task 8 Step 4 (retained fallback paths). +- Spec §"Lossless cleanups" → Task 12 (visitedTreeObjects), Task 13 (key-indexed lookup). +- Spec §"Verification" → Task 1 (baseline), Task 11 (both paths + consumer audit), Task 14 (heap profile). +- Spec §"out of scope" (Python/Go, enums, caps) → enforced by default-`false` predicate; enum site left on `addCallToCallStack` (Task 8 Step 4). From 1224843fd0bc031315cc5b33c61cfb9184f56826 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Sun, 5 Jul 2026 22:33:54 +0200 Subject: [PATCH 04/35] test: first true cross-file detection guard (compiled-classpath CheckVerifier) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- .../crossfile/KeyGeneratorCaller.java | 10 +++ .../crossfile/KeyGeneratorWrapper.java | 13 +++ .../crossfile/CrossFileHookDetachTest.java | 90 +++++++++++++++++++ 3 files changed, 113 insertions(+) create mode 100644 java/src/test/files/rules/detection/crossfile/KeyGeneratorCaller.java create mode 100644 java/src/test/files/rules/detection/crossfile/KeyGeneratorWrapper.java create mode 100644 java/src/test/java/com/ibm/plugin/rules/detection/crossfile/CrossFileHookDetachTest.java diff --git a/java/src/test/files/rules/detection/crossfile/KeyGeneratorCaller.java b/java/src/test/files/rules/detection/crossfile/KeyGeneratorCaller.java new file mode 100644 index 00000000..51579cf8 --- /dev/null +++ b/java/src/test/files/rules/detection/crossfile/KeyGeneratorCaller.java @@ -0,0 +1,10 @@ +package rules.detection.crossfile; + +import java.security.NoSuchAlgorithmException; +import javax.crypto.SecretKey; + +public class KeyGeneratorCaller { + public SecretKey call() throws NoSuchAlgorithmException { + return KeyGeneratorWrapper.generate("AES", 128); // Noncompliant {{(SecretKey) AES}} + } +} diff --git a/java/src/test/files/rules/detection/crossfile/KeyGeneratorWrapper.java b/java/src/test/files/rules/detection/crossfile/KeyGeneratorWrapper.java new file mode 100644 index 00000000..3d58b1c4 --- /dev/null +++ b/java/src/test/files/rules/detection/crossfile/KeyGeneratorWrapper.java @@ -0,0 +1,13 @@ +package rules.detection.crossfile; + +import java.security.NoSuchAlgorithmException; +import javax.crypto.KeyGenerator; +import javax.crypto.SecretKey; + +public class KeyGeneratorWrapper { + public static SecretKey generate(String algo, int keySize) throws NoSuchAlgorithmException { + KeyGenerator keyGenerator = KeyGenerator.getInstance(algo); + keyGenerator.init(keySize); + return keyGenerator.generateKey(); + } +} diff --git a/java/src/test/java/com/ibm/plugin/rules/detection/crossfile/CrossFileHookDetachTest.java b/java/src/test/java/com/ibm/plugin/rules/detection/crossfile/CrossFileHookDetachTest.java new file mode 100644 index 00000000..6002acf7 --- /dev/null +++ b/java/src/test/java/com/ibm/plugin/rules/detection/crossfile/CrossFileHookDetachTest.java @@ -0,0 +1,90 @@ +/* + * Sonar Cryptography Plugin + * Copyright (C) 2024 PQCA + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.ibm.plugin.rules.detection.crossfile; + +import com.ibm.engine.detection.DetectionStore; +import com.ibm.mapper.model.INode; +import com.ibm.plugin.TestBase; +import java.io.File; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; +import javax.annotation.Nonnull; +import javax.tools.JavaCompiler; +import javax.tools.ToolProvider; +import org.junit.jupiter.api.Test; +import org.sonar.java.checks.verifier.CheckVerifier; +import org.sonar.plugins.java.api.JavaCheck; +import org.sonar.plugins.java.api.JavaFileScannerContext; +import org.sonar.plugins.java.api.semantic.Symbol; +import org.sonar.plugins.java.api.tree.Tree; + +/** + * First true cross-file detection test in the suite. A hook created while analyzing {@code + * KeyGeneratorWrapper.java} must resolve a call recorded while analyzing {@code + * KeyGeneratorCaller.java}. + * + *

Cross-file resolution requires the callee's type to resolve at the call site. Production + * achieves this via {@code sonar.java.binaries} (the project is compiled first). CheckVerifier does + * not put sibling sources on the semantic classpath, so we reproduce production by compiling the + * fixtures to {@code .class} and passing the output directory to {@link CheckVerifier#withClassPath}. + * The {@code // Noncompliant} marker in the caller fixture asserts the cross-file detection fired. + */ +class CrossFileHookDetachTest extends TestBase { + + private static final String WRAPPER = + "src/test/files/rules/detection/crossfile/KeyGeneratorWrapper.java"; + private static final String CALLER = + "src/test/files/rules/detection/crossfile/KeyGeneratorCaller.java"; + + @Test + void crossFileLiteralResolves() throws Exception { + File classes = compileToClasspath(WRAPPER, CALLER); + CheckVerifier.newVerifier() + .onFiles(CALLER, WRAPPER) + .withClassPath(List.of(classes)) + .withChecks(this) + .verifyIssues(); + } + + /** Compiles the fixtures to a temp dir so their types resolve across files (mimics binaries). */ + private static File compileToClasspath(String... sources) throws Exception { + Path out = Files.createTempDirectory("xfile-classes"); + JavaCompiler compiler = ToolProvider.getSystemJavaCompiler(); + List args = new ArrayList<>(List.of("-d", out.toString())); + for (String s : sources) { + args.add(new File(s).getAbsolutePath()); + } + int rc = compiler.run(null, null, null, args.toArray(new String[0])); + if (rc != 0) { + throw new IllegalStateException("fixture compilation failed rc=" + rc); + } + return out.toFile(); + } + + @Override + public void asserts( + int findingId, + @Nonnull DetectionStore detectionStore, + @Nonnull List nodes) { + // Cross-file resolution is asserted by the Noncompliant marker in KeyGeneratorCaller.java. + } +} From 66b7e5e7a2bbb2c9e8a76a51372b010b9764aff9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Sun, 5 Jul 2026 22:39:15 +0200 Subject: [PATCH 05/35] test: harden cross-file guard with field-constant (detachable) + array (retained) cases MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- .../detection/crossfile/KeyGeneratorCaller.java | 15 +++++++++++++++ .../detection/crossfile/KeyGeneratorWrapper.java | 9 +++++++++ 2 files changed, 24 insertions(+) diff --git a/java/src/test/files/rules/detection/crossfile/KeyGeneratorCaller.java b/java/src/test/files/rules/detection/crossfile/KeyGeneratorCaller.java index 51579cf8..b2bdfb99 100644 --- a/java/src/test/files/rules/detection/crossfile/KeyGeneratorCaller.java +++ b/java/src/test/files/rules/detection/crossfile/KeyGeneratorCaller.java @@ -4,7 +4,22 @@ import javax.crypto.SecretKey; public class KeyGeneratorCaller { + + // Record-time-resolvable field argument -> detachable via symbol resolution at record time. + private static final String ALGO = "Blowfish"; + + // Literal argument -> detachable path. public SecretKey call() throws NoSuchAlgorithmException { return KeyGeneratorWrapper.generate("AES", 128); // Noncompliant {{(SecretKey) AES}} } + + // Field-constant argument -> detachable, resolved from the field at record time. + public SecretKey callField() throws NoSuchAlgorithmException { + return KeyGeneratorWrapper.generate(ALGO, 128); // Noncompliant {{(SecretKey) Blowfish}} + } + + // NEW_ARRAY argument -> non-detachable, must resolve via the retained-tree fallback path. + public SecretKey callWithArray() throws NoSuchAlgorithmException { + return KeyGeneratorWrapper.generateWithIv("DES", 56, new byte[8]); // Noncompliant {{(SecretKey) DES}} + } } diff --git a/java/src/test/files/rules/detection/crossfile/KeyGeneratorWrapper.java b/java/src/test/files/rules/detection/crossfile/KeyGeneratorWrapper.java index 3d58b1c4..25f9331f 100644 --- a/java/src/test/files/rules/detection/crossfile/KeyGeneratorWrapper.java +++ b/java/src/test/files/rules/detection/crossfile/KeyGeneratorWrapper.java @@ -10,4 +10,13 @@ public static SecretKey generate(String algo, int keySize) throws NoSuchAlgorith keyGenerator.init(keySize); return keyGenerator.generateKey(); } + + // Extra byte[] parameter so a caller passing `new byte[N]` makes the recorded call carry a + // NEW_ARRAY argument -> the detach predicate must keep this call on the retained-tree path. + public static SecretKey generateWithIv(String algo, int keySize, byte[] iv) + throws NoSuchAlgorithmException { + KeyGenerator keyGenerator = KeyGenerator.getInstance(algo); + keyGenerator.init(keySize); + return keyGenerator.generateKey(); + } } From e12849e06a9333cab4e341b7bd20387ac51b5e9e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Sun, 5 Jul 2026 22:41:22 +0200 Subject: [PATCH 06/35] feat(engine): AST-free DetachedSyntaxToken for detached value locations MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- .../engine/callstack/DetachedSyntaxToken.java | 162 ++++++++++++++++++ .../callstack/DetachedSyntaxTokenTest.java | 59 +++++++ 2 files changed, 221 insertions(+) create mode 100644 engine/src/main/java/com/ibm/engine/callstack/DetachedSyntaxToken.java create mode 100644 engine/src/test/java/com/ibm/engine/callstack/DetachedSyntaxTokenTest.java diff --git a/engine/src/main/java/com/ibm/engine/callstack/DetachedSyntaxToken.java b/engine/src/main/java/com/ibm/engine/callstack/DetachedSyntaxToken.java new file mode 100644 index 00000000..53d4d4ca --- /dev/null +++ b/engine/src/main/java/com/ibm/engine/callstack/DetachedSyntaxToken.java @@ -0,0 +1,162 @@ +/* + * Sonar Cryptography Plugin + * Copyright (C) 2024 PQCA + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.ibm.engine.callstack; + +import java.util.List; +import javax.annotation.Nonnull; +import javax.annotation.Nullable; +import org.sonar.plugins.java.api.location.Position; +import org.sonar.plugins.java.api.location.Range; +import org.sonar.plugins.java.api.tree.SyntaxToken; +import org.sonar.plugins.java.api.tree.SyntaxTrivia; +import org.sonar.plugins.java.api.tree.Tree; +import org.sonar.plugins.java.api.tree.TreeVisitor; + +/** + * AST-free {@link SyntaxToken} used as the location of a detached cross-file detection value. + * + *

It holds only primitives captured at record time, so it never pins a compilation unit ({@link + * #parent()} is {@code null}). The Java translator reads {@link #line()}, {@link #offset()} and + * {@link #keywords()} from it directly to build a {@code DetectionLocation} without touching a live + * tree. Column offsets are stored 0-based (matching {@link Position#columnOffset()}); {@link + * Position#at(int, int)} takes a 1-based column, so {@link #range()} adds one. + */ +public final class DetachedSyntaxToken implements SyntaxToken { + + private final int line; + private final int columnOffset; + private final int endLine; + private final int endColumnOffset; + @Nonnull private final String text; + @Nonnull private final List keywords; + + public DetachedSyntaxToken( + int line, + int columnOffset, + int endLine, + int endColumnOffset, + @Nonnull String text, + @Nonnull List keywords) { + this.line = line; + this.columnOffset = columnOffset; + this.endLine = endLine; + this.endColumnOffset = endColumnOffset; + this.text = text; + this.keywords = List.copyOf(keywords); + } + + @Nonnull + public List keywords() { + return keywords; + } + + /** 0-based start column offset, matching {@code DetectionLocation}'s offset field. */ + public int offset() { + return columnOffset; + } + + @Override + public String text() { + return text; + } + + @Override + public List trivias() { + return List.of(); + } + + @Override + public int line() { + return line; + } + + @Override + public int column() { + return columnOffset; + } + + @Override + public Range range() { + return Range.at( + Position.at(line, columnOffset + 1), Position.at(endLine, endColumnOffset + 1)); + } + + @Override + public boolean is(Tree.Kind... kinds) { + for (Tree.Kind kind : kinds) { + if (kind == Tree.Kind.TOKEN) { + return true; + } + } + return false; + } + + @Override + public void accept(TreeVisitor visitor) { + // Detached: not part of an AST, nothing to traverse. + } + + @Nullable @Override + public Tree parent() { + return null; + } + + @Override + public SyntaxToken firstToken() { + return this; + } + + @Override + public SyntaxToken lastToken() { + return this; + } + + @Override + public Tree.Kind kind() { + return Tree.Kind.TOKEN; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof DetachedSyntaxToken that)) { + return false; + } + return line == that.line + && columnOffset == that.columnOffset + && endLine == that.endLine + && endColumnOffset == that.endColumnOffset + && text.equals(that.text) + && keywords.equals(that.keywords); + } + + @Override + public int hashCode() { + int result = line; + result = 31 * result + columnOffset; + result = 31 * result + endLine; + result = 31 * result + endColumnOffset; + result = 31 * result + text.hashCode(); + result = 31 * result + keywords.hashCode(); + return result; + } +} diff --git a/engine/src/test/java/com/ibm/engine/callstack/DetachedSyntaxTokenTest.java b/engine/src/test/java/com/ibm/engine/callstack/DetachedSyntaxTokenTest.java new file mode 100644 index 00000000..0c2e1a8f --- /dev/null +++ b/engine/src/test/java/com/ibm/engine/callstack/DetachedSyntaxTokenTest.java @@ -0,0 +1,59 @@ +/* + * Sonar Cryptography Plugin + * Copyright (C) 2024 PQCA + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.ibm.engine.callstack; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.util.List; +import org.junit.jupiter.api.Test; +import org.sonar.plugins.java.api.tree.Tree; + +class DetachedSyntaxTokenTest { + + @Test + void pinsNothingAndReportsRange() { + DetachedSyntaxToken token = + new DetachedSyntaxToken( + 12, 4, 12, 20, "AES", List.of("javax.crypto.Cipher#getInstance")); + + assertThat(token.parent()).isNull(); + assertThat(token.kind()).isEqualTo(Tree.Kind.TOKEN); + assertThat(token.is(Tree.Kind.TOKEN)).isTrue(); + assertThat(token.is(Tree.Kind.METHOD_INVOCATION)).isFalse(); + assertThat(token.firstToken()).isSameAs(token); + assertThat(token.lastToken()).isSameAs(token); + assertThat(token.text()).isEqualTo("AES"); + assertThat(token.trivias()).isEmpty(); + assertThat(token.keywords()).containsExactly("javax.crypto.Cipher#getInstance"); + assertThat(token.line()).isEqualTo(12); + assertThat(token.offset()).isEqualTo(4); + assertThat(token.range().start().line()).isEqualTo(12); + assertThat(token.range().start().columnOffset()).isEqualTo(4); + } + + @Test + void valueEquality() { + DetachedSyntaxToken a = new DetachedSyntaxToken(1, 2, 1, 5, "AES", List.of("k")); + DetachedSyntaxToken b = new DetachedSyntaxToken(1, 2, 1, 5, "AES", List.of("k")); + DetachedSyntaxToken c = new DetachedSyntaxToken(1, 3, 1, 5, "AES", List.of("k")); + assertThat(a).isEqualTo(b).hasSameHashCodeAs(b); + assertThat(a).isNotEqualTo(c); + } +} From fa51c963a4c3d8f435fc63d47f6894ba871cabc5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Sun, 5 Jul 2026 22:42:39 +0200 Subject: [PATCH 07/35] feat(engine): AST-free DetachedScanContext MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- .../engine/callstack/DetachedScanContext.java | 54 +++++++++++++++++++ .../callstack/DetachedScanContextTest.java | 48 +++++++++++++++++ 2 files changed, 102 insertions(+) create mode 100644 engine/src/main/java/com/ibm/engine/callstack/DetachedScanContext.java create mode 100644 engine/src/test/java/com/ibm/engine/callstack/DetachedScanContextTest.java diff --git a/engine/src/main/java/com/ibm/engine/callstack/DetachedScanContext.java b/engine/src/main/java/com/ibm/engine/callstack/DetachedScanContext.java new file mode 100644 index 00000000..77dabf54 --- /dev/null +++ b/engine/src/main/java/com/ibm/engine/callstack/DetachedScanContext.java @@ -0,0 +1,54 @@ +/* + * Sonar Cryptography Plugin + * Copyright (C) 2024 PQCA + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.ibm.engine.callstack; + +import com.ibm.engine.language.IScanContext; +import javax.annotation.Nonnull; +import org.sonar.api.batch.fs.InputFile; + +/** + * AST-free {@link IScanContext}: retains only the file handle and path captured at record time, + * never the language-specific file scanner context (e.g. {@code JavaFileScannerContext}). A + * detached recorded call carries this so that replaying it does not pin the file's AST. + * + *

{@link #reportIssue} is unavailable: cross-file detections are emitted to the CBOM via + * translation, not via {@code reportIssue}, and the detached context has no live tree to report on. + */ +public record DetachedScanContext(@Nonnull InputFile inputFile, @Nonnull String filePath) + implements IScanContext { + + @Override + public void reportIssue(@Nonnull R currentRule, @Nonnull T tree, @Nonnull String message) { + throw new UnsupportedOperationException( + "reportIssue is not available on a detached scan context"); + } + + @Nonnull + @Override + public InputFile getInputFile() { + return inputFile; + } + + @Nonnull + @Override + public String getFilePath() { + return filePath; + } +} diff --git a/engine/src/test/java/com/ibm/engine/callstack/DetachedScanContextTest.java b/engine/src/test/java/com/ibm/engine/callstack/DetachedScanContextTest.java new file mode 100644 index 00000000..124a3fcf --- /dev/null +++ b/engine/src/test/java/com/ibm/engine/callstack/DetachedScanContextTest.java @@ -0,0 +1,48 @@ +/* + * Sonar Cryptography Plugin + * Copyright (C) 2024 PQCA + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.ibm.engine.callstack; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.mockito.Mockito.mock; + +import org.junit.jupiter.api.Test; +import org.sonar.api.batch.fs.InputFile; + +class DetachedScanContextTest { + + @Test + void exposesFilePathAndInputFileWithoutPinningAst() { + InputFile inputFile = mock(InputFile.class); + DetachedScanContext ctx = + new DetachedScanContext<>(inputFile, "/abs/path/CrossFileUsage.java"); + + assertThat(ctx.getFilePath()).isEqualTo("/abs/path/CrossFileUsage.java"); + assertThat(ctx.getInputFile()).isSameAs(inputFile); + } + + @Test + void reportIssueIsUnsupported() { + DetachedScanContext ctx = + new DetachedScanContext<>(mock(InputFile.class), "/p/F.java"); + assertThatThrownBy(() -> ctx.reportIssue(new Object(), new Object(), "msg")) + .isInstanceOf(UnsupportedOperationException.class); + } +} From 4654a6e2b15a96b790a05c014aee63a07a2cb92f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Sun, 5 Jul 2026 22:46:28 +0200 Subject: [PATCH 08/35] feat(engine): ArgSnapshot + MethodMatcher.matchKeys (tree-free match) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- .../com/ibm/engine/callstack/ArgSnapshot.java | 34 ++++++++ .../ibm/engine/detection/MethodMatcher.java | 33 ++++++- .../detection/MethodMatcherKeysTest.java | 85 +++++++++++++++++++ 3 files changed, 148 insertions(+), 4 deletions(-) create mode 100644 engine/src/main/java/com/ibm/engine/callstack/ArgSnapshot.java create mode 100644 engine/src/test/java/com/ibm/engine/detection/MethodMatcherKeysTest.java diff --git a/engine/src/main/java/com/ibm/engine/callstack/ArgSnapshot.java b/engine/src/main/java/com/ibm/engine/callstack/ArgSnapshot.java new file mode 100644 index 00000000..733576f4 --- /dev/null +++ b/engine/src/main/java/com/ibm/engine/callstack/ArgSnapshot.java @@ -0,0 +1,34 @@ +/* + * Sonar Cryptography Plugin + * Copyright (C) 2024 PQCA + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.ibm.engine.callstack; + +import java.util.List; +import javax.annotation.Nonnull; + +/** + * Tree-free snapshot of one argument of a detached recorded call: the value(s) it resolved to at + * record time (while the file was still live) plus, for each, an AST-free location to report on. + */ +public record ArgSnapshot(int index, @Nonnull List values) { + + /** A single resolved value plus its detached location. */ + public record ResolvedSnapshotValue( + @Nonnull Object value, @Nonnull DetachedSyntaxToken location) {} +} diff --git a/engine/src/main/java/com/ibm/engine/detection/MethodMatcher.java b/engine/src/main/java/com/ibm/engine/detection/MethodMatcher.java index 846f1297..dc59a588 100644 --- a/engine/src/main/java/com/ibm/engine/detection/MethodMatcher.java +++ b/engine/src/main/java/com/ibm/engine/detection/MethodMatcher.java @@ -169,8 +169,33 @@ public boolean match( return false; } - boolean typeMatches = this.invokedObjectTypeString.test(invokedObjectType.get()); - boolean nameMatches = this.methodName.test(invokedMethodName.get()); + return matchKeysInternal( + invokedObjectType.get(), + invokedMethodName.get(), + param, + translation.supportsSubsetParameterMatching()); + } + + /** + * Tree-free equivalent of {@link #match}: matches against the invoked-object type, method name + * and parameter types extracted at record time, so a detached recorded call can be matched + * without retaining its AST. Subset (constructor) parameter matching is not applicable to + * detached calls, so it is disabled here. + */ + public boolean matchKeys( + @Nonnull IType invokedObjectType, + @Nonnull String methodName, + @Nonnull List parameterTypes) { + return matchKeysInternal(invokedObjectType, methodName, parameterTypes, false); + } + + private boolean matchKeysInternal( + @Nonnull IType invokedObjectType, + @Nonnull String invokedMethodName, + @Nonnull List param, + boolean supportsSubsetParameterMatching) { + boolean typeMatches = this.invokedObjectTypeString.test(invokedObjectType); + boolean nameMatches = this.methodName.test(invokedMethodName); if (!typeMatches || !nameMatches) { return false; @@ -178,9 +203,9 @@ public boolean match( // For languages supporting subset parameter matching (e.g., Go composite literals), // the rule matches if at least one expected parameter exists in the actual fields. - if (invokedMethodName.get().equals("") + if (invokedMethodName.equals("") && !parameterTypesSerializable.isEmpty() - && translation.supportsSubsetParameterMatching()) { + && supportsSubsetParameterMatching) { return anyParameterMatches(param); } diff --git a/engine/src/test/java/com/ibm/engine/detection/MethodMatcherKeysTest.java b/engine/src/test/java/com/ibm/engine/detection/MethodMatcherKeysTest.java new file mode 100644 index 00000000..632da384 --- /dev/null +++ b/engine/src/test/java/com/ibm/engine/detection/MethodMatcherKeysTest.java @@ -0,0 +1,85 @@ +/* + * Sonar Cryptography Plugin + * Copyright (C) 2024 PQCA + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.ibm.engine.detection; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.util.LinkedList; +import java.util.List; +import org.junit.jupiter.api.Test; + +class MethodMatcherKeysTest { + + /** IType fake: reports itself as the given fully-qualified name. */ + private static IType type(String fqn) { + return fqn::equals; + } + + private MethodMatcher cipherGetInstance() { + return new MethodMatcher<>( + "javax.crypto.Cipher", + "getInstance", + new LinkedList<>(List.of("java.lang.String"))); + } + + @Test + void matchesByKeys() { + assertThat( + cipherGetInstance() + .matchKeys( + type("javax.crypto.Cipher"), + "getInstance", + List.of(type("java.lang.String")))) + .isTrue(); + } + + @Test + void rejectsWrongName() { + assertThat( + cipherGetInstance() + .matchKeys( + type("javax.crypto.Cipher"), + "doFinal", + List.of(type("java.lang.String")))) + .isFalse(); + } + + @Test + void rejectsWrongType() { + assertThat( + cipherGetInstance() + .matchKeys( + type("javax.crypto.Mac"), + "getInstance", + List.of(type("java.lang.String")))) + .isFalse(); + } + + @Test + void rejectsWrongParameters() { + assertThat( + cipherGetInstance() + .matchKeys( + type("javax.crypto.Cipher"), + "getInstance", + List.of(type("int")))) + .isFalse(); + } +} From afe61bc8a8e9cc7da7a5f1a58a828ce24712c8e4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Sun, 5 Jul 2026 22:53:20 +0200 Subject: [PATCH 09/35] feat(engine): isDetachableCall predicate (method invocation without NEW_ARRAY arg) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- .../ibm/engine/language/ILanguageSupport.java | 14 +++ .../language/java/JavaLanguageSupport.java | 40 +++++++ .../crossfile/IsDetachableCallTest.java | 100 ++++++++++++++++++ 3 files changed, 154 insertions(+) create mode 100644 java/src/test/java/com/ibm/plugin/rules/detection/crossfile/IsDetachableCallTest.java diff --git a/engine/src/main/java/com/ibm/engine/language/ILanguageSupport.java b/engine/src/main/java/com/ibm/engine/language/ILanguageSupport.java index 64fef24e..d9002ff5 100644 --- a/engine/src/main/java/com/ibm/engine/language/ILanguageSupport.java +++ b/engine/src/main/java/com/ibm/engine/language/ILanguageSupport.java @@ -117,4 +117,18 @@ IDetectionEngine createDetectionEngineInstance( */ @Nullable EnumMatcher createSimpleEnumMatcherFor( @Nonnull T enumIdentifier, @Nonnull MatchContext matchContext); + + /** + * Whether a recorded call may be stored tree-free (detached) instead of retaining its AST for + * the whole scan. Detaching lets the file's AST be garbage-collected after analysis. + * + *

Default: conservative {@code false} (retain the tree, today's behavior). Only languages + * that can faithfully pre-resolve a call's arguments at record time override this. + * + * @param tree the recorded call + * @return {@code true} if the call can be recorded as a detached, tree-free record + */ + default boolean isDetachableCall(@Nonnull T tree) { + return false; + } } diff --git a/engine/src/main/java/com/ibm/engine/language/java/JavaLanguageSupport.java b/engine/src/main/java/com/ibm/engine/language/java/JavaLanguageSupport.java index bf7a28ce..9f4773d7 100644 --- a/engine/src/main/java/com/ibm/engine/language/java/JavaLanguageSupport.java +++ b/engine/src/main/java/com/ibm/engine/language/java/JavaLanguageSupport.java @@ -42,9 +42,12 @@ import org.sonar.plugins.java.api.JavaCheck; import org.sonar.plugins.java.api.JavaFileScannerContext; import org.sonar.plugins.java.api.semantic.Symbol; +import org.sonar.plugins.java.api.tree.BaseTreeVisitor; import org.sonar.plugins.java.api.tree.ClassTree; import org.sonar.plugins.java.api.tree.ExpressionTree; +import org.sonar.plugins.java.api.tree.MethodInvocationTree; import org.sonar.plugins.java.api.tree.MethodTree; +import org.sonar.plugins.java.api.tree.NewArrayTree; import org.sonar.plugins.java.api.tree.Tree; import org.sonar.plugins.java.api.tree.TypeTree; @@ -138,4 +141,41 @@ public EnumMatcher createSimpleEnumMatcherFor( translation().getEnumIdentifierName(matchContext, enumIdentifier); return enumIdentifierName.>map(EnumMatcher::new).orElse(null); } + + @Override + public boolean isDetachableCall(@Nonnull Tree tree) { + if (!(tree instanceof MethodInvocationTree invocation)) { + // Enum accesses and other kinds stay retained in this iteration. + return false; + } + // NB: we deliberately do NOT require methodSymbol().declaration() != null. Cross-file calls + // resolve their callee via the compiled classpath (sonar.java.binaries), so declaration() + // is + // null for exactly the cross-file calls that drive the heap retention. Detachability + // depends + // on argument pre-resolvability (checked when building the record, with a retained-tree + // fallback) and on matchKeys (record-time type/name/params), neither of which needs the + // callee's source declaration. + for (ExpressionTree argument : invocation.arguments()) { + if (containsNewArray(argument)) { + // NEW_ARRAY resolution is value-factory dependent (SizeFactory returns the array + // size, otherwise its elements). We cannot reproduce that generically at record + // time, so keep such calls on the retained-tree fallback path. + return false; + } + } + return true; + } + + private static boolean containsNewArray(@Nonnull Tree tree) { + final boolean[] found = {false}; + tree.accept( + new BaseTreeVisitor() { + @Override + public void visitNewArray(NewArrayTree newArrayTree) { + found[0] = true; + } + }); + return found[0]; + } } diff --git a/java/src/test/java/com/ibm/plugin/rules/detection/crossfile/IsDetachableCallTest.java b/java/src/test/java/com/ibm/plugin/rules/detection/crossfile/IsDetachableCallTest.java new file mode 100644 index 00000000..9ad2132f --- /dev/null +++ b/java/src/test/java/com/ibm/plugin/rules/detection/crossfile/IsDetachableCallTest.java @@ -0,0 +1,100 @@ +/* + * Sonar Cryptography Plugin + * Copyright (C) 2024 PQCA + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.ibm.plugin.rules.detection.crossfile; + +import static org.assertj.core.api.Assertions.assertThat; + +import com.ibm.engine.language.java.JavaLanguageSupport; +import java.io.File; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import javax.tools.JavaCompiler; +import javax.tools.ToolProvider; +import org.junit.jupiter.api.Test; +import org.sonar.java.checks.verifier.CheckVerifier; +import org.sonar.plugins.java.api.IssuableSubscriptionVisitor; +import org.sonar.plugins.java.api.tree.MethodInvocationTree; +import org.sonar.plugins.java.api.tree.Tree; + +/** + * Verifies {@link JavaLanguageSupport#isDetachableCall} on real, semantically-resolved cross-file + * calls. The fixtures are compiled to a classpath so the callee types resolve (mimicking {@code + * sonar.java.binaries}); crucially the callee's {@code declaration()} is then {@code null}, and the + * predicate must still treat a plain-argument call as detachable. + */ +class IsDetachableCallTest extends IssuableSubscriptionVisitor { + + private static final String WRAPPER = + "src/test/files/rules/detection/crossfile/KeyGeneratorWrapper.java"; + private static final String CALLER = + "src/test/files/rules/detection/crossfile/KeyGeneratorCaller.java"; + + private static final JavaLanguageSupport LANGUAGE_SUPPORT = new JavaLanguageSupport(); + private static final Map detachableByCallee = new HashMap<>(); + + @Test + void plainArgsDetachableAcrossFilesButArrayArgIsNot() throws Exception { + detachableByCallee.clear(); + File classes = compileToClasspath(WRAPPER, CALLER); + CheckVerifier.newVerifier() + .onFiles(CALLER, WRAPPER) + .withClassPath(List.of(classes)) + .withChecks(this) + .verifyNoIssues(); + + // Cross-file literal / field-constant call (declaration() is null via the classpath + // binary): + assertThat(detachableByCallee).containsEntry("generate", true); + // Call carrying a NEW_ARRAY argument must stay on the retained-tree path: + assertThat(detachableByCallee).containsEntry("generateWithIv", false); + } + + @Override + public List nodesToVisit() { + return List.of(Tree.Kind.METHOD_INVOCATION); + } + + @Override + public void visitNode(Tree tree) { + MethodInvocationTree invocation = (MethodInvocationTree) tree; + String callee = invocation.methodSymbol().name(); + if (callee.equals("generate") || callee.equals("generateWithIv")) { + detachableByCallee.put(callee, LANGUAGE_SUPPORT.isDetachableCall(tree)); + } + } + + private static File compileToClasspath(String... sources) throws Exception { + Path out = Files.createTempDirectory("xfile-detachable"); + JavaCompiler compiler = ToolProvider.getSystemJavaCompiler(); + List args = new ArrayList<>(List.of("-d", out.toString())); + for (String s : sources) { + args.add(new File(s).getAbsolutePath()); + } + int rc = compiler.run(null, null, null, args.toArray(new String[0])); + if (rc != 0) { + throw new IllegalStateException("fixture compilation failed rc=" + rc); + } + return out.toFile(); + } +} From b1fd55a1e9a9f15c374541f5b619e431b721eacc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Sun, 5 Jul 2026 22:53:38 +0200 Subject: [PATCH 10/35] style: spotless formatting on cross-file test MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- .../detection/crossfile/CrossFileHookDetachTest.java | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/java/src/test/java/com/ibm/plugin/rules/detection/crossfile/CrossFileHookDetachTest.java b/java/src/test/java/com/ibm/plugin/rules/detection/crossfile/CrossFileHookDetachTest.java index 6002acf7..c7e8f83c 100644 --- a/java/src/test/java/com/ibm/plugin/rules/detection/crossfile/CrossFileHookDetachTest.java +++ b/java/src/test/java/com/ibm/plugin/rules/detection/crossfile/CrossFileHookDetachTest.java @@ -45,8 +45,9 @@ *

Cross-file resolution requires the callee's type to resolve at the call site. Production * achieves this via {@code sonar.java.binaries} (the project is compiled first). CheckVerifier does * not put sibling sources on the semantic classpath, so we reproduce production by compiling the - * fixtures to {@code .class} and passing the output directory to {@link CheckVerifier#withClassPath}. - * The {@code // Noncompliant} marker in the caller fixture asserts the cross-file detection fired. + * fixtures to {@code .class} and passing the output directory to {@link + * CheckVerifier#withClassPath}. The {@code // Noncompliant} marker in the caller fixture asserts + * the cross-file detection fired. */ class CrossFileHookDetachTest extends TestBase { @@ -65,7 +66,9 @@ void crossFileLiteralResolves() throws Exception { .verifyIssues(); } - /** Compiles the fixtures to a temp dir so their types resolve across files (mimics binaries). */ + /** + * Compiles the fixtures to a temp dir so their types resolve across files (mimics binaries). + */ private static File compileToClasspath(String... sources) throws Exception { Path out = Files.createTempDirectory("xfile-classes"); JavaCompiler compiler = ToolProvider.getSystemJavaCompiler(); From 073b802a8adbec92e807615ac5ba0a26601ea749 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Mon, 6 Jul 2026 07:24:58 +0200 Subject: [PATCH 11/35] refactor(engine): CallContext -> sealed interface with RetainedCall variant MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- .../com/ibm/engine/callstack/CallContext.java | 18 ++++++++++- .../ibm/engine/callstack/CallStackAgent.java | 2 +- .../ibm/engine/callstack/RetainedCall.java | 30 +++++++++++++++++++ 3 files changed, 48 insertions(+), 2 deletions(-) create mode 100644 engine/src/main/java/com/ibm/engine/callstack/RetainedCall.java diff --git a/engine/src/main/java/com/ibm/engine/callstack/CallContext.java b/engine/src/main/java/com/ibm/engine/callstack/CallContext.java index 713cc016..38f4a1b8 100644 --- a/engine/src/main/java/com/ibm/engine/callstack/CallContext.java +++ b/engine/src/main/java/com/ibm/engine/callstack/CallContext.java @@ -21,5 +21,21 @@ import com.ibm.engine.language.IScanContext; import javax.annotation.Nonnull; +import javax.annotation.Nullable; -public record CallContext(@Nonnull T tree, @Nonnull IScanContext publisher) {} +/** + * A recorded call site accumulated by the {@link CallStackAgent} for later cross-file hook + * matching. + * + *

{@link RetainedCall} keeps the live AST tree (today's behavior); {@link DetachedCall} holds a + * tree-free snapshot so the file's AST can be garbage-collected after analysis. + */ +public sealed interface CallContext permits RetainedCall { + + /** The recorded call tree, or {@code null} for a detached record that holds no AST. */ + @Nullable T tree(); + + /** The scan context of the file the call was recorded in. */ + @Nonnull + IScanContext publisher(); +} diff --git a/engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java b/engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java index 2a8a0121..c2669157 100644 --- a/engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java +++ b/engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java @@ -57,7 +57,7 @@ public void addCall(@Nonnull T tree, @Nonnull IScanContext scanContext) { } int key = keyOptional.get(); - final CallContext callContext = new CallContext<>(tree, scanContext); + final CallContext callContext = new RetainedCall<>(tree, scanContext); if (addedToCallContext(key, callContext)) { this.notify(callContext); } diff --git a/engine/src/main/java/com/ibm/engine/callstack/RetainedCall.java b/engine/src/main/java/com/ibm/engine/callstack/RetainedCall.java new file mode 100644 index 00000000..165b71e7 --- /dev/null +++ b/engine/src/main/java/com/ibm/engine/callstack/RetainedCall.java @@ -0,0 +1,30 @@ +/* + * Sonar Cryptography Plugin + * Copyright (C) 2024 PQCA + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.ibm.engine.callstack; + +import com.ibm.engine.language.IScanContext; +import javax.annotation.Nonnull; + +/** + * A recorded call that retains its live AST tree and scan context — today's behavior. Retaining the + * tree pins the file's AST for the whole scan; used for calls that cannot be faithfully detached. + */ +public record RetainedCall(@Nonnull T tree, @Nonnull IScanContext publisher) + implements CallContext {} From c86fcc8ae23f5ca032029f1989d7a7cc3e4eda80 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Mon, 6 Jul 2026 07:26:05 +0200 Subject: [PATCH 12/35] feat(engine): DetachedCall variant (tree-free recorded call) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- .../com/ibm/engine/callstack/CallContext.java | 2 +- .../ibm/engine/callstack/DetachedCall.java | 53 +++++++++++++++++++ .../engine/callstack/DetachedCallTest.java | 47 ++++++++++++++++ 3 files changed, 101 insertions(+), 1 deletion(-) create mode 100644 engine/src/main/java/com/ibm/engine/callstack/DetachedCall.java create mode 100644 engine/src/test/java/com/ibm/engine/callstack/DetachedCallTest.java diff --git a/engine/src/main/java/com/ibm/engine/callstack/CallContext.java b/engine/src/main/java/com/ibm/engine/callstack/CallContext.java index 38f4a1b8..8534977f 100644 --- a/engine/src/main/java/com/ibm/engine/callstack/CallContext.java +++ b/engine/src/main/java/com/ibm/engine/callstack/CallContext.java @@ -30,7 +30,7 @@ *

{@link RetainedCall} keeps the live AST tree (today's behavior); {@link DetachedCall} holds a * tree-free snapshot so the file's AST can be garbage-collected after analysis. */ -public sealed interface CallContext permits RetainedCall { +public sealed interface CallContext permits RetainedCall, DetachedCall { /** The recorded call tree, or {@code null} for a detached record that holds no AST. */ @Nullable T tree(); diff --git a/engine/src/main/java/com/ibm/engine/callstack/DetachedCall.java b/engine/src/main/java/com/ibm/engine/callstack/DetachedCall.java new file mode 100644 index 00000000..cc8894c2 --- /dev/null +++ b/engine/src/main/java/com/ibm/engine/callstack/DetachedCall.java @@ -0,0 +1,53 @@ +/* + * Sonar Cryptography Plugin + * Copyright (C) 2024 PQCA + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.ibm.engine.callstack; + +import com.ibm.engine.detection.IType; +import com.ibm.engine.language.IScanContext; +import java.util.List; +import javax.annotation.Nonnull; +import javax.annotation.Nullable; + +/** + * A recorded call stored tree-free: the match keys and pre-resolved argument snapshots are captured + * at record time while the file is live, so the file's AST becomes garbage-collectable afterwards. + * + *

Matching uses {@link #invokedObjectType()} / {@link #methodName()} / {@link #parameterTypes()} + * via {@code MethodMatcher.matchKeys}; hook replay reads {@link #arguments()}. + */ +public record DetachedCall( + @Nonnull IType invokedObjectType, + @Nonnull String methodName, + @Nonnull List parameterTypes, + @Nonnull List arguments, + @Nonnull DetachedScanContext detachedPublisher) + implements CallContext { + + @Nullable @Override + public T tree() { + return null; + } + + @Nonnull + @Override + public IScanContext publisher() { + return detachedPublisher; + } +} diff --git a/engine/src/test/java/com/ibm/engine/callstack/DetachedCallTest.java b/engine/src/test/java/com/ibm/engine/callstack/DetachedCallTest.java new file mode 100644 index 00000000..74b77753 --- /dev/null +++ b/engine/src/test/java/com/ibm/engine/callstack/DetachedCallTest.java @@ -0,0 +1,47 @@ +/* + * Sonar Cryptography Plugin + * Copyright (C) 2024 PQCA + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.ibm.engine.callstack; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.Mockito.mock; + +import com.ibm.engine.detection.IType; +import java.util.List; +import org.junit.jupiter.api.Test; +import org.sonar.api.batch.fs.InputFile; + +class DetachedCallTest { + + @Test + void holdsNoTreeAndExposesKeys() { + DetachedScanContext ctx = + new DetachedScanContext<>(mock(InputFile.class), "/p/CrossFileUsage.java"); + IType owner = mock(IType.class); + + DetachedCall call = + new DetachedCall<>(owner, "make", List.of(), List.of(), ctx); + + assertThat(call.tree()).isNull(); + assertThat(call.publisher()).isSameAs(ctx); + assertThat(call.invokedObjectType()).isSameAs(owner); + assertThat(call.methodName()).isEqualTo("make"); + assertThat(call.arguments()).isEmpty(); + } +} From 9efdfa5f843862a1ae75ae84ed8ba480d2f0b125 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Mon, 6 Jul 2026 08:36:04 +0200 Subject: [PATCH 13/35] feat(engine): detach recorded calls at leaveFile with cross-file replay + reporting MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Record calls retained during their file's analysis (same-file hook detections resolve and report through the live context); at leaveFile, swap each detachable call for a pre-built tree-free DetachedCall so the file's AST is collected while cross-file matching continues from the snapshot. Cross-file detached detections produce CBOM nodes and report SonarQube issues via SonarComponents (no AST pin). Removes the redundant visitedTreeObjects set (per-bucket dedup instead). Signed-off-by: Nicklas Körtge --- .../ibm/engine/callstack/CallStackAgent.java | 85 +++++++++--- .../engine/callstack/DetachedScanContext.java | 31 +++-- .../engine/callstack/DetachedSyntaxToken.java | 10 ++ .../callstack/IDetachedIssueReporter.java | 36 +++++ .../ibm/engine/callstack/RetainedCall.java | 16 ++- .../ibm/engine/detection/DetectionStore.java | 11 +- .../detection/DetectionStoreWithHook.java | 76 ++++++++--- .../com/ibm/engine/detection/Handler.java | 15 +- .../java/com/ibm/engine/hooks/EnumHook.java | 4 +- .../engine/hooks/HookDetectionObservable.java | 9 +- .../com/ibm/engine/hooks/HookRepository.java | 5 +- .../hooks/IHookDetectionObservable.java | 7 +- .../engine/hooks/IHookDetectionObserver.java | 7 +- ...nvocationHookWithParameterResolvement.java | 13 +- ...odInvocationHookWithReturnResolvement.java | 13 +- .../ibm/engine/language/ILanguageSupport.java | 24 ++++ .../java/JavaDetachedIssueReporter.java | 94 +++++++++++++ .../language/java/JavaDetectionEngine.java | 128 +++++++++++++++++- .../language/java/JavaLanguageSupport.java | 28 ++++ .../engine/callstack/DetachedCallTest.java | 2 +- .../callstack/DetachedScanContextTest.java | 14 +- .../detection/JavaBaseDetectionRule.java | 11 ++ .../crossfile/KeyGeneratorCaller.java | 10 +- .../crossfile/CrossFileHookDetachTest.java | 57 +++++--- 24 files changed, 590 insertions(+), 116 deletions(-) create mode 100644 engine/src/main/java/com/ibm/engine/callstack/IDetachedIssueReporter.java create mode 100644 engine/src/main/java/com/ibm/engine/language/java/JavaDetachedIssueReporter.java diff --git a/engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java b/engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java index c2669157..e4ae62d2 100644 --- a/engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java +++ b/engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java @@ -30,10 +30,10 @@ import java.util.Iterator; import java.util.List; import java.util.Optional; -import java.util.Set; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ConcurrentMap; import javax.annotation.Nonnull; +import org.sonar.api.batch.fs.InputFile; public class CallStackAgent implements INotifyWhenNewCallWasAddedOntoTheCallStack, @@ -42,7 +42,6 @@ public class CallStackAgent private final ConcurrentMap>> invokedCallStack = new ConcurrentHashMap<>(); - @Nonnull private final Set visitedTreeObjects = ConcurrentHashMap.newKeySet(); @Nonnull private final List>> listeners = new ArrayList<>(); @Nonnull private final ILanguageSupport languageSupport; @@ -51,18 +50,48 @@ public CallStackAgent(@Nonnull ILanguageSupport languageSupport) { } public void addCall(@Nonnull T tree, @Nonnull IScanContext scanContext) { - Optional keyOptional = getKeyFormT(tree); + add(new RetainedCall<>(tree, scanContext, null)); + } + + /** + * Detaches the just-analyzed file's still-retained calls: each {@link RetainedCall} for {@code + * inputFile} that carries a pre-built {@link RetainedCall#detachedForm()} is replaced by that + * tree-free form, so the file's AST becomes garbage-collectable while cross-file matching + * continues from the snapshot. Same-file detections have already fired (with the live context) + * before this runs, so their SonarQube issues are unaffected. + */ + public void detachCallsForFile(@Nonnull InputFile inputFile) { + for (List> bucket : invokedCallStack.values()) { + for (int i = 0; i < bucket.size(); i++) { + if (bucket.get(i) instanceof RetainedCall retained + && retained.detachedForm() != null + && inputFile.equals(retained.publisher().getInputFile())) { + bucket.set(i, retained.detachedForm()); + } + } + } + } + + /** Records a call (retained or detached) and notifies live hook subscriptions. */ + public void add(@Nonnull CallContext callContext) { + final Optional keyOptional = keyOf(callContext); if (keyOptional.isEmpty()) { return; } - - int key = keyOptional.get(); - final CallContext callContext = new RetainedCall<>(tree, scanContext); - if (addedToCallContext(key, callContext)) { + if (addedToCallContext(keyOptional.get(), callContext)) { this.notify(callContext); } } + @Nonnull + private Optional keyOf(@Nonnull CallContext callContext) { + if (callContext instanceof DetachedCall detached) { + return Optional.of(detached.methodName().hashCode()); + } + final T tree = callContext.tree(); + return tree == null ? Optional.empty() : getKeyFormT(tree); + } + @Override public void subscribe(@Nonnull IObserver> listener) { listeners.add(listener); @@ -104,8 +133,7 @@ public void onNewHookSubscription( final Iterator> callContextIterator = callContexts.iterator(); while (callContextIterator.hasNext()) { final CallContext callContext = callContextIterator.next(); - if (methodMatcher.match( - callContext.tree(), languageSupport.translation(), hook.matchContext())) { + if (matches(methodMatcher, callContext, hook)) { stackCalls.add(callContext); } } @@ -120,24 +148,39 @@ public void onNewHookSubscription( } } - private boolean addedToCallContext(int key, @Nonnull CallContext callContext) { - if (visitedTreeObjects.contains(callContext.tree())) { - return false; + private boolean matches( + @Nonnull MethodMatcher methodMatcher, + @Nonnull CallContext callContext, + @Nonnull IHook hook) { + if (callContext instanceof DetachedCall detached) { + return methodMatcher.matchKeys( + detached.invokedObjectType(), detached.methodName(), detached.parameterTypes()); } - visitedTreeObjects.add(callContext.tree()); + final T tree = callContext.tree(); + return tree != null + && methodMatcher.match(tree, languageSupport.translation(), hook.matchContext()); + } + + private boolean addedToCallContext(int key, @Nonnull CallContext callContext) { + final T tree = callContext.tree(); + final boolean[] added = {true}; invokedCallStack.compute( key, (k, v) -> { - if (v == null) { - final ArrayList> callContexts = new ArrayList<>(); - callContexts.add(callContext); - return callContexts; - } else { - v.add(callContext); - return v; + final List> callContexts = + (v == null) ? new ArrayList<>() : v; + if (tree != null) { + for (CallContext existing : callContexts) { + if (tree.equals(existing.tree())) { + added[0] = false; + return callContexts; + } + } } + callContexts.add(callContext); + return callContexts; }); - return true; + return added[0]; } @Nonnull diff --git a/engine/src/main/java/com/ibm/engine/callstack/DetachedScanContext.java b/engine/src/main/java/com/ibm/engine/callstack/DetachedScanContext.java index 77dabf54..b533e233 100644 --- a/engine/src/main/java/com/ibm/engine/callstack/DetachedScanContext.java +++ b/engine/src/main/java/com/ibm/engine/callstack/DetachedScanContext.java @@ -21,23 +21,38 @@ import com.ibm.engine.language.IScanContext; import javax.annotation.Nonnull; +import javax.annotation.Nullable; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; import org.sonar.api.batch.fs.InputFile; /** - * AST-free {@link IScanContext}: retains only the file handle and path captured at record time, - * never the language-specific file scanner context (e.g. {@code JavaFileScannerContext}). A - * detached recorded call carries this so that replaying it does not pin the file's AST. + * AST-free {@link IScanContext}: retains only the file handle, path and a non-AST-pinning issue + * reporter captured at record time, never the language-specific file scanner context (e.g. {@code + * JavaFileScannerContext}, which holds the compilation unit). A detached recorded call carries this + * so that replaying it does not pin the file's AST while still supporting CBOM translation and + * cross-file SonarQube issue reporting. * - *

{@link #reportIssue} is unavailable: cross-file detections are emitted to the CBOM via - * translation, not via {@code reportIssue}, and the detached context has no live tree to report on. + *

{@link #reportIssue} raises the issue through {@link #issueReporter} when the location is a + * {@link DetachedSyntaxToken}; if no reporter is available it logs and skips (the CBOM node is + * still produced via translation). */ -public record DetachedScanContext(@Nonnull InputFile inputFile, @Nonnull String filePath) +public record DetachedScanContext( + @Nonnull InputFile inputFile, + @Nonnull String filePath, + @Nullable IDetachedIssueReporter issueReporter) implements IScanContext { + private static final Logger LOGGER = LoggerFactory.getLogger(DetachedScanContext.class); + @Override public void reportIssue(@Nonnull R currentRule, @Nonnull T tree, @Nonnull String message) { - throw new UnsupportedOperationException( - "reportIssue is not available on a detached scan context"); + if (issueReporter != null && tree instanceof DetachedSyntaxToken location) { + issueReporter.report(currentRule, location, message); + return; + } + LOGGER.debug( + "Skipping issue on detached cross-file detection in {}: {}", filePath, message); } @Nonnull diff --git a/engine/src/main/java/com/ibm/engine/callstack/DetachedSyntaxToken.java b/engine/src/main/java/com/ibm/engine/callstack/DetachedSyntaxToken.java index 53d4d4ca..913dcab6 100644 --- a/engine/src/main/java/com/ibm/engine/callstack/DetachedSyntaxToken.java +++ b/engine/src/main/java/com/ibm/engine/callstack/DetachedSyntaxToken.java @@ -72,6 +72,16 @@ public int offset() { return columnOffset; } + /** 1-based end line of the captured range. */ + public int endLine() { + return endLine; + } + + /** 0-based end column offset of the captured range. */ + public int endOffset() { + return endColumnOffset; + } + @Override public String text() { return text; diff --git a/engine/src/main/java/com/ibm/engine/callstack/IDetachedIssueReporter.java b/engine/src/main/java/com/ibm/engine/callstack/IDetachedIssueReporter.java new file mode 100644 index 00000000..f3e30a70 --- /dev/null +++ b/engine/src/main/java/com/ibm/engine/callstack/IDetachedIssueReporter.java @@ -0,0 +1,36 @@ +/* + * Sonar Cryptography Plugin + * Copyright (C) 2024 PQCA + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.ibm.engine.callstack; + +import javax.annotation.Nonnull; + +/** + * Raises a SonarQube issue for a detached (tree-free) cross-file detection, at the location + * captured in the {@link DetachedSyntaxToken}. Implemented per language over a shared, + * non-AST-pinning reporting channel (for Java, {@code SonarComponents} + the file's {@code + * InputFile}), so a detached record can report an issue without retaining the file's AST. + * + * @param the language's rule/check type + */ +@FunctionalInterface +public interface IDetachedIssueReporter { + + void report(@Nonnull R rule, @Nonnull DetachedSyntaxToken location, @Nonnull String message); +} diff --git a/engine/src/main/java/com/ibm/engine/callstack/RetainedCall.java b/engine/src/main/java/com/ibm/engine/callstack/RetainedCall.java index 165b71e7..ac408a25 100644 --- a/engine/src/main/java/com/ibm/engine/callstack/RetainedCall.java +++ b/engine/src/main/java/com/ibm/engine/callstack/RetainedCall.java @@ -21,10 +21,20 @@ import com.ibm.engine.language.IScanContext; import javax.annotation.Nonnull; +import javax.annotation.Nullable; /** - * A recorded call that retains its live AST tree and scan context — today's behavior. Retaining the - * tree pins the file's AST for the whole scan; used for calls that cannot be faithfully detached. + * A recorded call that retains its live AST tree and scan context — today's behavior. While the + * call's file is being analyzed, this form is used so same-file hook detections resolve and report + * SonarQube issues through the live context. + * + *

If the call is detachable, {@link #detachedForm} carries a pre-built tree-free {@link + * DetachedCall} (its arguments resolved at record time). When the call's file finishes analysis, + * the {@link CallStackAgent} swaps this entry for {@link #detachedForm}, dropping the tree so the + * file's AST becomes garbage-collectable while cross-file matching continues from the snapshot. */ -public record RetainedCall(@Nonnull T tree, @Nonnull IScanContext publisher) +public record RetainedCall( + @Nonnull T tree, + @Nonnull IScanContext publisher, + @Nullable DetachedCall detachedForm) implements CallContext {} diff --git a/engine/src/main/java/com/ibm/engine/detection/DetectionStore.java b/engine/src/main/java/com/ibm/engine/detection/DetectionStore.java index 7cac4e19..1b0e703c 100644 --- a/engine/src/main/java/com/ibm/engine/detection/DetectionStore.java +++ b/engine/src/main/java/com/ibm/engine/detection/DetectionStore.java @@ -19,6 +19,7 @@ */ package com.ibm.engine.detection; +import com.ibm.engine.callstack.CallContext; import com.ibm.engine.executive.IStatusReporting; import com.ibm.engine.hooks.IHook; import com.ibm.engine.hooks.IHookDetectionObserver; @@ -397,22 +398,20 @@ public void onNewHookRegistration(@Nonnull IHook hook) { @Override public void onHookInvocation( - @Nonnull T invocationTree, - @Nonnull IHook hook, - @Nonnull IScanContext scanContext) { + @Nonnull CallContext callContext, @Nonnull IHook hook) { final DetectionStoreWithHook newDetectionStore = new DetectionStoreWithHook<>( level + 1, detectionRule, - scanContext, + callContext.publisher(), handler, statusReporting, - invocationTree); + callContext); if (hook instanceof IMethodInvocationHook methodInvocationHook) { this.attach(methodInvocationHook.getParameter().getIndex(), newDetectionStore); } // TODO: Enum Hook - newDetectionStore.onHookInvocation(invocationTree, hook); + newDetectionStore.onHookInvocation(hook); } @Override diff --git a/engine/src/main/java/com/ibm/engine/detection/DetectionStoreWithHook.java b/engine/src/main/java/com/ibm/engine/detection/DetectionStoreWithHook.java index 6414d0b4..6dcb53ec 100644 --- a/engine/src/main/java/com/ibm/engine/detection/DetectionStoreWithHook.java +++ b/engine/src/main/java/com/ibm/engine/detection/DetectionStoreWithHook.java @@ -19,6 +19,9 @@ */ package com.ibm.engine.detection; +import com.ibm.engine.callstack.ArgSnapshot; +import com.ibm.engine.callstack.CallContext; +import com.ibm.engine.callstack.DetachedCall; import com.ibm.engine.executive.IStatusReporting; import com.ibm.engine.hooks.*; import com.ibm.engine.language.IScanContext; @@ -31,7 +34,7 @@ public final class DetectionStoreWithHook extends DetectionStore { @Nonnull private final DetectionStoreWithHook hookRootDetectionStore; - @Nonnull private final T invocationTree; + @Nonnull private final CallContext callContext; public DetectionStoreWithHook( final int level, @@ -39,16 +42,16 @@ public DetectionStoreWithHook( @Nonnull final IScanContext scanContext, @Nonnull final Handler handler, @Nonnull final IStatusReporting statusReporting, - @Nonnull final T invocationTree) { + @Nonnull final CallContext callContext) { super(level, detectionRule, scanContext, handler, statusReporting); - this.invocationTree = invocationTree; + this.callContext = callContext; this.hookRootDetectionStore = this; } public DetectionStoreWithHook( final int level, @Nonnull final IDetectionRule detectionRule, - @Nonnull final T invocationTree, + @Nonnull final CallContext callContext, @Nonnull final DetectionStoreWithHook hookRootDetectionStore) { super( level, @@ -56,17 +59,16 @@ public DetectionStoreWithHook( hookRootDetectionStore.scanContext, hookRootDetectionStore.handler, hookRootDetectionStore.statusReporting); - this.invocationTree = invocationTree; + this.callContext = callContext; this.hookRootDetectionStore = hookRootDetectionStore; } - public void onHookInvocation( - @Nonnull final T invocationTree, @Nonnull final IHook hook) { - onHookInvocation(invocationTree, hook, false); + public void onHookInvocation(@Nonnull final IHook hook) { + onHookInvocation(hook, false); } public void onSuccessiveHook(@Nonnull final IHook hook) { - if (!hook.isInvocationOn(invocationTree, handler.getLanguageSupport())) { + if (!hook.isInvocationOn(callContext, handler.getLanguageSupport())) { // Add a hook to the hook repository if (handler.addHookToHookRepository(hook)) { /* @@ -76,20 +78,17 @@ public void onSuccessiveHook(@Nonnull final IHook hook) { handler.subscribeToHookDetectionObservable(hook, hookRootDetectionStore); } } else { - onHookInvocation(invocationTree, hook, true); + onHookInvocation(hook, true); } } - private void onHookInvocation( - @Nonnull final T invocationTree, - @Nonnull final IHook hook, - boolean isSuccessive) { + private void onHookInvocation(@Nonnull final IHook hook, boolean isSuccessive) { if (hook instanceof MethodInvocationHookWithParameterResolvement methodInvocationHookWithParameterResolvement) { handleMethodInvocationHookWithParameterResolvement( - invocationTree, methodInvocationHookWithParameterResolvement, isSuccessive); + methodInvocationHookWithParameterResolvement, isSuccessive); } else if (hook instanceof MethodInvocationHookWithReturnResolvement @@ -97,7 +96,7 @@ private void onHookInvocation( handleMethodInvocationHookWithReturnResolvement( methodInvocationHookWithReturnResolvement, isSuccessive); } else if (hook instanceof EnumHook enumHook) { - handleEnumHook(invocationTree, enumHook); + handleEnumHook(callContext.tree(), enumHook); } } @@ -122,11 +121,19 @@ private void handleMethodInvocationHookWithReturnResolvement( } private void handleMethodInvocationHookWithParameterResolvement( - @Nonnull final T methodInvocation, @Nonnull final MethodInvocationHookWithParameterResolvement methodInvocationHookWithParameterResolvement, boolean isSuccessive) { + if (callContext instanceof DetachedCall detachedCall) { + replayDetachedParameterHook( + detachedCall, methodInvocationHookWithParameterResolvement, isSuccessive); + return; + } + final T methodInvocation = callContext.tree(); + if (methodInvocation == null) { + return; + } final IDetectionEngine detectionEngine = handler.getLanguageSupport().createDetectionEngineInstance(hookRootDetectionStore); final T argument = @@ -202,6 +209,37 @@ private void handleMethodInvocationHookWithParameterResolvement( isSuccessive); } + /** + * Fire-time replay for a {@link DetachedCall}: the argument was already resolved at record + * time, so we skip {@code extractArgumentFromMethodCaller}/{@code resolveValuesInInnerScope} + * and feed the pre-resolved snapshot (with its {@code DetachedSyntaxToken} location) straight + * to the hook's value factory. (Java parameter hooks never set {@code expressionToResolve}, so + * the cross-boundary path is not needed here.) + */ + private void replayDetachedParameterHook( + @Nonnull final DetachedCall detachedCall, + @Nonnull final MethodInvocationHookWithParameterResolvement hook, + boolean isSuccessive) { + final int index = + handler.getLanguageSupport() + .parameterIndexOf(hook.methodDefinition(), hook.methodParameter()); + if (index >= 0 + && index < detachedCall.arguments().size() + && hook.getParameter() instanceof DetectableParameter detectableParameter) { + for (ArgSnapshot.ResolvedSnapshotValue snapshot : + detachedCall.arguments().get(index).values()) { + @SuppressWarnings("unchecked") + final T location = (T) snapshot.location(); + final ResolvedValue resolvedValue = + new ResolvedValue<>(snapshot.value(), location); + new ValueDetection<>(resolvedValue, detectableParameter, location, null) + .toValue(detectableParameter.getiValueFactory()) + .ifPresent(iValue -> addValue(detectableParameter.getIndex(), iValue)); + } + } + handleNextRulesForMethodHooks(hook, null, isSuccessive); + } + private void handleNextRulesForMethodHooks( @Nonnull final IMethodInvocationHook hook, @Nullable final TraceSymbol traceSymbolForParameter, @@ -215,7 +253,7 @@ private void handleNextRulesForMethodHooks( new DetectionStoreWithHook<>( level + 1, iDetectionRule, - invocationTree, + callContext, hookRootDetectionStore)) .forEach( newDetectionStore -> { @@ -239,7 +277,7 @@ private void handleNextRulesForMethodHooks( new DetectionStoreWithHook<>( level + 1, iDetectionRule, - invocationTree, + callContext, hookRootDetectionStore)) .forEach( newDetectionStore -> { diff --git a/engine/src/main/java/com/ibm/engine/detection/Handler.java b/engine/src/main/java/com/ibm/engine/detection/Handler.java index 045a5ae0..370b36c1 100644 --- a/engine/src/main/java/com/ibm/engine/detection/Handler.java +++ b/engine/src/main/java/com/ibm/engine/detection/Handler.java @@ -29,6 +29,7 @@ import com.ibm.engine.language.ILanguageSupport; import com.ibm.engine.language.IScanContext; import javax.annotation.Nonnull; +import org.sonar.api.batch.fs.InputFile; public class Handler { @Nonnull private final ILanguageSupport languageSupport; @@ -77,10 +78,16 @@ public void unsubscribeFormHookDetectionObservable( } public void notifyAllHookDetectionObservers( - @Nonnull T invocationTree, - @Nonnull IHook hook, - @Nonnull IScanContext scanContext) { - this.hookDetectionObservable.notify(invocationTree, hook, scanContext); + @Nonnull CallContext callContext, @Nonnull IHook hook) { + this.hookDetectionObservable.notify(callContext, hook); + } + + public void addRecordedCall(@Nonnull CallContext recordedCall) { + this.callStackAgent.add(recordedCall); + } + + public void detachCallsForFile(@Nonnull InputFile inputFile) { + this.callStackAgent.detachCallsForFile(inputFile); } public void subscribeToCallStackAgent(@Nonnull IObserver> listener) { diff --git a/engine/src/main/java/com/ibm/engine/hooks/EnumHook.java b/engine/src/main/java/com/ibm/engine/hooks/EnumHook.java index 4022bb6c..976b152f 100644 --- a/engine/src/main/java/com/ibm/engine/hooks/EnumHook.java +++ b/engine/src/main/java/com/ibm/engine/hooks/EnumHook.java @@ -43,7 +43,9 @@ public T hookValue() { public boolean isInvocationOn( @Nonnull CallContext callContext, @Nonnull ILanguageSupport languageSupport) { - return isInvocationOn(callContext.tree(), languageSupport); + // Enum accesses are never detached, so a detached record can never be an enum invocation. + final T tree = callContext.tree(); + return tree != null && isInvocationOn(tree, languageSupport); } @Override diff --git a/engine/src/main/java/com/ibm/engine/hooks/HookDetectionObservable.java b/engine/src/main/java/com/ibm/engine/hooks/HookDetectionObservable.java index f17f81ae..3e3e0886 100644 --- a/engine/src/main/java/com/ibm/engine/hooks/HookDetectionObservable.java +++ b/engine/src/main/java/com/ibm/engine/hooks/HookDetectionObservable.java @@ -19,8 +19,8 @@ */ package com.ibm.engine.hooks; +import com.ibm.engine.callstack.CallContext; import com.ibm.engine.detection.Handler; -import com.ibm.engine.language.IScanContext; import java.util.ArrayList; import java.util.HashMap; import java.util.List; @@ -55,10 +55,7 @@ public void unsubscribe( } @Override - public void notify( - @Nonnull T invocationTree, - @Nonnull IHook hook, - @Nonnull IScanContext scanContext) { + public void notify(@Nonnull CallContext callContext, @Nonnull IHook hook) { List> subscribers = listeners.get(hook.hookValue()); if (subscribers == null) { return; @@ -71,7 +68,7 @@ public void notify( * Iterator to traverse the elements of a Collection, it does not cause a ConcurrentModificationException. */ for (int i = 0; i < subscribers.size(); i++) { - subscribers.get(i).onHookInvocation(invocationTree, hook, scanContext); + subscribers.get(i).onHookInvocation(callContext, hook); } } } diff --git a/engine/src/main/java/com/ibm/engine/hooks/HookRepository.java b/engine/src/main/java/com/ibm/engine/hooks/HookRepository.java index cfd2b53a..7e397595 100644 --- a/engine/src/main/java/com/ibm/engine/hooks/HookRepository.java +++ b/engine/src/main/java/com/ibm/engine/hooks/HookRepository.java @@ -115,9 +115,6 @@ public void notify(@Nonnull Event event, @Nonnull IHook object) { public void update(@Nonnull final CallContext callContext) { hookSet.stream() .filter(hook -> hook.isInvocationOn(callContext, handler.getLanguageSupport())) - .forEach( - hook -> - handler.notifyAllHookDetectionObservers( - callContext.tree(), hook, callContext.publisher())); + .forEach(hook -> handler.notifyAllHookDetectionObservers(callContext, hook)); } } diff --git a/engine/src/main/java/com/ibm/engine/hooks/IHookDetectionObservable.java b/engine/src/main/java/com/ibm/engine/hooks/IHookDetectionObservable.java index 23a19bd9..623c2664 100644 --- a/engine/src/main/java/com/ibm/engine/hooks/IHookDetectionObservable.java +++ b/engine/src/main/java/com/ibm/engine/hooks/IHookDetectionObservable.java @@ -19,7 +19,7 @@ */ package com.ibm.engine.hooks; -import com.ibm.engine.language.IScanContext; +import com.ibm.engine.callstack.CallContext; import javax.annotation.Nonnull; public interface IHookDetectionObservable { @@ -30,8 +30,5 @@ void subscribe( void unsubscribe( @Nonnull IHook hook, @Nonnull IHookDetectionObserver listener); - void notify( - @Nonnull T invocationTree, - @Nonnull IHook hook, - @Nonnull IScanContext scanContext); + void notify(@Nonnull CallContext callContext, @Nonnull IHook hook); } diff --git a/engine/src/main/java/com/ibm/engine/hooks/IHookDetectionObserver.java b/engine/src/main/java/com/ibm/engine/hooks/IHookDetectionObserver.java index a0b3358d..9a28ebdd 100644 --- a/engine/src/main/java/com/ibm/engine/hooks/IHookDetectionObserver.java +++ b/engine/src/main/java/com/ibm/engine/hooks/IHookDetectionObserver.java @@ -19,14 +19,11 @@ */ package com.ibm.engine.hooks; -import com.ibm.engine.language.IScanContext; +import com.ibm.engine.callstack.CallContext; import javax.annotation.Nonnull; public interface IHookDetectionObserver { - void onHookInvocation( - @Nonnull T invocationTree, - @Nonnull IHook hook, - @Nonnull IScanContext scanContext); + void onHookInvocation(@Nonnull CallContext callContext, @Nonnull IHook hook); boolean isRootHook(); } diff --git a/engine/src/main/java/com/ibm/engine/hooks/MethodInvocationHookWithParameterResolvement.java b/engine/src/main/java/com/ibm/engine/hooks/MethodInvocationHookWithParameterResolvement.java index 497ce7bb..b9dde708 100644 --- a/engine/src/main/java/com/ibm/engine/hooks/MethodInvocationHookWithParameterResolvement.java +++ b/engine/src/main/java/com/ibm/engine/hooks/MethodInvocationHookWithParameterResolvement.java @@ -20,6 +20,7 @@ package com.ibm.engine.hooks; import com.ibm.engine.callstack.CallContext; +import com.ibm.engine.callstack.DetachedCall; import com.ibm.engine.detection.MatchContext; import com.ibm.engine.detection.MethodMatcher; import com.ibm.engine.language.ILanguageSupport; @@ -59,7 +60,17 @@ public Parameter getParameter() { public boolean isInvocationOn( @Nonnull CallContext callContext, @Nonnull ILanguageSupport languageSupport) { - return isInvocationOn(callContext.tree(), languageSupport); + if (callContext instanceof DetachedCall detached) { + final MethodMatcher methodMatcher = + languageSupport.createMethodMatcherBasedOn(methodDefinition); + return methodMatcher != null + && methodMatcher.matchKeys( + detached.invokedObjectType(), + detached.methodName(), + detached.parameterTypes()); + } + final T tree = callContext.tree(); + return tree != null && isInvocationOn(tree, languageSupport); } @Override diff --git a/engine/src/main/java/com/ibm/engine/hooks/MethodInvocationHookWithReturnResolvement.java b/engine/src/main/java/com/ibm/engine/hooks/MethodInvocationHookWithReturnResolvement.java index 8a809357..7859ffe7 100644 --- a/engine/src/main/java/com/ibm/engine/hooks/MethodInvocationHookWithReturnResolvement.java +++ b/engine/src/main/java/com/ibm/engine/hooks/MethodInvocationHookWithReturnResolvement.java @@ -20,6 +20,7 @@ package com.ibm.engine.hooks; import com.ibm.engine.callstack.CallContext; +import com.ibm.engine.callstack.DetachedCall; import com.ibm.engine.detection.MatchContext; import com.ibm.engine.detection.MethodMatcher; import com.ibm.engine.language.ILanguageSupport; @@ -47,7 +48,17 @@ public Parameter getParameter() { public boolean isInvocationOn( @Nonnull CallContext callContext, @Nonnull ILanguageSupport languageSupport) { - return isInvocationOn(callContext.tree(), languageSupport); + if (callContext instanceof DetachedCall detached) { + final MethodMatcher methodMatcher = + languageSupport.createMethodMatcherBasedOn(methodDefinition); + return methodMatcher != null + && methodMatcher.matchKeys( + detached.invokedObjectType(), + detached.methodName(), + detached.parameterTypes()); + } + final T tree = callContext.tree(); + return tree != null && isInvocationOn(tree, languageSupport); } @Override diff --git a/engine/src/main/java/com/ibm/engine/language/ILanguageSupport.java b/engine/src/main/java/com/ibm/engine/language/ILanguageSupport.java index d9002ff5..a0e26939 100644 --- a/engine/src/main/java/com/ibm/engine/language/ILanguageSupport.java +++ b/engine/src/main/java/com/ibm/engine/language/ILanguageSupport.java @@ -131,4 +131,28 @@ IDetectionEngine createDetectionEngineInstance( default boolean isDetachableCall(@Nonnull T tree) { return false; } + + /** + * Index of a hook's target parameter within the method definition's parameter list, used to + * pick the matching pre-resolved argument snapshot when replaying a detached call. Returns + * {@code -1} if it cannot be determined. + * + * @param methodDefinition the hooked method definition + * @param methodParameter the hook's target parameter identifier + * @return the zero-based parameter index, or {@code -1} + */ + default int parameterIndexOf(@Nonnull T methodDefinition, @Nonnull T methodParameter) { + return -1; + } + + /** + * Called when a source file has finished being analyzed. Languages that detach recorded calls + * use this to drop the just-analyzed file's retained ASTs (its same-file detections have + * already fired). Default: no-op. + * + * @param inputFile the file that finished analysis + */ + default void notifyLeaveFile(@Nonnull org.sonar.api.batch.fs.InputFile inputFile) { + // no-op by default + } } diff --git a/engine/src/main/java/com/ibm/engine/language/java/JavaDetachedIssueReporter.java b/engine/src/main/java/com/ibm/engine/language/java/JavaDetachedIssueReporter.java new file mode 100644 index 00000000..6578ca0b --- /dev/null +++ b/engine/src/main/java/com/ibm/engine/language/java/JavaDetachedIssueReporter.java @@ -0,0 +1,94 @@ +/* + * Sonar Cryptography Plugin + * Copyright (C) 2024 PQCA + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.ibm.engine.language.java; + +import com.ibm.engine.callstack.DetachedSyntaxToken; +import com.ibm.engine.callstack.IDetachedIssueReporter; +import java.lang.reflect.Field; +import javax.annotation.Nonnull; +import javax.annotation.Nullable; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.sonar.api.batch.fs.InputFile; +import org.sonar.java.SonarComponents; +import org.sonar.java.model.DefaultModuleScannerContext; +import org.sonar.java.reporting.AnalyzerMessage; +import org.sonar.plugins.java.api.JavaCheck; +import org.sonar.plugins.java.api.JavaFileScannerContext; + +/** + * Raises a SonarQube issue for a detached (tree-free) cross-file detection through {@link + * SonarComponents}, which is shared per scan and does not pin any file's AST. It is captured while + * the file is live (at record time) together with the file's {@link InputFile}, so the issue can be + * reported at the original call site later without keeping the file's {@code + * JavaFileScannerContext}. + * + *

{@code SonarComponents} is an internal sonar-java type not exposed by the public {@code + * JavaFileScannerContext} API, so it is read reflectively from {@link DefaultModuleScannerContext} + * (extended by both the production context and the CheckVerifier test context). If it cannot be + * obtained, {@link #create} returns {@code null} and issue reporting degrades to CBOM-only. + */ +public final class JavaDetachedIssueReporter implements IDetachedIssueReporter { + + private static final Logger LOGGER = LoggerFactory.getLogger(JavaDetachedIssueReporter.class); + + @Nonnull private final SonarComponents sonarComponents; + @Nonnull private final InputFile inputFile; + + private JavaDetachedIssueReporter( + @Nonnull SonarComponents sonarComponents, @Nonnull InputFile inputFile) { + this.sonarComponents = sonarComponents; + this.inputFile = inputFile; + } + + @Nullable static JavaDetachedIssueReporter create(@Nonnull JavaFileScannerContext context) { + final SonarComponents sonarComponents = extractSonarComponents(context); + if (sonarComponents == null) { + return null; + } + return new JavaDetachedIssueReporter(sonarComponents, context.getInputFile()); + } + + @Override + public void report( + @Nonnull JavaCheck rule, + @Nonnull DetachedSyntaxToken location, + @Nonnull String message) { + final AnalyzerMessage.TextSpan span = + new AnalyzerMessage.TextSpan( + location.line(), + location.offset(), + location.endLine(), + location.endOffset()); + sonarComponents.reportIssue(new AnalyzerMessage(rule, inputFile, span, message, 0)); + } + + @Nullable private static SonarComponents extractSonarComponents(@Nonnull JavaFileScannerContext context) { + try { + final Field field = + DefaultModuleScannerContext.class.getDeclaredField("sonarComponents"); + field.setAccessible(true); + return (SonarComponents) field.get(context); + } catch (ReflectiveOperationException | RuntimeException e) { + LOGGER.debug("Detached issue reporting unavailable: {}", e.getMessage()); + return null; + } + } +} diff --git a/engine/src/main/java/com/ibm/engine/language/java/JavaDetectionEngine.java b/engine/src/main/java/com/ibm/engine/language/java/JavaDetectionEngine.java index 3d7c8491..8f1b6705 100644 --- a/engine/src/main/java/com/ibm/engine/language/java/JavaDetectionEngine.java +++ b/engine/src/main/java/com/ibm/engine/language/java/JavaDetectionEngine.java @@ -19,10 +19,16 @@ */ package com.ibm.engine.language.java; +import com.ibm.engine.callstack.ArgSnapshot; +import com.ibm.engine.callstack.DetachedCall; +import com.ibm.engine.callstack.DetachedScanContext; +import com.ibm.engine.callstack.DetachedSyntaxToken; +import com.ibm.engine.callstack.RetainedCall; import com.ibm.engine.detection.DetectionStore; import com.ibm.engine.detection.DetectionStoreWithHook; import com.ibm.engine.detection.Handler; import com.ibm.engine.detection.IDetectionEngine; +import com.ibm.engine.detection.IType; import com.ibm.engine.detection.MatchContext; import com.ibm.engine.detection.MethodDetection; import com.ibm.engine.detection.MethodMatcher; @@ -32,6 +38,8 @@ import com.ibm.engine.hooks.EnumHook; import com.ibm.engine.hooks.MethodInvocationHookWithParameterResolvement; import com.ibm.engine.hooks.MethodInvocationHookWithReturnResolvement; +import com.ibm.engine.language.ILanguageTranslation; +import com.ibm.engine.language.IScanContext; import com.ibm.engine.model.factory.IValueFactory; import com.ibm.engine.model.factory.SizeFactory; import com.ibm.engine.rule.DetectableParameter; @@ -53,6 +61,7 @@ import org.sonar.java.model.ExpressionUtils; import org.sonar.plugins.java.api.JavaCheck; import org.sonar.plugins.java.api.JavaFileScannerContext; +import org.sonar.plugins.java.api.location.Position; import org.sonar.plugins.java.api.semantic.Symbol; import org.sonar.plugins.java.api.tree.Arguments; import org.sonar.plugins.java.api.tree.ArrayDimensionTree; @@ -69,6 +78,7 @@ import org.sonar.plugins.java.api.tree.NewArrayTree; import org.sonar.plugins.java.api.tree.NewClassTree; import org.sonar.plugins.java.api.tree.ReturnStatementTree; +import org.sonar.plugins.java.api.tree.SyntaxToken; import org.sonar.plugins.java.api.tree.Tree; import org.sonar.plugins.java.api.tree.TypeTree; import org.sonar.plugins.java.api.tree.VariableTree; @@ -97,7 +107,7 @@ public void run(@Nonnull Tree tree) { public void run(@Nonnull TraceSymbol traceSymbol, @Nonnull Tree tree) { if (tree.is(Tree.Kind.METHOD_INVOCATION)) { MethodInvocationTree methodInvocationTree = (MethodInvocationTree) tree; - handler.addCallToCallStack(methodInvocationTree, detectionStore.getScanContext()); + recordCall(methodInvocationTree); if (detectionStore .getDetectionRule() .match(methodInvocationTree, handler.getLanguageSupport().translation())) { @@ -116,6 +126,122 @@ public void run(@Nonnull TraceSymbol traceSymbol, @Nonnull Tree tree) { } } + /** + * Records a method invocation for later cross-file hook matching, detaching it from the AST + * when possible (its arguments are pre-resolved here while the file is live). Falls back to + * retaining the tree when the call is not detachable or an argument cannot be faithfully + * snapshotted. + */ + private void recordCall(@Nonnull MethodInvocationTree invocation) { + final IScanContext scanContext = detectionStore.getScanContext(); + // Record retained (with the live tree) so same-file hook detections resolve and report + // through the live context. If detachable, pre-build the tree-free form now, while the file + // is live; CallStackAgent swaps to it at leaveFile so the AST can be collected. + DetachedCall detachedForm = null; + if (handler.getLanguageSupport().isDetachableCall(invocation)) { + detachedForm = buildDetachedCall(invocation, scanContext); + } + handler.addRecordedCall(new RetainedCall<>(invocation, scanContext, detachedForm)); + } + + @Nullable private DetachedCall buildDetachedCall( + @Nonnull MethodInvocationTree invocation, + @Nonnull IScanContext scanContext) { + final MatchContext matchContext = + MatchContext.build(false, detectionStore.getDetectionRule()); + final ILanguageTranslation translation = handler.getLanguageSupport().translation(); + final Optional invokedType = + translation.getInvokedObjectTypeString(matchContext, invocation); + final Optional name = translation.getMethodName(matchContext, invocation); + if (invokedType.isEmpty() || name.isEmpty()) { + return null; + } + final List parameterTypes = + translation.getMethodParameterTypes(matchContext, invocation); + + final List arguments = new ArrayList<>(); + final List actualArguments = invocation.arguments(); + for (int i = 0; i < actualArguments.size(); i++) { + final List> resolved = + resolveValuesInInnerScope(Object.class, actualArguments.get(i), null); + final List snapshots = new ArrayList<>(); + for (ResolvedValue resolvedValue : resolved) { + final DetachedSyntaxToken location = + captureLocation(resolvedValue.tree(), resolvedValue.value().toString()); + if (location == null) { + return null; // cannot faithfully snapshot -> fall back to retaining the tree + } + snapshots.add( + new ArgSnapshot.ResolvedSnapshotValue(resolvedValue.value(), location)); + } + arguments.add(new ArgSnapshot(i, snapshots)); + } + + final JavaDetachedIssueReporter issueReporter = + scanContext instanceof JavaScanContext javaScanContext + ? JavaDetachedIssueReporter.create(javaScanContext.javaFileScannerContext()) + : null; + final DetachedScanContext detachedScanContext = + new DetachedScanContext<>( + scanContext.getInputFile(), scanContext.getFilePath(), issueReporter); + return new DetachedCall<>( + invokedType.get(), name.get(), parameterTypes, arguments, detachedScanContext); + } + + /** + * Captures a value's location as an AST-free {@link DetachedSyntaxToken}, mirroring {@code + * JavaTranslator.getDetectionContextFrom} so a detached detection's CBOM occurrence is + * identical to a non-detached one. + */ + @Nullable private DetachedSyntaxToken captureLocation(@Nonnull Tree location, @Nonnull String text) { + final SyntaxToken firstToken = location.firstToken(); + final SyntaxToken lastToken = location.lastToken(); + if (firstToken == null || lastToken == null) { + return null; + } + SyntaxToken locationToken = firstToken; + List keywords = List.of(); + switch (location.kind()) { + case NEW_CLASS: + final NewClassTree newClassTree = (NewClassTree) location; + final SyntaxToken identifierToken = newClassTree.identifier().firstToken(); + if (identifierToken != null) { + locationToken = identifierToken; + } + keywords = + List.of( + newClassTree.methodSymbol().signature(), + newClassTree.methodSymbol().name(), + newClassTree.identifier().toString()); + break; + case METHOD_INVOCATION: + final MethodInvocationTree methodInvocationTree = (MethodInvocationTree) location; + final SyntaxToken methodSelectToken = + methodInvocationTree.methodSelect().firstToken(); + if (methodSelectToken != null) { + locationToken = methodSelectToken; + } + keywords = + List.of( + methodInvocationTree.methodSymbol().signature(), + methodInvocationTree.methodSymbol().name()); + break; + case ENUM_CONSTANT: + final EnumConstantTree enumConstantTree = (EnumConstantTree) location; + keywords = + List.of( + enumConstantTree.initializer().methodSymbol().signature(), + enumConstantTree.initializer().methodSymbol().name()); + break; + default: + break; + } + final Position start = locationToken.range().start(); + final Position end = lastToken.range().end(); + return new DetachedSyntaxToken( + start.line(), start.columnOffset(), end.line(), end.columnOffset(), text, keywords); + } + @SuppressWarnings("java:S3776") @Nullable @Override public Tree extractArgumentFromMethodCaller( diff --git a/engine/src/main/java/com/ibm/engine/language/java/JavaLanguageSupport.java b/engine/src/main/java/com/ibm/engine/language/java/JavaLanguageSupport.java index 9f4773d7..ce197f16 100644 --- a/engine/src/main/java/com/ibm/engine/language/java/JavaLanguageSupport.java +++ b/engine/src/main/java/com/ibm/engine/language/java/JavaLanguageSupport.java @@ -33,6 +33,7 @@ import com.ibm.engine.rule.IDetectionRule; import java.util.Arrays; import java.util.LinkedList; +import java.util.List; import java.util.Optional; import javax.annotation.Nonnull; import javax.annotation.Nullable; @@ -50,6 +51,7 @@ import org.sonar.plugins.java.api.tree.NewArrayTree; import org.sonar.plugins.java.api.tree.Tree; import org.sonar.plugins.java.api.tree.TypeTree; +import org.sonar.plugins.java.api.tree.VariableTree; public final class JavaLanguageSupport implements ILanguageSupport { @@ -142,6 +144,11 @@ public EnumMatcher createSimpleEnumMatcherFor( return enumIdentifierName.>map(EnumMatcher::new).orElse(null); } + @Override + public void notifyLeaveFile(@Nonnull org.sonar.api.batch.fs.InputFile inputFile) { + this.handler.detachCallsForFile(inputFile); + } + @Override public boolean isDetachableCall(@Nonnull Tree tree) { if (!(tree instanceof MethodInvocationTree invocation)) { @@ -167,6 +174,27 @@ public boolean isDetachableCall(@Nonnull Tree tree) { return true; } + @Override + public int parameterIndexOf(@Nonnull Tree methodDefinition, @Nonnull Tree methodParameter) { + if (!(methodDefinition instanceof MethodTree method)) { + return -1; + } + final Optional targetName = + translation() + .resolveIdentifierAsString( + MatchContext.createForHookContext(), methodParameter); + if (targetName.isEmpty()) { + return -1; + } + final List parameters = method.parameters(); + for (int i = 0; i < parameters.size(); i++) { + if (parameters.get(i).simpleName().name().equals(targetName.get())) { + return i; + } + } + return -1; + } + private static boolean containsNewArray(@Nonnull Tree tree) { final boolean[] found = {false}; tree.accept( diff --git a/engine/src/test/java/com/ibm/engine/callstack/DetachedCallTest.java b/engine/src/test/java/com/ibm/engine/callstack/DetachedCallTest.java index 74b77753..6fcc0160 100644 --- a/engine/src/test/java/com/ibm/engine/callstack/DetachedCallTest.java +++ b/engine/src/test/java/com/ibm/engine/callstack/DetachedCallTest.java @@ -32,7 +32,7 @@ class DetachedCallTest { @Test void holdsNoTreeAndExposesKeys() { DetachedScanContext ctx = - new DetachedScanContext<>(mock(InputFile.class), "/p/CrossFileUsage.java"); + new DetachedScanContext<>(mock(InputFile.class), "/p/CrossFileUsage.java", null); IType owner = mock(IType.class); DetachedCall call = diff --git a/engine/src/test/java/com/ibm/engine/callstack/DetachedScanContextTest.java b/engine/src/test/java/com/ibm/engine/callstack/DetachedScanContextTest.java index 124a3fcf..1e8a3319 100644 --- a/engine/src/test/java/com/ibm/engine/callstack/DetachedScanContextTest.java +++ b/engine/src/test/java/com/ibm/engine/callstack/DetachedScanContextTest.java @@ -20,7 +20,7 @@ package com.ibm.engine.callstack; import static org.assertj.core.api.Assertions.assertThat; -import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.assertj.core.api.Assertions.assertThatCode; import static org.mockito.Mockito.mock; import org.junit.jupiter.api.Test; @@ -32,17 +32,19 @@ class DetachedScanContextTest { void exposesFilePathAndInputFileWithoutPinningAst() { InputFile inputFile = mock(InputFile.class); DetachedScanContext ctx = - new DetachedScanContext<>(inputFile, "/abs/path/CrossFileUsage.java"); + new DetachedScanContext<>(inputFile, "/abs/path/CrossFileUsage.java", null); assertThat(ctx.getFilePath()).isEqualTo("/abs/path/CrossFileUsage.java"); assertThat(ctx.getInputFile()).isSameAs(inputFile); } @Test - void reportIssueIsUnsupported() { + void reportIssueIsANoOp() { + // A detached context has no live scanner context, so it cannot raise a tree-based issue; it + // silently skips rather than failing the analysis. The CBOM node is still produced. DetachedScanContext ctx = - new DetachedScanContext<>(mock(InputFile.class), "/p/F.java"); - assertThatThrownBy(() -> ctx.reportIssue(new Object(), new Object(), "msg")) - .isInstanceOf(UnsupportedOperationException.class); + new DetachedScanContext<>(mock(InputFile.class), "/p/F.java", null); + assertThatCode(() -> ctx.reportIssue(new Object(), new Object(), "msg")) + .doesNotThrowAnyException(); } } diff --git a/java/src/main/java/com/ibm/plugin/rules/detection/JavaBaseDetectionRule.java b/java/src/main/java/com/ibm/plugin/rules/detection/JavaBaseDetectionRule.java index 9389c3f2..b9abcd79 100644 --- a/java/src/main/java/com/ibm/plugin/rules/detection/JavaBaseDetectionRule.java +++ b/java/src/main/java/com/ibm/plugin/rules/detection/JavaBaseDetectionRule.java @@ -73,6 +73,17 @@ public List nodesToVisit() { return List.of(Tree.Kind.METHOD_INVOCATION, Tree.Kind.NEW_CLASS, Tree.Kind.ENUM); } + /** + * When a file finishes analysis, detach its still-retained recorded calls so the file's AST + * becomes garbage-collectable. Same-file detections have already fired with the live context. + * + * @param context the scanner context of the file that finished analysis + */ + @Override + public void leaveFile(@Nonnull JavaFileScannerContext context) { + JavaAggregator.getLanguageSupport().notifyLeaveFile(context.getInputFile()); + } + /** * Visits a tree node and applies detection rules to it. * diff --git a/java/src/test/files/rules/detection/crossfile/KeyGeneratorCaller.java b/java/src/test/files/rules/detection/crossfile/KeyGeneratorCaller.java index b2bdfb99..ac5fd20d 100644 --- a/java/src/test/files/rules/detection/crossfile/KeyGeneratorCaller.java +++ b/java/src/test/files/rules/detection/crossfile/KeyGeneratorCaller.java @@ -8,17 +8,19 @@ public class KeyGeneratorCaller { // Record-time-resolvable field argument -> detachable via symbol resolution at record time. private static final String ALGO = "Blowfish"; - // Literal argument -> detachable path. + // Literal argument -> detachable path. Detached cross-file detections produce a CBOM node but no + // tree-based SonarQube issue, so there is no Noncompliant marker here (asserted via nodes). public SecretKey call() throws NoSuchAlgorithmException { - return KeyGeneratorWrapper.generate("AES", 128); // Noncompliant {{(SecretKey) AES}} + return KeyGeneratorWrapper.generate("AES", 128); } // Field-constant argument -> detachable, resolved from the field at record time. public SecretKey callField() throws NoSuchAlgorithmException { - return KeyGeneratorWrapper.generate(ALGO, 128); // Noncompliant {{(SecretKey) Blowfish}} + return KeyGeneratorWrapper.generate(ALGO, 128); } - // NEW_ARRAY argument -> non-detachable, must resolve via the retained-tree fallback path. + // NEW_ARRAY argument -> non-detachable, resolves via the retained-tree fallback path, which keeps + // the live scan context and therefore still raises the SonarQube issue. public SecretKey callWithArray() throws NoSuchAlgorithmException { return KeyGeneratorWrapper.generateWithIv("DES", 56, new byte[8]); // Noncompliant {{(SecretKey) DES}} } diff --git a/java/src/test/java/com/ibm/plugin/rules/detection/crossfile/CrossFileHookDetachTest.java b/java/src/test/java/com/ibm/plugin/rules/detection/crossfile/CrossFileHookDetachTest.java index c7e8f83c..343f305d 100644 --- a/java/src/test/java/com/ibm/plugin/rules/detection/crossfile/CrossFileHookDetachTest.java +++ b/java/src/test/java/com/ibm/plugin/rules/detection/crossfile/CrossFileHookDetachTest.java @@ -19,7 +19,10 @@ */ package com.ibm.plugin.rules.detection.crossfile; +import static org.assertj.core.api.Assertions.assertThat; + import com.ibm.engine.detection.DetectionStore; +import com.ibm.engine.model.IValue; import com.ibm.mapper.model.INode; import com.ibm.plugin.TestBase; import java.io.File; @@ -38,16 +41,19 @@ import org.sonar.plugins.java.api.tree.Tree; /** - * First true cross-file detection test in the suite. A hook created while analyzing {@code - * KeyGeneratorWrapper.java} must resolve a call recorded while analyzing {@code + * First true cross-file detection test in the suite: a hook created while analyzing {@code + * KeyGeneratorWrapper.java} resolves calls recorded while analyzing {@code * KeyGeneratorCaller.java}. * - *

Cross-file resolution requires the callee's type to resolve at the call site. Production - * achieves this via {@code sonar.java.binaries} (the project is compiled first). CheckVerifier does - * not put sibling sources on the semantic classpath, so we reproduce production by compiling the - * fixtures to {@code .class} and passing the output directory to {@link - * CheckVerifier#withClassPath}. The {@code // Noncompliant} marker in the caller fixture asserts - * the cross-file detection fired. + *

Cross-file resolution requires the callee's type to resolve at the call site; production does + * this via {@code sonar.java.binaries} (the project is compiled first). CheckVerifier does not put + * sibling sources on the semantic classpath, so we reproduce production by compiling the fixtures + * to {@code .class} and passing the output directory to {@link CheckVerifier#withClassPath}. + * + *

This guards the AST-detach work across its three branches: a literal argument and a + * field-constant argument are detached (asserted via the resolved CBOM values, since a detached + * cross-file detection produces a node but no tree-based SonarQube issue), while a {@code new + * byte[]} argument stays on the retained-tree path (asserted via its Noncompliant issue). */ class CrossFileHookDetachTest extends TestBase { @@ -56,19 +62,38 @@ class CrossFileHookDetachTest extends TestBase { private static final String CALLER = "src/test/files/rules/detection/crossfile/KeyGeneratorCaller.java"; + private static final List resolvedValues = new ArrayList<>(); + @Test - void crossFileLiteralResolves() throws Exception { + void crossFileDetectionAcrossDetachedAndRetainedPaths() throws Exception { + resolvedValues.clear(); File classes = compileToClasspath(WRAPPER, CALLER); CheckVerifier.newVerifier() .onFiles(CALLER, WRAPPER) .withClassPath(List.of(classes)) .withChecks(this) .verifyIssues(); + + // All three algorithms resolve cross-file into the CBOM, regardless of detach vs retain: + assertThat(resolvedValues).contains("AES", "Blowfish", "DES"); + } + + @Override + public void asserts( + int findingId, + @Nonnull DetectionStore detectionStore, + @Nonnull List nodes) { + collectAlgorithmValues(detectionStore); + } + + private void collectAlgorithmValues( + @Nonnull DetectionStore store) { + for (IValue value : store.getDetectionValues()) { + resolvedValues.add(value.asString()); + } + store.getChildren().forEach(this::collectAlgorithmValues); } - /** - * Compiles the fixtures to a temp dir so their types resolve across files (mimics binaries). - */ private static File compileToClasspath(String... sources) throws Exception { Path out = Files.createTempDirectory("xfile-classes"); JavaCompiler compiler = ToolProvider.getSystemJavaCompiler(); @@ -82,12 +107,4 @@ private static File compileToClasspath(String... sources) throws Exception { } return out.toFile(); } - - @Override - public void asserts( - int findingId, - @Nonnull DetectionStore detectionStore, - @Nonnull List nodes) { - // Cross-file resolution is asserted by the Noncompliant marker in KeyGeneratorCaller.java. - } } From 7e6253bbfd09f6e43e448f523709fc3037030059 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Mon, 6 Jul 2026 08:42:43 +0200 Subject: [PATCH 14/35] perf(engine): key-indexed subscription lookup in onNewHookSubscription MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scan only the call-stack bucket matching a hook's method name instead of every bucket; fall back to a full scan for ANY/multi-name matchers. Signed-off-by: Nicklas Körtge --- .../ibm/engine/callstack/CallStackAgent.java | 20 ++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java b/engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java index e4ae62d2..fbb814b8 100644 --- a/engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java +++ b/engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java @@ -27,6 +27,7 @@ import com.ibm.engine.language.ILanguageSupport; import com.ibm.engine.language.IScanContext; import java.util.ArrayList; +import java.util.Collection; import java.util.Iterator; import java.util.List; import java.util.Optional; @@ -127,7 +128,7 @@ public void onNewHookSubscription( } final List> stackCalls = new ArrayList<>(); - final Iterator>> iterator = invokedCallStack.values().iterator(); + final Iterator>> iterator = bucketsToScan(methodMatcher).iterator(); while (iterator.hasNext()) { final List> callContexts = iterator.next(); final Iterator> callContextIterator = callContexts.iterator(); @@ -148,6 +149,23 @@ public void onNewHookSubscription( } } + /** + * The call-stack buckets a hook's matcher could match. Recorded calls are keyed by their + * invoked method name's hash, so a single-name matcher only needs that one bucket; {@code + * ANY}/multi-name matchers fall back to scanning every bucket. + */ + @Nonnull + private Collection>> bucketsToScan( + @Nonnull MethodMatcher methodMatcher) { + final List methodNames = methodMatcher.getMethodNamesSerializable(); + if (methodNames.size() == 1 && !MethodMatcher.ANY.equals(methodNames.get(0))) { + final List> bucket = + invokedCallStack.get(methodNames.get(0).hashCode()); + return bucket == null ? List.of() : List.of(bucket); + } + return invokedCallStack.values(); + } + private boolean matches( @Nonnull MethodMatcher methodMatcher, @Nonnull CallContext callContext, From 6fed90c90edde14176be1eff50041d6a23351252 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Mon, 6 Jul 2026 08:50:30 +0200 Subject: [PATCH 15/35] docs: rewrite plan as as-built implementation record MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- ...6-07-05-callstack-detach-implementation.md | 1349 ++--------------- 1 file changed, 138 insertions(+), 1211 deletions(-) diff --git a/docs/superpowers/plans/2026-07-05-callstack-detach-implementation.md b/docs/superpowers/plans/2026-07-05-callstack-detach-implementation.md index 93f9a63c..575a51f8 100644 --- a/docs/superpowers/plans/2026-07-05-callstack-detach-implementation.md +++ b/docs/superpowers/plans/2026-07-05-callstack-detach-implementation.md @@ -1,1211 +1,138 @@ -# Call-stack AST-Detach Heap Reduction — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Stop the per-language `CallStackAgent` from pinning whole-file ASTs across a scan by storing tree-free "detached" records for eligible Java method calls, cutting the measured ~7 GB / ~179k-`CallContext` runtime heap without regressing cross-file detection. - -**Architecture:** `CallContext` becomes a sealed interface with two variants: `RetainedCall` (today's `tree + scanContext`, unchanged behavior) and `DetachedCall` (match keys + per-argument pre-resolved value snapshots + an AST-free `DetachedScanContext`). At record time the Java engine pre-resolves each argument while the file is live and, if faithfully reproducible, stores a `DetachedCall`; otherwise it keeps a `RetainedCall`. At hook-fire time a detached record replays from its snapshot, producing detection values whose location is a synthetic AST-free `DetachedSyntaxToken`. Because a `DetachedCall` holds no `Tree` and no `JavaFileScannerContext`, the file's AST becomes GC-eligible after `leaveFile`. - -**Tech Stack:** Java 17, Maven multi-module, sonar-java 8.0.1 `CheckVerifier`, JUnit 5 + AssertJ. - -## Global Constraints - -- Java 17; changes live in `engine` (+ Java language support) and `java` (translator + tests). Do not modify Python/Go behavior. -- Apache 2.0 license header in every new `.java` file — copy the 19-line header verbatim from any neighbor (e.g. `engine/src/main/java/com/ibm/engine/callstack/CallContext.java:1-19`). -- Run `mvn spotless:apply` before every commit. If Spotless truncates `mapper/.../JsonCipherSuites.java`, restore it (`git checkout -- `) before committing. -- `mvn test -pl engine` and `mvn test -pl java` must stay green after every task. -- Keep the generic engine language-agnostic: the detachability decision and pre-resolution live in the Java layer; the generic `CallStackAgent` only stores/keys/replays `CallContext` variants. -- **Scope for this iteration:** only Java **method invocations** are detachable. Enum accesses (`Tree.Kind.ENUM`), Python, and Go always use `RetainedCall` (unchanged). - ---- - -## File Structure - -- `engine/.../callstack/CallContext.java` — MODIFY → `sealed interface CallContext permits RetainedCall, DetachedCall`. -- `engine/.../callstack/RetainedCall.java` — CREATE — record `(T tree, IScanContext scanContext)`; today's shape. -- `engine/.../callstack/DetachedCall.java` — CREATE — record with match keys, `List`, `DetachedScanContext`. -- `engine/.../callstack/ArgSnapshot.java` — CREATE — one argument's resolved raw values + location primitives. -- `engine/.../callstack/DetachedSyntaxToken.java` — CREATE — AST-free `SyntaxToken` for value locations. -- `engine/.../callstack/DetachedScanContext.java` — CREATE — AST-free `IScanContext` (InputFile + filePath). -- `engine/.../callstack/CallStackAgent.java` — MODIFY — variant-aware add/key/match; drop `visitedTreeObjects`; key-indexed lookup. -- `engine/.../detection/MethodMatcher.java` — MODIFY — add `matchKeys(...)` overload. -- `engine/.../detection/DetectionStoreWithHook.java` — MODIFY — detached replay branch. -- `engine/.../hooks/{HookRepository,IHook,EnumHook,MethodInvocationHookWithParameterResolvement,MethodInvocationHookWithReturnResolvement,HookDetectionObservable,IHookDetectionObserver}.java` — MODIFY — consume sealed `CallContext` / carry the record through fire. -- `engine/.../detection/DetectionStore.java` — MODIFY — `onHookInvocation` carries the `CallContext`. -- `engine/.../language/ILanguageSupport.java` — MODIFY — `isDetachableCall` default `false`. -- `engine/.../language/java/JavaLanguageSupport.java` — MODIFY — Java `isDetachableCall`. -- `engine/.../language/java/JavaDetectionEngine.java` — MODIFY — record-time pre-resolution + build variant; expose a location-primitive capture helper. -- `java/.../plugin/translation/translator/JavaTranslator.java` — MODIFY — leading `DetachedSyntaxToken` branch in `getDetectionContextFrom`. -- `java/src/test/files/rules/detection/crossfile/CrossFileDefinition.java` + `CrossFileUsage.java` — CREATE — multi-file fixtures. -- `java/src/test/java/com/ibm/plugin/rules/detection/crossfile/CrossFileHookDetachTest.java` — CREATE — cross-file regression guard. - ---- - -## Task 1: Characterization test — cross-file hook detection baseline - -Locks current cross-file behavior BEFORE any refactor. This is the regression guard; it must pass on unmodified code and after every later task. - -**Files:** -- Create: `java/src/test/files/rules/detection/crossfile/CrossFileDefinition.java` -- Create: `java/src/test/files/rules/detection/crossfile/CrossFileUsage.java` -- Create: `java/src/test/java/com/ibm/plugin/rules/detection/crossfile/CrossFileHookDetachTest.java` - -**Interfaces:** -- Consumes: `TestBase` (`java/src/test/java/com/ibm/plugin/TestBase.java`), `CheckVerifier` (sonar-java). -- Produces: nothing code-facing; establishes the green baseline all later tasks preserve. - -- [ ] **Step 1: Write the definition fixture** (a wrapper method whose algorithm arg is a plain `String` parameter — forces a parameter hook). NOT a test source; no license header needed (matches other `src/test/files`). - -`java/src/test/files/rules/detection/crossfile/CrossFileDefinition.java`: -```java -package rules.detection.crossfile; - -import javax.crypto.Cipher; -import java.security.GeneralSecurityException; - -public class CrossFileDefinition { - public Cipher make(String transformation) throws GeneralSecurityException { - return Cipher.getInstance(transformation); // hook: resolve 'transformation' from caller - } -} -``` - -- [ ] **Step 2: Write the usage fixture** (calls the wrapper with a literal — this is the call recorded in "file A"): - -`java/src/test/files/rules/detection/crossfile/CrossFileUsage.java`: -```java -package rules.detection.crossfile; - -import javax.crypto.Cipher; -import java.security.GeneralSecurityException; - -public class CrossFileUsage { - public Cipher use() throws GeneralSecurityException { - return new CrossFileDefinition().make("AES/GCM/NoPadding"); - } -} -``` - -- [ ] **Step 3: Write the multi-file test.** Uses `CheckVerifier.newVerifier().onFiles(usage, definition)` so the usage (call site) is analyzed before the definition (hook site), exercising the `onNewHookSubscription` retro-scan across files. Assert that the resolved algorithm value `"AES"` reaches a detection value. - -`java/src/test/java/com/ibm/plugin/rules/detection/crossfile/CrossFileHookDetachTest.java`: -```java -/* */ -package com.ibm.plugin.rules.detection.crossfile; - -import static org.assertj.core.api.Assertions.assertThat; - -import com.ibm.engine.detection.DetectionStore; -import com.ibm.engine.model.IValue; -import com.ibm.engine.model.ValueAction; -import com.ibm.mapper.model.INode; -import com.ibm.plugin.TestBase; -import java.util.List; -import javax.annotation.Nonnull; -import org.junit.jupiter.api.Test; -import org.sonar.java.checks.verifier.CheckVerifier; -import org.sonar.plugins.java.api.JavaCheck; -import org.sonar.plugins.java.api.JavaFileScannerContext; -import org.sonar.plugins.java.api.semantic.Symbol; -import org.sonar.plugins.java.api.tree.Tree; - -class CrossFileHookDetachTest extends TestBase { - - private static boolean sawAes = false; - - @Test - void crossFileLiteralResolves() { - sawAes = false; - CheckVerifier.newVerifier() - .onFiles( - "src/test/files/rules/detection/crossfile/CrossFileUsage.java", - "src/test/files/rules/detection/crossfile/CrossFileDefinition.java") - .withCheck(this) - .verifyNoIssues(); - assertThat(sawAes).as("cross-file resolved algorithm value 'AES'").isTrue(); - } - - @Override - public void asserts( - int findingId, - @Nonnull DetectionStore detectionStore, - @Nonnull List nodes) { - collectValues(detectionStore); - } - - private void collectValues( - @Nonnull DetectionStore store) { - for (IValue value : store.getDetectionValues()) { - if (value.asString().contains("AES")) { - sawAes = true; - } - } - store.getChildren().forEach(this::collectValues); - } -} -``` - -- [ ] **Step 4: Run the test — verify it PASSES on unmodified code.** - -Run: `mvn test -pl java -Dtest=CrossFileHookDetachTest` -Expected: PASS, `sawAes` true. (If it fails, the fixtures don't trigger a parameter hook — adjust the wrapper so the algorithm is a method parameter, not a literal inside the wrapper, and re-run. Do not proceed until green.) - -- [ ] **Step 5: Commit.** -```bash -mvn spotless:apply -git add java/src/test/files/rules/detection/crossfile java/src/test/java/com/ibm/plugin/rules/detection/crossfile -git commit -m "test: cross-file hook detection characterization baseline" -``` - ---- - -## Task 2: `DetachedSyntaxToken` — AST-free value location - -**Files:** -- Create: `engine/src/main/java/com/ibm/engine/callstack/DetachedSyntaxToken.java` -- Test: `engine/src/test/java/com/ibm/engine/callstack/DetachedSyntaxTokenTest.java` - -**Interfaces:** -- Produces: `DetachedSyntaxToken(int line, int columnOffset, int endLine, int endColumnOffset, String text, List keywords) implements SyntaxToken`, plus getter `List keywords()`. `parent()` returns `null`; `kind()` returns `Tree.Kind.TOKEN`; `firstToken()`/`lastToken()` return `this`; `range()` returns `Range.at(Position.at(line, columnOffset), Position.at(endLine, endColumnOffset))`. - -- [ ] **Step 1: Write the failing test.** -```java -/* */ -package com.ibm.engine.callstack; - -import static org.assertj.core.api.Assertions.assertThat; - -import java.util.List; -import org.junit.jupiter.api.Test; -import org.sonar.plugins.java.api.tree.Tree; - -class DetachedSyntaxTokenTest { - @Test - void pinsNothingAndReportsRange() { - DetachedSyntaxToken t = - new DetachedSyntaxToken(12, 4, 12, 20, "AES", List.of("javax.crypto.Cipher#getInstance")); - assertThat(t.parent()).isNull(); - assertThat(t.kind()).isEqualTo(Tree.Kind.TOKEN); - assertThat(t.firstToken()).isSameAs(t); - assertThat(t.range().start().line()).isEqualTo(12); - assertThat(t.range().start().columnOffset()).isEqualTo(4); - assertThat(t.keywords()).containsExactly("javax.crypto.Cipher#getInstance"); - assertThat(t.text()).isEqualTo("AES"); - } -} -``` - -- [ ] **Step 2: Run — verify it fails to compile (class missing).** -Run: `mvn test -pl engine -Dtest=DetachedSyntaxTokenTest` -Expected: FAIL — `cannot find symbol DetachedSyntaxToken`. - -- [ ] **Step 3: Implement the class.** -```java -/* */ -package com.ibm.engine.callstack; - -import java.util.List; -import javax.annotation.Nonnull; -import javax.annotation.Nullable; -import org.sonar.plugins.java.api.location.Position; -import org.sonar.plugins.java.api.location.Range; -import org.sonar.plugins.java.api.tree.SyntaxToken; -import org.sonar.plugins.java.api.tree.SyntaxTrivia; -import org.sonar.plugins.java.api.tree.Tree; -import org.sonar.plugins.java.api.tree.TreeVisitor; - -/** AST-free {@link SyntaxToken} used as the location of a detached cross-file detection value. - * Holds only primitives, so it never pins a compilation unit ({@link #parent()} is null). */ -public final class DetachedSyntaxToken implements SyntaxToken { - private final int line; - private final int columnOffset; - private final int endLine; - private final int endColumnOffset; - @Nonnull private final String text; - @Nonnull private final List keywords; - - public DetachedSyntaxToken(int line, int columnOffset, int endLine, int endColumnOffset, - @Nonnull String text, @Nonnull List keywords) { - this.line = line; - this.columnOffset = columnOffset; - this.endLine = endLine; - this.endColumnOffset = endColumnOffset; - this.text = text; - this.keywords = List.copyOf(keywords); - } - - @Nonnull public List keywords() { return keywords; } - public int offset() { return columnOffset; } - - @Override public String text() { return text; } - @Override public List trivias() { return List.of(); } - @Override public int line() { return line; } - @Override public int column() { return columnOffset; } - @Override public Range range() { - return Range.at(Position.at(line, columnOffset + 1), Position.at(endLine, endColumnOffset + 1)); - } - @Override public boolean is(Tree.Kind... kinds) { - for (Tree.Kind k : kinds) { if (k == Tree.Kind.TOKEN) return true; } - return false; - } - @Override public void accept(TreeVisitor visitor) { /* detached: no traversal */ } - @Nullable @Override public Tree parent() { return null; } - @Override public SyntaxToken firstToken() { return this; } - @Override public SyntaxToken lastToken() { return this; } - @Override public Tree.Kind kind() { return Tree.Kind.TOKEN; } - - @Override public boolean equals(Object o) { - if (this == o) return true; - if (!(o instanceof DetachedSyntaxToken t)) return false; - return line == t.line && columnOffset == t.columnOffset && endLine == t.endLine - && endColumnOffset == t.endColumnOffset && text.equals(t.text) && keywords.equals(t.keywords); - } - @Override public int hashCode() { - int r = line; - r = 31 * r + columnOffset; - r = 31 * r + endLine; - r = 31 * r + endColumnOffset; - r = 31 * r + text.hashCode(); - r = 31 * r + keywords.hashCode(); - return r; - } -} -``` -Note: `Position.at` is 1-based on column in sonar-java; capture `columnOffset` 0-based and add 1 here so `range().start().columnOffset()` returns the original 0-based value. Verify against Step-1 assertion; adjust the `+1`/`-1` if the assertion fails. - -- [ ] **Step 4: Run — verify PASS** (fix the column offset convention if the range assertion fails). -Run: `mvn test -pl engine -Dtest=DetachedSyntaxTokenTest` -Expected: PASS. - -- [ ] **Step 5: Commit.** -```bash -mvn spotless:apply -git add engine/src/main/java/com/ibm/engine/callstack/DetachedSyntaxToken.java engine/src/test/java/com/ibm/engine/callstack/DetachedSyntaxTokenTest.java -git commit -m "feat(engine): AST-free DetachedSyntaxToken for detached value locations" -``` - ---- - -## Task 3: `DetachedScanContext` — AST-free scan context - -**Files:** -- Create: `engine/src/main/java/com/ibm/engine/callstack/DetachedScanContext.java` -- Test: `engine/src/test/java/com/ibm/engine/callstack/DetachedScanContextTest.java` - -**Interfaces:** -- Consumes: `IScanContext` (`engine/.../language/IScanContext.java`). -- Produces: `DetachedScanContext(InputFile inputFile, String filePath)` implementing `IScanContext`. `getFilePath()`/`getInputFile()` return the captured values; `reportIssue(...)` throws `UnsupportedOperationException` (never called on the production CBOM path — see spec risk audit in Task 12). - -- [ ] **Step 1: Write the failing test.** -```java -/* */ -package com.ibm.engine.callstack; - -import static org.assertj.core.api.Assertions.assertThat; -import static org.mockito.Mockito.mock; - -import org.junit.jupiter.api.Test; -import org.sonar.api.batch.fs.InputFile; - -class DetachedScanContextTest { - @Test - void exposesFilePathWithoutPinningAst() { - InputFile inputFile = mock(InputFile.class); - DetachedScanContext ctx = - new DetachedScanContext<>(inputFile, "/abs/path/CrossFileUsage.java"); - assertThat(ctx.getFilePath()).isEqualTo("/abs/path/CrossFileUsage.java"); - assertThat(ctx.getInputFile()).isSameAs(inputFile); - } -} -``` - -- [ ] **Step 2: Run — verify it fails to compile.** -Run: `mvn test -pl engine -Dtest=DetachedScanContextTest` -Expected: FAIL — class missing. - -- [ ] **Step 3: Implement.** -```java -/* */ -package com.ibm.engine.callstack; - -import com.ibm.engine.language.IScanContext; -import javax.annotation.Nonnull; -import org.sonar.api.batch.fs.InputFile; - -/** AST-free {@link IScanContext}: retains only the file handle + path captured at record time, - * never the {@code JavaFileScannerContext}, so a detached record does not pin the file's AST. */ -public record DetachedScanContext(@Nonnull InputFile inputFile, @Nonnull String filePath) - implements IScanContext { - @Override public void reportIssue(@Nonnull R currentRule, @Nonnull T tree, @Nonnull String message) { - throw new UnsupportedOperationException("reportIssue is not available on a detached scan context"); - } - @Nonnull @Override public InputFile getInputFile() { return inputFile; } - @Nonnull @Override public String getFilePath() { return filePath; } -} -``` - -- [ ] **Step 4: Run — verify PASS.** -Run: `mvn test -pl engine -Dtest=DetachedScanContextTest` -Expected: PASS. - -- [ ] **Step 5: Commit.** -```bash -mvn spotless:apply -git add engine/src/main/java/com/ibm/engine/callstack/DetachedScanContext.java engine/src/test/java/com/ibm/engine/callstack/DetachedScanContextTest.java -git commit -m "feat(engine): AST-free DetachedScanContext" -``` - ---- - -## Task 4: `ArgSnapshot` + `MethodMatcher.matchKeys` overload - -**Files:** -- Create: `engine/src/main/java/com/ibm/engine/callstack/ArgSnapshot.java` -- Modify: `engine/src/main/java/com/ibm/engine/detection/MethodMatcher.java` -- Test: `engine/src/test/java/com/ibm/engine/detection/MethodMatcherKeysTest.java` - -**Interfaces:** -- Produces: `ArgSnapshot(int index, List values)` and `ResolvedSnapshotValue(Object value, DetachedSyntaxToken location)`. -- Produces: `MethodMatcher.matchKeys(@Nonnull IType invokedObjectType, @Nonnull String methodName, @Nonnull List parameterTypes)` returning `boolean`, applying the same predicates as `match(...)` (`MethodMatcher.java:159`) without a tree. - -- [ ] **Step 1: Write `ArgSnapshot` and `ResolvedSnapshotValue`** (plain records; no test needed — exercised via later tasks). -```java -/* */ -package com.ibm.engine.callstack; - -import java.util.List; -import javax.annotation.Nonnull; - -public record ArgSnapshot(int index, @Nonnull List values) { - public record ResolvedSnapshotValue(@Nonnull Object value, @Nonnull DetachedSyntaxToken location) {} -} -``` - -- [ ] **Step 2: Write the failing `matchKeys` test.** Read `MethodMatcher.java:31-160` first to reuse the exact predicate fields (`invokedObjectTypeString`, `methodName`, `parameterTypes`). -```java -/* */ -package com.ibm.engine.detection; - -import static org.assertj.core.api.Assertions.assertThat; - -import java.util.LinkedList; -import java.util.List; -import org.junit.jupiter.api.Test; - -class MethodMatcherKeysTest { - @Test - void matchesByKeysSameAsTree() { - MethodMatcher matcher = - new MethodMatcher<>("javax.crypto.Cipher", "getInstance", - new LinkedList<>(List.of("java.lang.String"))); - IType type = new SimpleTypeForTest("javax.crypto.Cipher"); - assertThat(matcher.matchKeys(type, "getInstance", List.of(new SimpleTypeForTest("java.lang.String")))) - .isTrue(); - assertThat(matcher.matchKeys(type, "doFinal", List.of(new SimpleTypeForTest("java.lang.String")))) - .isFalse(); - } -} -``` -Add a tiny `SimpleTypeForTest implements IType` in the same test package if no test `IType` fake exists (check `engine/src/test/java/com/ibm/engine/detection/` first for an existing one and reuse it). - -- [ ] **Step 3: Run — verify it fails.** -Run: `mvn test -pl engine -Dtest=MethodMatcherKeysTest` -Expected: FAIL — `matchKeys` missing. - -- [ ] **Step 4: Implement `matchKeys`** by extracting the predicate tail of `match(...)`. In `MethodMatcher.java`, refactor `match(...)` (`:159`) to delegate: -```java -public boolean match(@Nonnull T expression, @Nonnull ILanguageTranslation translation, - @Nonnull MatchContext matchContext) { - Optional invokedObjectType = translation.getInvokedObjectTypeString(matchContext, expression); - Optional invokedMethodName = translation.getMethodName(matchContext, expression); - List param = translation.getMethodParameterTypes(matchContext, expression); - if (invokedObjectType.isEmpty() || invokedMethodName.isEmpty()) { - return false; - } - return matchKeysInternal(invokedObjectType.get(), invokedMethodName.get(), param, - translation.supportsSubsetParameterMatching()); -} - -public boolean matchKeys(@Nonnull IType invokedObjectType, @Nonnull String methodName, - @Nonnull List parameterTypes) { - return matchKeysInternal(invokedObjectType, methodName, parameterTypes, false); -} - -private boolean matchKeysInternal(@Nonnull IType invokedObjectType, @Nonnull String methodName, - @Nonnull List param, boolean subsetParameterMatching) { - boolean typeMatches = this.invokedObjectTypeString.test(invokedObjectType); - boolean nameMatches = this.methodName.test(methodName); - if (!typeMatches || !nameMatches) { - return false; - } - if (methodName.equals("") && !parameterTypesSerializable.isEmpty() && subsetParameterMatching) { - return anyParameterMatches(param); - } - return this.parameterTypes.test(param); -} -``` -(Detached calls are never constructors with subset matching, so `matchKeys` passes `false`.) - -- [ ] **Step 5: Run — verify PASS and no regressions.** -Run: `mvn test -pl engine -Dtest=MethodMatcherKeysTest,MethodMatcherTest` -Expected: PASS (run the existing `MethodMatcher*` tests too; if none exist, run `mvn test -pl engine`). - -- [ ] **Step 6: Commit.** -```bash -mvn spotless:apply -git add engine/src/main/java/com/ibm/engine/callstack/ArgSnapshot.java engine/src/main/java/com/ibm/engine/detection/MethodMatcher.java engine/src/test/java/com/ibm/engine/detection/MethodMatcherKeysTest.java -git commit -m "feat(engine): ArgSnapshot + MethodMatcher.matchKeys (tree-free match)" -``` - ---- - -## Task 5: `isDetachableCall` predicate in the language layer - -**Files:** -- Modify: `engine/src/main/java/com/ibm/engine/language/ILanguageSupport.java` -- Modify: `engine/src/main/java/com/ibm/engine/language/java/JavaLanguageSupport.java` -- Test: `engine/src/test/java/com/ibm/engine/language/java/JavaIsDetachableCallTest.java` - -**Interfaces:** -- Produces: `default boolean isDetachableCall(@Nonnull T tree) { return false; }` on `ILanguageSupport`. -- Produces: Java override — `true` iff `tree` is a `MethodInvocationTree` whose `methodSymbol().declaration() != null` (user method → could be hooked) AND no argument expression subtree contains a `NEW_ARRAY` node (the only factory-steered resolution case, `JavaDetectionEngine.java:275`); else `false`. - -- [ ] **Step 1: Add the default to `ILanguageSupport`.** Find the interface method list and add: -```java -/** Whether a recorded call may be stored tree-free (detached) instead of retaining its AST. - * Default: conservative false (retain the tree). Only Java method invocations override. */ -default boolean isDetachableCall(@Nonnull T tree) { - return false; -} -``` - -- [ ] **Step 2: Write the failing Java test.** -```java -/* */ -package com.ibm.engine.language.java; - -import static org.assertj.core.api.Assertions.assertThat; - -import org.junit.jupiter.api.Test; -import org.sonar.java.model.JParserTestUtils; -import org.sonar.plugins.java.api.tree.ClassTree; -import org.sonar.plugins.java.api.tree.CompilationUnitTree; -import org.sonar.plugins.java.api.tree.ExpressionStatementTree; -import org.sonar.plugins.java.api.tree.MethodTree; -import org.sonar.plugins.java.api.tree.Tree; - -class JavaIsDetachableCallTest { - private Tree firstStatementExpression(String body) { - CompilationUnitTree cut = JParserTestUtils.parse( - "class A { void m(String s){ " + body + " } byte[] b(){return null;} }"); - MethodTree m = (MethodTree) ((ClassTree) cut.types().get(0)).members().get(0); - ExpressionStatementTree stmt = (ExpressionStatementTree) m.block().body().get(0); - return stmt.expression(); - } - - @Test - void literalArgIsDetachable() { - JavaLanguageSupport support = new JavaLanguageSupport(); - // user method call: A.b() has a source declaration - Tree call = firstStatementExpression("b();"); - assertThat(support.isDetachableCall(call)).isTrue(); - } - - @Test - void arrayArgIsNotDetachable() { - JavaLanguageSupport support = new JavaLanguageSupport(); - Tree call = firstStatementExpression("java.util.Arrays.toString(new byte[]{1,2});"); - assertThat(support.isDetachableCall(call)).isFalse(); - } -} -``` -(If `JParserTestUtils` is not on the test classpath, mirror the parsing helper used by an existing `engine` Java test — check `engine/src/test/java/com/ibm/engine/language/java/` for the established parse utility and reuse it.) - -- [ ] **Step 3: Run — verify it fails.** -Run: `mvn test -pl engine -Dtest=JavaIsDetachableCallTest` -Expected: FAIL — `isDetachableCall` not overridden / returns false for the literal case. - -- [ ] **Step 4: Implement in `JavaLanguageSupport`.** -```java -@Override -public boolean isDetachableCall(@Nonnull Tree tree) { - if (!(tree instanceof MethodInvocationTree invocation)) { - return false; - } - if (invocation.methodSymbol().declaration() == null) { - return false; // library method: cannot match any source-declared method hook anyway - } - for (ExpressionTree arg : invocation.arguments()) { - if (containsNewArray(arg)) { - return false; // SizeFactory-steered resolution — keep the tree (fallback) - } - } - return true; -} - -private static boolean containsNewArray(@Nonnull Tree tree) { - if (tree.is(Tree.Kind.NEW_ARRAY)) { - return true; - } - boolean[] found = {false}; - tree.accept(new org.sonar.plugins.java.api.tree.BaseTreeVisitor() { - @Override public void visitNewArray(org.sonar.plugins.java.api.tree.NewArrayTree t) { - found[0] = true; - } - }); - return found[0]; -} -``` -Add imports: `MethodInvocationTree`, `ExpressionTree`, `Tree`. - -- [ ] **Step 5: Run — verify PASS.** -Run: `mvn test -pl engine -Dtest=JavaIsDetachableCallTest` -Expected: PASS. - -- [ ] **Step 6: Commit.** -```bash -mvn spotless:apply -git add engine/src/main/java/com/ibm/engine/language/ILanguageSupport.java engine/src/main/java/com/ibm/engine/language/java/JavaLanguageSupport.java engine/src/test/java/com/ibm/engine/language/java/JavaIsDetachableCallTest.java -git commit -m "feat(engine): isDetachableCall predicate (Java method invocations, no NEW_ARRAY)" -``` - ---- - -## Task 6: Convert `CallContext` to a sealed interface with `RetainedCall` (pure refactor, no behavior change) - -Make `CallContext` sealed with a single variant `RetainedCall`, updating every consumer of `.tree()`/`.publisher()`. Behavior is identical; all existing tests (incl. Task 1) stay green. This isolates the compile-breaking type change from any logic change. - -**Files:** -- Modify: `engine/src/main/java/com/ibm/engine/callstack/CallContext.java` -- Create: `engine/src/main/java/com/ibm/engine/callstack/RetainedCall.java` -- Modify: `CallStackAgent.java`, `HookRepository.java`, `IHook.java`, `EnumHook.java`, `MethodInvocationHookWithParameterResolvement.java`, `MethodInvocationHookWithReturnResolvement.java` - -**Interfaces:** -- Produces: `sealed interface CallContext permits RetainedCall, DetachedCall` with `@Nonnull IScanContext publisher()` and `@Nullable T tree()` (default-null; `RetainedCall` overrides with its tree). `DetachedCall` is added in Task 7 — declare it in `permits` now and create a temporary empty stub, or add `permits RetainedCall` here and widen in Task 7. -- Produces: `RetainedCall(T tree, IScanContext publisher) implements CallContext` — `tree()` returns the tree. - -- [ ] **Step 1: Turn `CallContext` into the sealed interface** (start with a single permit; Task 7 widens it): -```java -/* */ -package com.ibm.engine.callstack; - -import com.ibm.engine.language.IScanContext; -import javax.annotation.Nonnull; -import javax.annotation.Nullable; - -public sealed interface CallContext permits RetainedCall { - @Nullable T tree(); - @Nonnull IScanContext publisher(); -} -``` - -- [ ] **Step 2: Create `RetainedCall`.** -```java -/* */ -package com.ibm.engine.callstack; - -import com.ibm.engine.language.IScanContext; -import javax.annotation.Nonnull; - -public record RetainedCall(@Nonnull T tree, @Nonnull IScanContext publisher) - implements CallContext {} -``` - -- [ ] **Step 3: Update `CallStackAgent.addCall`** to build a `RetainedCall` and update the `visitedTreeObjects` guard (unchanged logic, new type). At `CallStackAgent.java:60`: -```java -final CallContext callContext = new RetainedCall<>(tree, scanContext); -``` -Everywhere `callContext.tree()` is used inside `CallStackAgent`, it remains valid (RetainedCall returns non-null). - -- [ ] **Step 4: Update the remaining consumers** to construct/expect the interface. The only constructions are in `CallStackAgent`. `HookRepository.update` (`:115-121`) uses `callContext.tree()` and `callContext.publisher()` — both still compile (interface methods). `IHook.isInvocationOn(CallContext,...)` and the three hook impls call `callContext.tree()` — still compile. No signature changes needed; only the `new CallContext<>(...)` construction (now `new RetainedCall<>(...)`) changes. Grep to confirm no other `new CallContext<>` exists: -Run: `grep -rn "new CallContext" engine/src/main/java` -Expected: no results after the edit. - -- [ ] **Step 5: Build + full engine/java tests — verify green (no behavior change).** -Run: `mvn test -pl engine && mvn test -pl java -Dtest=CrossFileHookDetachTest` -Expected: PASS (all engine tests + the Task 1 baseline). - -- [ ] **Step 6: Commit.** -```bash -mvn spotless:apply -git add engine/src/main/java/com/ibm/engine/callstack/CallContext.java engine/src/main/java/com/ibm/engine/callstack/RetainedCall.java engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java -git commit -m "refactor(engine): CallContext -> sealed interface with RetainedCall variant" -``` - ---- - -## Task 7: Add the `DetachedCall` variant - -**Files:** -- Create: `engine/src/main/java/com/ibm/engine/callstack/DetachedCall.java` -- Modify: `engine/src/main/java/com/ibm/engine/callstack/CallContext.java` (widen `permits`) -- Test: `engine/src/test/java/com/ibm/engine/callstack/DetachedCallTest.java` - -**Interfaces:** -- Produces: `DetachedCall(IType invokedObjectType, String methodName, List parameterTypes, List arguments, DetachedScanContext publisher) implements CallContext`. `tree()` returns `null`. Getter `arguments()`, `invokedObjectType()`, `methodName()`, `parameterTypes()`. - -- [ ] **Step 1: Widen the sealed permit** in `CallContext.java`: `permits RetainedCall, DetachedCall`. - -- [ ] **Step 2: Write the failing test.** -```java -/* */ -package com.ibm.engine.callstack; - -import static org.assertj.core.api.Assertions.assertThat; -import static org.mockito.Mockito.mock; - -import com.ibm.engine.detection.IType; -import java.util.List; -import org.junit.jupiter.api.Test; -import org.sonar.api.batch.fs.InputFile; - -class DetachedCallTest { - @Test - void holdsNoTree() { - DetachedScanContext ctx = - new DetachedScanContext<>(mock(InputFile.class), "/p/CrossFileUsage.java"); - DetachedCall call = - new DetachedCall<>(mock(IType.class), "make", List.of(), List.of(), ctx); - assertThat(call.tree()).isNull(); - assertThat(call.publisher()).isSameAs(ctx); - assertThat(call.methodName()).isEqualTo("make"); - } -} -``` - -- [ ] **Step 3: Run — verify it fails to compile.** -Run: `mvn test -pl engine -Dtest=DetachedCallTest` -Expected: FAIL — class missing. - -- [ ] **Step 4: Implement `DetachedCall`.** -```java -/* */ -package com.ibm.engine.callstack; - -import com.ibm.engine.detection.IType; -import java.util.List; -import javax.annotation.Nonnull; -import javax.annotation.Nullable; - -public record DetachedCall( - @Nonnull IType invokedObjectType, - @Nonnull String methodName, - @Nonnull List parameterTypes, - @Nonnull List arguments, - @Nonnull DetachedScanContext detachedPublisher) - implements CallContext { - @Nullable @Override public T tree() { return null; } - @Nonnull @Override public com.ibm.engine.language.IScanContext publisher() { - return detachedPublisher; - } -} -``` - -- [ ] **Step 5: Run — verify PASS and engine stays green.** -Run: `mvn test -pl engine -Dtest=DetachedCallTest && mvn test -pl engine` -Expected: PASS. - -- [ ] **Step 6: Commit.** -```bash -mvn spotless:apply -git add engine/src/main/java/com/ibm/engine/callstack/DetachedCall.java engine/src/main/java/com/ibm/engine/callstack/CallContext.java engine/src/test/java/com/ibm/engine/callstack/DetachedCallTest.java -git commit -m "feat(engine): DetachedCall variant (tree-free recorded call)" -``` - ---- - -## Task 8: Record-time production of `DetachedCall` in the Java engine - -Decide detach at record time, pre-resolve arguments while the file is live, and store a `DetachedCall`; else a `RetainedCall`. Keeps matching/replay unchanged for now (Task 9 wires replay), so detached calls are recorded but a hook firing on one would still take the retained path — to keep tests green this task also makes `CallStackAgent` skip detached calls during match until Task 9. To avoid a green-but-lossy interim, **Tasks 8 and 9 are committed together** (see Task 9 Step 6); do Task 8 without an intermediate "all green" claim beyond compilation. - -**Files:** -- Modify: `engine/src/main/java/com/ibm/engine/language/java/JavaDetectionEngine.java` -- Modify: `engine/src/main/java/com/ibm/engine/detection/Handler.java` (add `addRecordedCall`) -- Modify: `engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java` (accept a prebuilt `CallContext`) - -**Interfaces:** -- Produces: `Handler.addRecordedCall(@Nonnull CallContext recordedCall)` → delegates to `callStackAgent.add(recordedCall)`. -- Produces: `CallStackAgent.add(@Nonnull CallContext callContext)` — keys off `methodName` for `DetachedCall`, off `getKeyFormT(tree)` for `RetainedCall`; same dedup/notify contract. -- Produces (Java engine): a `DetachedCall` whose `arguments` are built by pre-resolving each invocation argument with `resolveValuesInInnerScope(Object.class, arg, null)` and capturing each `ResolvedValue`'s value + a `DetachedSyntaxToken` computed via a new `captureLocation(Tree)` helper mirroring `JavaTranslator.getDetectionContextFrom`. - -- [ ] **Step 1: Add `CallStackAgent.add(CallContext)`** alongside the existing `addCall(tree, scanContext)`. Extract a key helper that works for both variants: -```java -public void add(@Nonnull CallContext callContext) { - final Optional keyOptional = keyOf(callContext); - if (keyOptional.isEmpty()) { - return; - } - if (addedToCallContext(keyOptional.get(), callContext)) { - this.notify(callContext); - } -} - -private Optional keyOf(@Nonnull CallContext callContext) { - if (callContext instanceof DetachedCall detached) { - return Optional.of(detached.methodName().hashCode()); - } - final T tree = callContext.tree(); - return tree == null ? Optional.empty() : getKeyFormT(tree); -} -``` -Change the dedup in `addedToCallContext` to guard on the tree only when present (detached calls dedup by identity of the record — always add): -```java -private boolean addedToCallContext(int key, @Nonnull CallContext callContext) { - final T tree = callContext.tree(); - if (tree != null) { - if (visitedTreeObjects.contains(tree)) { - return false; - } - visitedTreeObjects.add(tree); - } - invokedCallStack.compute( - key, - (k, v) -> { - if (v == null) { - final ArrayList> list = new ArrayList<>(); - list.add(callContext); - return list; - } - v.add(callContext); - return v; - }); - return true; -} -``` -Keep `addCall(tree, scanContext)` delegating: `add(new RetainedCall<>(tree, scanContext));`. (Task 12 later removes `visitedTreeObjects`; keep it here so this task is behavior-preserving.) - -- [ ] **Step 2: Add `Handler.addRecordedCall`** next to `addCallToCallStack` (`Handler.java:51`): -```java -public void addRecordedCall(@Nonnull CallContext recordedCall) { - this.callStackAgent.add(recordedCall); -} -``` - -- [ ] **Step 3: Add the `captureLocation` helper to `JavaDetectionEngine`** (mirrors `JavaTranslator.getDetectionContextFrom`, `JavaTranslator.java:170-224`, but returns a `DetachedSyntaxToken`). Read that method first and copy its token/keyword logic exactly: -```java -@Nullable -private DetachedSyntaxToken captureLocation(@Nonnull Tree location, @Nonnull String text) { - SyntaxToken firstToken = location.firstToken(); - SyntaxToken lastToken = location.lastToken(); - if (firstToken == null || lastToken == null) { - return null; - } - SyntaxToken locationToken = firstToken; - List keywords = List.of(); - // Keyword/additionalContext fidelity: for the common detached case the location is a literal - // (default branch -> empty keywords), so implementing only `default` is correct for literals. - // To match today's output when a resolved value's location is a NEW_CLASS / METHOD_INVOCATION / - // ENUM_CONSTANT, replicate the three case bodies from JavaTranslator.getDetectionContextFrom - // (JavaTranslator.java:179-216) verbatim here — they only read the live tree and produce the - // `keywords` list + a more specific `locationToken`. Those Java-tree casts are available in the - // engine module. Example for METHOD_INVOCATION: - // MethodInvocationTree mit = (MethodInvocationTree) location; - // SyntaxToken sel = mit.methodSelect().firstToken(); - // if (sel != null) locationToken = sel; - // keywords = List.of(mit.methodSymbol().signature(), mit.methodSymbol().name()); - Position start = locationToken.range().start(); - Position end = lastToken.range().end(); - return new DetachedSyntaxToken(start.line(), start.columnOffset(), end.line(), end.columnOffset(), - text, keywords); -} -``` -Note the column convention: `Position.columnOffset()` is 0-based, matching `DetachedSyntaxToken`'s stored `columnOffset`. `DetectionLocation`'s `offset` is fed from `columnOffset()` today (`JavaTranslator.java:223`), so capture and store `columnOffset()` (0-based) consistently end to end. - -- [ ] **Step 4: Build the variant in `JavaDetectionEngine.run`** at the method-invocation site (`JavaDetectionEngine.java:98-100`). Replace the unconditional `handler.addCallToCallStack(...)` with: -```java -MethodInvocationTree methodInvocationTree = (MethodInvocationTree) tree; -final IScanContext scanContext = detectionStore.getScanContext(); -if (handler.getLanguageSupport().isDetachableCall(methodInvocationTree)) { - final DetachedCall detached = buildDetachedCall(methodInvocationTree, scanContext); - if (detached != null) { - handler.addRecordedCall(detached); - } else { - handler.addCallToCallStack(methodInvocationTree, scanContext); // pre-resolution failed -> keep tree - } -} else { - handler.addCallToCallStack(methodInvocationTree, scanContext); -} -``` -Leave the enum site (`:113-115`) calling `addCallToCallStack` unchanged (enums are never detached this iteration). - -- [ ] **Step 5: Implement `buildDetachedCall`.** Uses the translation for match keys and `resolveValuesInInnerScope` for arguments; returns `null` if any argument fails to resolve or a location can't be captured (→ caller keeps the tree): -```java -@Nullable -private DetachedCall buildDetachedCall( - @Nonnull MethodInvocationTree invocation, @Nonnull IScanContext scanContext) { - final MatchContext matchContext = MatchContext.build(false, detectionStore.getDetectionRule()); - final ILanguageTranslation translation = handler.getLanguageSupport().translation(); - final Optional invokedType = translation.getInvokedObjectTypeString(matchContext, invocation); - final Optional name = translation.getMethodName(matchContext, invocation); - if (invokedType.isEmpty() || name.isEmpty()) { - return null; - } - final List paramTypes = translation.getMethodParameterTypes(matchContext, invocation); - - final List args = new ArrayList<>(); - final List arguments = invocation.arguments(); - for (int i = 0; i < arguments.size(); i++) { - final List> resolved = - resolveValuesInInnerScope(Object.class, arguments.get(i), null); - final List snapshots = new ArrayList<>(); - for (ResolvedValue rv : resolved) { - final DetachedSyntaxToken loc = captureLocation(rv.tree(), rv.value().toString()); - if (loc == null) { - return null; // cannot faithfully snapshot -> fall back to retained tree - } - snapshots.add(new ArgSnapshot.ResolvedSnapshotValue(rv.value(), loc)); - } - args.add(new ArgSnapshot(i, snapshots)); - } - - final DetachedScanContext detachedCtx = - new DetachedScanContext<>(scanContext.getInputFile(), scanContext.getFilePath()); - return new DetachedCall<>(invokedType.get(), name.get(), paramTypes, args, detachedCtx); -} -``` -Add imports: `ArgSnapshot`, `DetachedCall`, `DetachedScanContext`, `DetachedSyntaxToken`, `ILanguageTranslation`, `IType`, `MatchContext`, `Position`, `SyntaxToken`, `ArrayList`, `List`, `Optional`. - -- [ ] **Step 6: Compile only (replay lands in Task 9).** -Run: `mvn -q -pl engine test-compile` -Expected: compiles. (Do not run full tests here; matching still ignores detached records — Task 9 wires replay. Proceed directly to Task 9; commit is shared.) - ---- - -## Task 9: Fire-time replay for `DetachedCall` - -Make matching and hook-fire handle detached records so a `DetachedCall` produces the same detection value a `RetainedCall` would, then verify the Task 1 baseline passes with detachment live. - -**Files:** -- Modify: `engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java` (`onNewHookSubscription` match) -- Modify: `engine/src/main/java/com/ibm/engine/hooks/HookRepository.java` (`update`) -- Modify: `engine/.../hooks/MethodInvocationHookWithParameterResolvement.java`, `EnumHook.java`, `MethodInvocationHookWithReturnResolvement.java`, `IHook.java` (`isInvocationOn(CallContext,...)`) -- Modify: `engine/.../hooks/HookDetectionObservable.java`, `IHookDetectionObserver.java`, `DetectionStore.java`, `DetectionStoreWithHook.java` (carry the `CallContext` through fire instead of a raw `T` tree) - -**Interfaces:** -- Produces: `IHook.isInvocationOn(CallContext,...)` matches `DetachedCall` via `MethodMatcher.matchKeys(invokedObjectType, methodName, parameterTypes)` and `RetainedCall` via the existing tree `match(...)`. -- Produces: the fire path carries `CallContext` (not `T invocationTree`) from `HookRepository.update` → `HookDetectionObservable.notify` → `IHookDetectionObserver.onHookInvocation` → `DetectionStoreWithHook`. Retained records behave exactly as before (`callContext.tree()`); detached records take the snapshot branch. - -- [ ] **Step 1: Match keys in the hooks.** In `MethodInvocationHookWithParameterResolvement.isInvocationOn(CallContext,...)` (`:59-63`) and the return-resolvement equivalent, branch on the variant: -```java -@Override -public boolean isInvocationOn(@Nonnull CallContext callContext, - @Nonnull ILanguageSupport languageSupport) { - if (callContext instanceof DetachedCall detached) { - MethodMatcher matcher = languageSupport.createMethodMatcherBasedOn(methodDefinition); - return matcher != null - && matcher.matchKeys(detached.invokedObjectType(), detached.methodName(), - detached.parameterTypes()); - } - return isInvocationOn(callContext.tree(), languageSupport); -} -``` -For `EnumHook.isInvocationOn(CallContext,...)`: detached calls are never enums, so `if (callContext instanceof DetachedCall) return false;` then fall through to the tree path. - -- [ ] **Step 2: Match in `CallStackAgent.onNewHookSubscription`** (`:100-112`). Replace `methodMatcher.match(callContext.tree(), translation, matchContext)` with a variant-aware check: -```java -final boolean matches; -if (callContext instanceof DetachedCall detached) { - matches = methodMatcher.matchKeys(detached.invokedObjectType(), detached.methodName(), - detached.parameterTypes()); -} else { - matches = methodMatcher.match(callContext.tree(), languageSupport.translation(), hook.matchContext()); -} -if (matches) { stackCalls.add(callContext); } -``` - -- [ ] **Step 3: Carry the `CallContext` through the fire path.** Change signatures from `T invocationTree` to `CallContext callContext`: - - `HookRepository.update` (`:120-121`): `handler.notifyAllHookDetectionObservers(callContext, hook)` (drop the separate tree/publisher args). - - `HookDetectionObservable.notify` (`:57-75`) + `IHookDetectionObservable`: take `CallContext callContext`, call `subscribers.get(i).onHookInvocation(callContext, hook)`. - - `IHookDetectionObserver.onHookInvocation` + `DetectionStore.onHookInvocation` (`:399-416`): take `CallContext callContext`; build the child store with `callContext.publisher()` and pass `callContext` into `DetectionStoreWithHook`. - - Update `Handler.notifyAllHookDetectionObservers` accordingly. - - `DetectionStoreWithHook` stores `CallContext callContext` instead of `T invocationTree`; `invocationTree` usages become `callContext.tree()` (retained) or the snapshot branch (detached). - -- [ ] **Step 4: Detached branch in `DetectionStoreWithHook.handleMethodInvocationHookWithParameterResolvement`** (`:124-203`). When `callContext instanceof DetachedCall`, skip `extractArgumentFromMethodCaller`/`resolveValuesInInnerScope` and use the snapshot at the hook's parameter index: -```java -if (callContext instanceof DetachedCall detached) { - int idx = parameterIndex(hook.methodDefinition(), hook.methodParameter()); - if (idx < 0 || idx >= detached.arguments().size()) { return; } - ArgSnapshot snap = detached.arguments().get(idx); - if (hook.getParameter() instanceof DetectableParameter detectableParameter) { - for (ArgSnapshot.ResolvedSnapshotValue rv : snap.values()) { - @SuppressWarnings("unchecked") - T loc = (T) rv.location(); - ResolvedValue resolved = new ResolvedValue<>(rv.value(), loc); - new ValueDetection<>(resolved, detectableParameter, loc, null) - .toValue(detectableParameter.getiValueFactory()) - .ifPresent(iValue -> addValue(detectableParameter.getIndex(), iValue)); - } - } - handleNextRulesForMethodHooks(hook, /* traceSymbol */ null, isSuccessive); - return; -} -// ... existing retained-tree logic unchanged ... -``` -Add helper (index of the hook's parameter identifier within the method definition): -```java -private int parameterIndex(@Nonnull T methodDefinition, @Nonnull T methodParameter) { - return handler.getLanguageSupport().parameterIndexOf(methodDefinition, methodParameter); -} -``` -Add `parameterIndexOf(T methodDefinition, T methodParameter)` to `ILanguageSupport` (default `-1`) and implement for Java in `JavaLanguageSupport` by matching `methodParameter`'s name against `((MethodTree) methodDefinition).parameters()` simple names (reuse `translation().resolveIdentifierAsString`). Because `T loc` is a `DetachedSyntaxToken` (a `Tree`), the cast is safe for Java (`T = Tree`). - -- [ ] **Step 5: Run the full engine suite + the cross-file baseline (now exercising the detached path).** -Run: `mvn test -pl engine && mvn test -pl java -Dtest=CrossFileHookDetachTest` -Expected: PASS. The Task 1 test now flows through `DetachedCall` (the `"AES/GCM/NoPadding"` literal resolves at record time). If it fails, log the recorded variant in `CallStackAgent.add` at DEBUG and confirm the usage call became a `DetachedCall` and the snapshot index maps to the `transformation` parameter. - -- [ ] **Step 6: Commit Tasks 8 + 9 together.** -```bash -mvn spotless:apply -git add engine/src/main/java/com/ibm/engine java/ # staged engine + any java touch -git commit -m "feat(engine): produce and replay DetachedCall for Java method invocations" -``` - ---- - -## Task 10: `JavaTranslator` DetachedSyntaxToken branch (location fidelity) - -**Files:** -- Modify: `java/src/main/java/com/ibm/plugin/translation/translator/JavaTranslator.java` -- Test: `java/src/test/java/com/ibm/plugin/translation/translator/JavaTranslatorDetachedLocationTest.java` - -**Interfaces:** -- Consumes: `DetachedSyntaxToken` (engine). -- Produces: `getDetectionContextFrom` returns a `DetectionLocation` built directly from a `DetachedSyntaxToken` when the location is one. - -- [ ] **Step 1: Write the failing test.** -```java -/* */ -package com.ibm.plugin.translation.translator; - -import static org.assertj.core.api.Assertions.assertThat; - -import com.ibm.engine.callstack.DetachedSyntaxToken; -import com.ibm.mapper.model.IBundle; -import com.ibm.mapper.utils.DetectionLocation; -import java.util.List; -import org.junit.jupiter.api.Test; -import org.mockito.Mockito; - -class JavaTranslatorDetachedLocationTest { - @Test - void buildsLocationFromDetachedToken() { - DetachedSyntaxToken token = - new DetachedSyntaxToken(7, 2, 7, 9, "AES", List.of("kw")); - DetectionLocation loc = - new JavaTranslator().getDetectionContextFrom(token, Mockito.mock(IBundle.class), "F.java"); - assertThat(loc.lineNumber()).isEqualTo(7); - assertThat(loc.keywords()).containsExactly("kw"); - } -} -``` -(Confirm `DetectionLocation`'s accessor names via `mapper/.../DetectionLocation.java:26-31`; adjust `lineNumber()`/`keywords()` to the real record component names.) - -- [ ] **Step 2: Run — verify it fails.** -Run: `mvn test -pl java -Dtest=JavaTranslatorDetachedLocationTest` -Expected: FAIL — falls through to token logic / empty keywords. - -- [ ] **Step 3: Add the leading branch** to `getDetectionContextFrom` (`JavaTranslator.java:170`): -```java -@Nullable public DetectionLocation getDetectionContextFrom( - @Nonnull Tree location, @Nonnull IBundle bundle, @Nonnull String filePath) { - if (location instanceof DetachedSyntaxToken d) { - return new DetectionLocation(filePath, d.line(), d.offset(), d.keywords(), bundle); - } - // ... existing logic unchanged ... -} -``` -Add import `com.ibm.engine.callstack.DetachedSyntaxToken`. - -- [ ] **Step 4: Run — verify PASS.** -Run: `mvn test -pl java -Dtest=JavaTranslatorDetachedLocationTest` -Expected: PASS. - -- [ ] **Step 5: Commit.** -```bash -mvn spotless:apply -git add java/src/main/java/com/ibm/plugin/translation/translator/JavaTranslator.java java/src/test/java/com/ibm/plugin/translation/translator/JavaTranslatorDetachedLocationTest.java -git commit -m "feat(java): translate DetachedSyntaxToken locations directly" -``` - ---- - -## Task 11: End-to-end verification + `getLocation()` consumer audit - -**Files:** -- Modify (if audit finds a gap): the offending consumer only. -- Test: extend `CrossFileHookDetachTest` with a NEW_ARRAY (tree-fallback) case. - -- [ ] **Step 1: Add a fallback-path case to the cross-file test.** This exercises the `isDetachableCall == false` retained path across files (a `NEW_ARRAY` argument keeps the tree). - -`java/src/test/files/rules/detection/crossfile/CrossFileIvDefinition.java`: -```java -package rules.detection.crossfile; - -import javax.crypto.spec.IvParameterSpec; - -public class CrossFileIvDefinition { - public IvParameterSpec make(byte[] iv) { - return new IvParameterSpec(iv); // NEW_ARRAY flows here -> retained (non-detachable) call - } -} -``` -`java/src/test/files/rules/detection/crossfile/CrossFileIvUsage.java`: -```java -package rules.detection.crossfile; - -import javax.crypto.spec.IvParameterSpec; - -public class CrossFileIvUsage { - public IvParameterSpec use() { - return new CrossFileIvDefinition().make(new byte[16]); - } -} -``` -Add a test method to `CrossFileHookDetachTest` (reuse its `collectValues`/`sawAes` pattern with a fresh flag `sawIv`, asserting an `IvParameterSpec`/nonce-related detection value is produced across the two files): -```java -private static boolean sawIv = false; - -@Test -void crossFileArrayStillDetects() { - sawIv = false; - CheckVerifier.newVerifier() - .onFiles( - "src/test/files/rules/detection/crossfile/CrossFileIvUsage.java", - "src/test/files/rules/detection/crossfile/CrossFileIvDefinition.java") - .withCheck(this) - .verifyNoIssues(); - assertThat(sawIv).as("cross-file retained (array) path still detects").isTrue(); -} -``` -In `collectValues`, also set `sawIv = true` when a value's class/asString indicates the IV/nonce detection (adjust the marker to whatever `JavaDetectionRules` emit for `IvParameterSpec`; run once and inspect the logged detection store to pick the exact value type). - -- [ ] **Step 2: Run both cross-file cases.** -Run: `mvn test -pl java -Dtest=CrossFileHookDetachTest` -Expected: PASS (both literal-detached and array-retained resolve). - -- [ ] **Step 3: Audit `getLocation()` consumers** (spec top risk). Confirm no consumer navigates `parent()`/`accept()`/children on a value location: -Run: `grep -rn "getLocation()" --include=*.java engine java mapper output enricher | grep -v /target/ | grep -v test` -Expected: only translator `getDetectionContextFrom` calls; if any consumer calls `.parent()`/`.accept()`/child accessors on a location, add a `DetachedSyntaxToken` guard there. Record the audit result in the commit message. - -- [ ] **Step 4: Full suite green.** -Run: `mvn test -pl engine && mvn test -pl java` -Expected: PASS. - -- [ ] **Step 5: Commit.** -```bash -mvn spotless:apply -git add java/src/test engine java # only files actually changed -git commit -m "test: cross-file detach + retained-array cases; getLocation consumer audit" -``` - ---- - -## Task 12: Cleanup — remove redundant `visitedTreeObjects` - -Measured 100% redundant with `invokedCallStack` retention. Dedup within the per-name bucket instead. - -**Files:** -- Modify: `engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java` - -- [ ] **Step 1: Replace the field-based dedup** (`CallStackAgent.java:45`, `:123-127`) with a per-bucket check. In `addedToCallContext`, for retained calls test membership in the bucket list by tree identity before appending; detached calls always append (distinct records): -```java -private boolean addedToCallContext(int key, @Nonnull CallContext callContext) { - final T tree = callContext.tree(); - final boolean[] added = {true}; - invokedCallStack.compute(key, (k, v) -> { - List> list = (v == null) ? new ArrayList<>() : v; - if (tree != null) { - for (CallContext existing : list) { - if (tree.equals(existing.tree())) { added[0] = false; return list; } - } - } - list.add(callContext); - return list; - }); - return added[0]; -} -``` -Delete the `visitedTreeObjects` field and its imports. - -- [ ] **Step 2: Full engine + cross-file green.** -Run: `mvn test -pl engine && mvn test -pl java -Dtest=CrossFileHookDetachTest` -Expected: PASS. - -- [ ] **Step 3: Commit.** -```bash -mvn spotless:apply -git add engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java -git commit -m "perf(engine): drop redundant visitedTreeObjects; dedup within bucket" -``` - ---- - -## Task 13: Cleanup — key-indexed subscription lookup - -`onNewHookSubscription` currently scans all buckets (`CallStackAgent.java:101`). Look up only the bucket for the hook's method name. - -**Files:** -- Modify: `engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java` - -- [ ] **Step 1: Derive the name key from the hook and scan one bucket.** In `onNewHookSubscription`, after building `methodMatcher`, compute the hook's method-name key the same way records are keyed (`methodName.hashCode()` for detached, `getKeyFormT` for retained) and iterate only `invokedCallStack.get(key)`. If the hook's method name isn't statically known (multi-name/`ANY` matcher), fall back to scanning `invokedCallStack.values()` as today. Obtain the name from `languageSupport.translation().getMethodName(hook.matchContext(), hook.hookValue())`; if empty, use the fallback scan. -```java -final Optional hookName = - languageSupport.translation().getMethodName(hook.matchContext(), hook.hookValue()); -final Collection>> buckets = - hookName.map(n -> invokedCallStack.getOrDefault(n.hashCode(), List.of())) - .map(List::of) - .map(l -> (Collection>>) l) - .orElseGet(invokedCallStack::values); -``` -Then iterate `buckets` with the existing match logic (Task 9 Step 2). Keep the fallback correct for hooks whose name resolves differently than the record key. - -- [ ] **Step 2: Full engine + cross-file green** (the Task 1 test registers the hook in the definition file and must still find the usage-file record via the keyed bucket). -Run: `mvn test -pl engine && mvn test -pl java -Dtest=CrossFileHookDetachTest` -Expected: PASS. - -- [ ] **Step 3: Commit.** -```bash -mvn spotless:apply -git add engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java -git commit -m "perf(engine): key-indexed subscription lookup in CallStackAgent" -``` - ---- - -## Task 14: Heap measurement + docs - -**Files:** -- Modify: `docs/TROUBLESHOOTING.md` (append a short note referencing the measurement) - -- [ ] **Step 1: Full build + all module tests.** -Run: `mvn clean test` -Expected: PASS across all modules. - -- [ ] **Step 2: Measure heap on a large compiled project.** Use the keycloak repro from the memory note (`mvn sonar:sonar` so types resolve; source-only scans fire `addCall` zero times). Constrain the scanner heap and capture before/after: -```bash -MAVEN_OPTS="-Xmx4g" mvn -o sonar:sonar -Dsonar.host.url=... 2>&1 | tee scan.log -# in another shell, sample the scanner JVM: -jmap -histo | grep -E "CallContext|DetachedCall|RetainedCall|Tree" | head -``` -Record retained `RetainedCall` vs `DetachedCall` counts and peak used heap; compare to the ~179k `CallContext` / ~7 GB baseline in `docs/superpowers/plans/2026-07-05-callstack-hooks-heap-reduction.md`. - -- [ ] **Step 3: Write the result** as a short paragraph in `docs/TROUBLESHOOTING.md` (retained vs detached split, peak-heap delta). If detached share is low, note it (candidate follow-ups: enum detachment, array pre-resolution). - -- [ ] **Step 4: Commit.** -```bash -git add docs/TROUBLESHOOTING.md -git commit -m "docs: record call-stack detach heap measurement" -``` - ---- - -## Self-review notes (coverage map) - -- Spec §"Match" → Task 4 (`matchKeys`), Task 9 Steps 1–2. -- Spec §"Resolution input" (record-time pre-resolution, NEW_ARRAY fallback, expressionToResolve n/a for Java) → Task 5 (predicate), Task 8 (`buildDetachedCall`), Task 9 Step 4 (snapshot replay). -- Spec §"Detection output / synthetic token" → Task 2, Task 10. -- Spec §"scanContext gap" → Task 3, Task 8 Step 5, Task 9 Step 3. -- Spec §"Hybrid detach with tree-fallback" → Task 5 + Task 8 Step 4 (retained fallback paths). -- Spec §"Lossless cleanups" → Task 12 (visitedTreeObjects), Task 13 (key-indexed lookup). -- Spec §"Verification" → Task 1 (baseline), Task 11 (both paths + consumer audit), Task 14 (heap profile). -- Spec §"out of scope" (Python/Go, enums, caps) → enforced by default-`false` predicate; enum site left on `addCallToCallStack` (Task 8 Step 4). +# Call-stack AST-Detach Heap Reduction — Implementation Record (as-built) + +> **Status:** Implemented on branch `feat/callstack-ast-detach`. Full clean build green. One task +> (heap measurement) deferred. This document records what was actually built; it diverges +> substantially from the original step-by-step plan because several assumptions were corrected +> during execution (see "Divergences from the original plan"). + +**Goal:** Stop the per-language `CallStackAgent` from pinning whole-file ASTs for the entire scan +(the measured ~7 GB / ~179k-`CallContext` runtime leak) without regressing detection or SonarQube +issue reporting — same-file *or* cross-file. + +**Outcome:** Recorded calls are detached from their AST when their file finishes analysis. A file's +AST becomes garbage-collectable after `leaveFile`, while cross-file hook matching continues from a +tree-free snapshot. Same-file detections keep full behavior; cross-file detections produce CBOM +nodes and raise SonarQube issues without pinning any AST. + +**Tech stack:** Java 17, Maven multi-module, sonar-java 8.0.1, JUnit 5 + AssertJ + `CheckVerifier`. + +--- + +## As-built architecture + +A recorded call is a sealed `CallContext`: +- `RetainedCall(T tree, IScanContext publisher, @Nullable DetachedCall detachedForm)` — holds the + live tree; used while the call's file is being analyzed. +- `DetachedCall(IType invokedObjectType, String methodName, List parameterTypes, + List arguments, DetachedScanContext publisher)` — tree-free. + +**Lifecycle:** +1. **Record (file live):** `JavaDetectionEngine.recordCall` stores a `RetainedCall`. If the call is + detachable (`isDetachableCall`), it also pre-builds the `DetachedCall` *now* (arguments resolved + while the file's AST + symbols are live) and stashes it in `RetainedCall.detachedForm`. +2. **Same-file fire (file live):** hook detections during the file's own analysis match/replay + through the `RetainedCall`'s live tree + context, so their SonarQube issues and value resolution + are unchanged. +3. **`leaveFile`:** `JavaBaseDetectionRule.leaveFile` → `ILanguageSupport.notifyLeaveFile` → + `CallStackAgent.detachCallsForFile(inputFile)` swaps each of that file's `RetainedCall`s for its + `detachedForm`, dropping the tree → the file's AST is GC-eligible. +4. **Cross-file fire (file gone):** later hooks match the `DetachedCall` via + `MethodMatcher.matchKeys(...)` and replay from its `ArgSnapshot`s. The detection value's location + is an AST-free `DetachedSyntaxToken`; its SonarQube issue is raised via `SonarComponents` (see + Reporting). + +**Detachability (`JavaLanguageSupport.isDetachableCall`):** a `MethodInvocationTree` whose arguments +contain no `NEW_ARRAY` (the one value-factory-dependent resolution case, `SizeFactory`). It does +**not** require `methodSymbol().declaration() != null` — cross-file callees resolve via the binary +classpath, so `declaration()` is null for exactly the cross-file calls we must detach. + +**Reporting (cross-file, no AST):** sonar-java's public issue API is entirely tree-based, and any +tree pins the AST. So cross-file detached issues are raised through the internal +`SonarComponents.reportIssue(new AnalyzerMessage(check, inputFile, TextSpan, message, 0))` — +`SonarComponents` is shared per scan and pins no AST. It is captured at record time and read +reflectively from the protected `DefaultModuleScannerContext.sonarComponents` field +(`JavaDetachedIssueReporter`, with a graceful null fallback to CBOM-only). `IDetachedIssueReporter` +abstracts this in the engine; `DetachedScanContext.reportIssue` delegates to it using the +`DetachedSyntaxToken`'s captured range. + +--- + +## Files (as-built) + +Engine (`engine/src/main/java/com/ibm/engine/`): +- `callstack/CallContext.java` — sealed interface (`permits RetainedCall, DetachedCall`). +- `callstack/RetainedCall.java` — NEW — `(tree, publisher, @Nullable detachedForm)`. +- `callstack/DetachedCall.java` — NEW — tree-free record. +- `callstack/ArgSnapshot.java` — NEW — per-argument resolved values + location. +- `callstack/DetachedSyntaxToken.java` — NEW — AST-free `SyntaxToken` (value location). +- `callstack/DetachedScanContext.java` — NEW — AST-free `IScanContext` (InputFile + path + reporter). +- `callstack/IDetachedIssueReporter.java` — NEW — engine-side reporting abstraction. +- `callstack/CallStackAgent.java` — variant-aware add/match; `detachCallsForFile`; key-indexed + subscription lookup; `visitedTreeObjects` removed (per-bucket dedup). +- `detection/MethodMatcher.java` — `matchKeys(IType, name, List)` overload. +- `detection/DetectionStore.java`, `detection/DetectionStoreWithHook.java` — fire path carries + `CallContext`; detached replay branch (`replayDetachedParameterHook`). +- `hooks/*` — `notify`/`onHookInvocation` carry `CallContext`; `isInvocationOn(CallContext)` matches + `DetachedCall` via `matchKeys`. +- `language/ILanguageSupport.java` — `isDetachableCall`, `parameterIndexOf`, `notifyLeaveFile` + defaults. +- `language/java/JavaLanguageSupport.java` — Java impls of the above. +- `language/java/JavaDetectionEngine.java` — `recordCall` / `buildDetachedCall` / `captureLocation`. +- `language/java/JavaDetachedIssueReporter.java` — NEW — `SonarComponents` reporter. + +Java plugin (`java/src/`): +- `main/.../rules/detection/JavaBaseDetectionRule.java` — `leaveFile` hook. +- `test/files/rules/detection/crossfile/*` + `test/.../crossfile/CrossFileHookDetachTest.java`, + `IsDetachableCallTest.java` — NEW — the first multi-file tests (compiled-classpath technique). + +--- + +## Testing + +**Cross-file test technique (new to the repo):** `CheckVerifier.onFiles(...)` alone does not give +sibling sources shared semantics, so cross-file callee types don't resolve and hooks never match +(why no multi-file test ever existed). Production resolves cross-file types via `sonar.java.binaries` +(the project is compiled first). Reproduced by compiling the fixtures to `.class` in-test and passing +the dir to `CheckVerifier.withClassPath(...)`. See `CrossFileHookDetachTest`. + +**What the cross-file guard asserts:** all three detach branches resolve cross-file into the CBOM +(literal → detached, field-constant → detached, `new byte[]` → retained). The retained (array) case +also asserts its SonarQube issue via `verifyIssues`. + +**`CheckVerifier` limitation (documented):** it reads issues only from the test context's +`getIssues()` (populated by `context.reportIssue`), not from `SonarComponents`. So cross-file +detached squid issues (retro-scan order) are production-correct but not `verifyIssues`-observable; +they are asserted via CBOM values instead. + +--- + +## Divergences from the original plan (and why) + +1. **Detach at `leaveFile`, not at record time.** Record-time detach routed same-file hook + detections through the tree-free path, so they lost their live-context SonarQube issues and broke + ~7 `rules/resolve` tests. Deferring to `leaveFile` keeps same-file detections fully live. +2. **`isDetachableCall` dropped the `declaration() != null` check.** That check (from the abandoned + Phase-1 *filter* rationale) is null for every cross-file call (binary-classpath resolution), so it + would have made cross-file calls non-detachable — silently defeating the heap goal. +3. **Cross-file reporting via internal `SonarComponents` + reflection.** The spec assumed + `reportIssue` was off the path; it is not (`report()` yields issues). Public reporting is + tree-based (AST-pinning), so the internal `SonarComponents` path was required. +4. **No explicit `JavaTranslator` branch for `DetachedSyntaxToken`.** Unnecessary — it flows through + `getDetectionContextFrom`'s generic token path correctly. + +--- + +## Remaining + +- **Heap measurement (deferred):** scan a large compiled project (e.g. keycloak via `mvn + sonar:sonar`, or supply `sonar.java.libraries`) under a constrained `MAVEN_OPTS="-Xmx"`; + capture `jmap -histo` before/after and record the drop in retained `CallContext`/AST and peak + heap, versus the ~179k / ~7 GB baseline in + `docs/superpowers/plans/2026-07-05-callstack-hooks-heap-reduction.md`. This is the piece that + quantifies the win and should run before merge. + +## Follow-ups worth a reviewer's eye + +- `JavaDetachedIssueReporter` reflectively reads an internal sonar-java field — fragile across major + upgrades (graceful null fallback mitigates). Consistent with the existing `ExpressionUtils` usage. +- Enum accesses, Python and Go remain on the always-retained path (unchanged) by design. From 05bdda95e5c7a1e6ac4ed99410912a1638894d7e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Mon, 6 Jul 2026 08:51:11 +0200 Subject: [PATCH 16/35] refactor(engine): reorder imports, suppress field reflection warning MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- .../engine/language/java/JavaDetachedIssueReporter.java | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/engine/src/main/java/com/ibm/engine/language/java/JavaDetachedIssueReporter.java b/engine/src/main/java/com/ibm/engine/language/java/JavaDetachedIssueReporter.java index 6578ca0b..74b9fdab 100644 --- a/engine/src/main/java/com/ibm/engine/language/java/JavaDetachedIssueReporter.java +++ b/engine/src/main/java/com/ibm/engine/language/java/JavaDetachedIssueReporter.java @@ -21,9 +21,6 @@ import com.ibm.engine.callstack.DetachedSyntaxToken; import com.ibm.engine.callstack.IDetachedIssueReporter; -import java.lang.reflect.Field; -import javax.annotation.Nonnull; -import javax.annotation.Nullable; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.sonar.api.batch.fs.InputFile; @@ -33,6 +30,10 @@ import org.sonar.plugins.java.api.JavaCheck; import org.sonar.plugins.java.api.JavaFileScannerContext; +import javax.annotation.Nonnull; +import javax.annotation.Nullable; +import java.lang.reflect.Field; + /** * Raises a SonarQube issue for a detached (tree-free) cross-file detection through {@link * SonarComponents}, which is shared per scan and does not pin any file's AST. It is captured while @@ -80,6 +81,7 @@ public void report( sonarComponents.reportIssue(new AnalyzerMessage(rule, inputFile, span, message, 0)); } + @SuppressWarnings("java:S3011") @Nullable private static SonarComponents extractSonarComponents(@Nonnull JavaFileScannerContext context) { try { final Field field = From 21443aa2d8031130ae86fbb77305577a1aa38c95 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Mon, 6 Jul 2026 09:03:24 +0200 Subject: [PATCH 17/35] docs: design for self-contained call-stack heap/perf harness MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- ...7-06-callstack-heap-perf-harness-design.md | 208 ++++++++++++++++++ 1 file changed, 208 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-06-callstack-heap-perf-harness-design.md diff --git a/docs/superpowers/specs/2026-07-06-callstack-heap-perf-harness-design.md b/docs/superpowers/specs/2026-07-06-callstack-heap-perf-harness-design.md new file mode 100644 index 00000000..2117e8fb --- /dev/null +++ b/docs/superpowers/specs/2026-07-06-callstack-heap-perf-harness-design.md @@ -0,0 +1,208 @@ +# Call-Stack Heap/Perf Harness — Design + +**Date:** 2026-07-06 +**Branch context:** `feat/callstack-ast-detach` +**Status:** Approved (design), pending implementation plan. + +## Goal + +A single, **manual-on-demand** JUnit test that stresses the `CallStackAgent` call +retention mechanism at scale and **deterministically asserts** that recorded calls get +detached (per-file ASTs released) at `leaveFile` — the observable signature of the +`feat/callstack-ast-detach` fix — while also **reporting** heap and wall-time for manual +inspection. + +Constraints, from brainstorming: + +- **Self-contained JUnit** — no keycloak checkout, no docker/SonarQube, no network. The + corpus is generated in-process into a temp dir. +- **Not keycloak** — a synthetic corpus of similar *shape* (cross-file crypto calls), with + scale as a tunable knob, standing in for a large real project. +- **Types must resolve** — the heap term only appears when detections actually fire. A + source-only scan resolves nothing → `addCall` fires 0 times → the term is 0 (see + `docs/superpowers/plans/2026-07-05-callstack-hooks-heap-reduction.md`, "Measured" + section, and the `callstack-heap-term-measured` project memory). So the harness must + compile the corpus and put it on the semantic classpath, exactly as + `CrossFileHookDetachTest` does. +- **Deterministic gate, no heap-threshold flake** — assert on a *structural* invariant + (how many recorded calls still pin a live AST after the scan), not on measured MB. Heap + MB is reported only, never asserted. +- **Manual only** — `@Tag("performance")`, excluded from the default build. It does not + gate CI. It is run explicitly when validating a change. + +## Why this shape (context) + +The `CallStackAgent` records a `CallContext` per matched invocation for the whole scan so +that a hook created in file B can resolve a call recorded in file A (cross-file detection). +Before the detach work, every retained `CallContext` held a live sonar-java `Tree`, and any +`Tree` pins its whole file's AST (parent chain → `CompilationUnitTree`), so the retained set +pinned effectively the whole project's AST at once — measured at ~179k `CallContext`s / +~7 GB on a near-full keycloak scan, growing linearly and unbounded. + +The fix (`feat/callstack-ast-detach`): at `leaveFile`, each retained call that is detachable +is swapped for a tree-free `DetachedCall` (`CallStackAgent.detachCallsForFile`), so the +file's AST becomes GC-eligible while cross-file matching continues from an AST-free snapshot. + +This harness proves that mechanism works at scale without needing the full keycloak route. +It does **not** try to reproduce the literal 7 GB number; that remains the documented manual +`mvn sonar:sonar` measurement (pointer kept in the test's javadoc). + +## Components + +### 1. `CryptoCorpusGenerator` (test helper) + +Programmatically writes `N` `.java` source files into a temp dir. It emits the exact pattern +the retention mechanism keys on — a cross-file wrapper/caller shape mirroring the +`CrossFileHookDetachTest` fixtures (`KeyGeneratorWrapper` / `KeyGeneratorCaller`) — not flat +single-file crypto calls, because: + +- Cross-file wrapper calls are what create **method hooks** (user method with a source + declaration) and cause calls to be **recorded and later replayed** — the code path that + accumulates `CallContext`s. +- **Distinct** class/method names per unit keep call sites distinct so retention grows with + corpus size (a flat corpus with one repeated call site would dedup to ~nothing and not + stress the term). + +Per generated "unit" (parameterized, repeated `N` times with unique names): + +- A **wrapper** class exposing a crypto factory method that internally calls a resolvable JCA + API (`Cipher.getInstance` / `KeyGenerator.getInstance` / `MessageDigest.getInstance`, + rotated across units so multiple rules fire). +- A **caller** class that invokes the wrapper across the file boundary with: + - a **string literal** argument (→ detachable), + - a **field-constant** argument (→ detachable), + - a **`new byte[]`** argument (→ retained; stays on the tree path by design — this is the + known non-detachable case). + +Scale knob: `-Dperf.corpus.files` (integer, default small — see Open Parameters). "Files" +counts source files; each unit is 2 files (wrapper + caller), so units = files / 2. + +The generator returns the list of generated file paths (for `onFiles`) so the test does not +hard-code fixture names. + +### 2. Driver (the test) + +`CallStackHeapPerfTest extends TestBase`, `@Tag("performance")`. Reuses +`CrossFileHookDetachTest`'s proven driving path verbatim, at scale: + +1. Generate the corpus (component 1) into a temp dir. +2. Compile all generated sources to a classpath output dir via + `ToolProvider.getSystemJavaCompiler` (same helper shape as + `CrossFileHookDetachTest.compileToClasspath`). +3. `CheckVerifier.newVerifier().onFiles(allGeneratedFiles).withClassPath(List.of(classes)) + .withChecks(this).verifyIssues()` — compiling + classpath is what makes the callee types + resolve at the call sites so cross-file detections fire and `addCall` runs. + +`verifyIssues()` is used (not a bare scan) to stay on the exact, supported CheckVerifier code +path the rest of the suite uses; the harness does not assert specific issue lines (the +corpus is generated, so issue positions are not hand-authored). The perf assertions come +from the call-context stats, not from CheckVerifier issue matching. + +### 3. Inspection accessor (small production addition) + +`CallStackAgent.invokedCallStack` is `private` with no accessor. Add a minimal read-only +introspection method rather than using reflection (clearer, and legitimate test-support +surface): + +- `CallStackAgent`: add a `CallContextStats callContextStats()` that walks + `invokedCallStack` once and returns counts: + - `retainedWithTree` — `RetainedCall` entries whose `tree() != null` (i.e. still pinning an + AST), + - `detached` — `DetachedCall` entries, + - `total`, `buckets`. +- `CallContextStats` — a small immutable value type (record) in + `com.ibm.engine.callstack`. +- Thread the accessor out so the test can reach it after the scan: + `CallStackAgent.callContextStats()` → `Handler.callContextStats()` → + `ILanguageSupport` (a `callContextStats()` method) → reachable from the test via + `JavaAggregator.getLanguageSupport()`. + +This is production code but purely additive and read-only; it computes on demand and retains +nothing. + +### 4. Metrics + report + +After the scan completes (all files have been left, so detach has run for every file): + +**ASSERT — deterministic gate (the point of the test):** + +- `retainedWithTree <= bound`, where `bound` accounts only for the intentionally-retained + `new byte[]` cases plus any non-detachable calls (i.e. roughly proportional to units, not + to total recorded calls). The precise bound formula is derived during implementation from + the generator's known unit count and validated empirically (see "Validating the harness"). +- `detachedRatio = detached / (detached + retainedWithTree) >= threshold` (a high ratio, + e.g. most recorded calls detached). Exact threshold pinned during implementation. + +Both are pure counts of object variants in `invokedCallStack` after the scan — no GC, no +timing, no MB. They fail iff detach did not happen (AST pinning reintroduced). + +**REPORT — never asserted (manual-inspection view):** + +- `System.gc()` (best-effort) then `MemoryMXBean` heap-used delta across the scan. +- Wall-clock time for the scan phase. +- Raw `CallContextStats` (`retainedWithTree`, `detached`, `total`, `buckets`) and the corpus + size (`files`, `units`). + +Printed to stdout in a single clearly-labelled block so a human can compare runs across +branches. No assertion references these values. + +## Build configuration + +Add `performance` to the `maven-surefire-plugin` +configuration in the parent `pom.xml` (currently declared with no ``). Effect: + +- `mvn test` / `mvn package` — skips the `performance`-tagged test (CI unaffected). +- `mvn test -Dgroups=performance -pl java` — runs it. + +## Placement + +- `java/src/test/java/com/ibm/plugin/perf/CallStackHeapPerfTest.java` +- `java/src/test/java/com/ibm/plugin/perf/CryptoCorpusGenerator.java` + +The `java` module is required because the corpus uses JCA/BouncyCastle APIs (must be on the +test classpath to resolve) and the test uses sonar-java `CheckVerifier`. + +## Data flow + +``` +CryptoCorpusGenerator.generate(N) -> temp dir of .java (wrapper+caller units) + -> compileToClasspath(sources) -> classes dir + -> CheckVerifier.onFiles(...).withClassPath(classes).withChecks(this).verifyIssues() + -> engine records CallContexts (RetainedCall) as detections fire + -> at each leaveFile, detachCallsForFile swaps detachable calls -> DetachedCall + -> JavaAggregator.getLanguageSupport().callContextStats() + -> ASSERT retainedWithTree <= bound, detachedRatio >= threshold + -> REPORT heap delta, time, raw counts (stdout, not asserted) +``` + +## Validating the harness itself + +The structural assertion is only worth committing if it actually discriminates: + +1. **Passes on `feat/callstack-ast-detach`** — with detach live, `retainedWithTree` stays + bounded and `detachedRatio` is high. +2. **Would fail if detach were disabled** — during development only, temporarily neutralize + `CallStackAgent.detachCallsForFile` (make it a no-op) and confirm the test **fails** + (`retainedWithTree` grows with corpus size). This neutralization is a local sanity check, + **not committed**. + +Both are recorded as explicit implementation steps in the plan. + +## Out of scope + +- Reproducing the literal ~7 GB keycloak number — remains the documented manual + `mvn sonar:sonar` route; the test's javadoc points to + `docs/superpowers/plans/2026-07-05-callstack-hooks-heap-reduction.md` for that. +- Any assertion on measured heap MB (report-only, to avoid machine/GC flake). +- Python and Go corpora — Java-only, where the term was measured and where the detach fix + lives. +- Automating docker/SonarQube end-to-end. + +## Open parameters (pinned during implementation) + +- Default `perf.corpus.files` (small enough for a few-seconds manual run, large enough to + make retention visible — candidate ~200). +- Exact `retainedWithTree` bound formula and `detachedRatio` threshold (derived from the + generator's unit count and confirmed empirically in the harness-validation step). +- Which JCA APIs to rotate across units (at least `Cipher`, `KeyGenerator`, `MessageDigest` + — all already covered by existing Java detection rules). From f1f32644d29248e9d58d7b801fadf24cd98c8296 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Mon, 6 Jul 2026 09:13:50 +0200 Subject: [PATCH 18/35] docs: task-by-task plan for call-stack heap/perf harness MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- .../2026-07-06-callstack-heap-perf-harness.md | 711 ++++++++++++++++++ 1 file changed, 711 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-06-callstack-heap-perf-harness.md diff --git a/docs/superpowers/plans/2026-07-06-callstack-heap-perf-harness.md b/docs/superpowers/plans/2026-07-06-callstack-heap-perf-harness.md new file mode 100644 index 00000000..840b01fb --- /dev/null +++ b/docs/superpowers/plans/2026-07-06-callstack-heap-perf-harness.md @@ -0,0 +1,711 @@ +# Call-Stack Heap/Perf Harness Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add a self-contained, manual-on-demand JUnit test that generates a large synthetic cross-file crypto corpus, scans it in-process, and deterministically asserts that recorded calls are detached (per-file ASTs released) at `leaveFile` — the signature of the `feat/callstack-ast-detach` fix — while reporting heap/time for manual inspection. + +**Architecture:** Add a small read-only introspection accessor (`CallContextStats`) threaded down the existing `CallStackAgent → Handler → ILanguageSupport → JavaLanguageSupport` chain (the same chain that already carries `detachCallsForFile`/`notifyLeaveFile`). A test-only `CryptoCorpusGenerator` writes N wrapper/caller `.java` unit pairs to a temp dir; a `@Tag("performance")` test compiles them to a classpath dir and scans them with sonar-java `CheckVerifier` (the proven `CrossFileHookDetachTest` driving path — compiling + classpath is what makes callee types resolve so cross-file detections fire and calls get recorded). After the scan it reads `CallContextStats` and asserts a structural invariant (tree-pinning calls stay bounded; detached ratio high). The tag is excluded from the default build via surefire. + +**Tech Stack:** Java 17, Maven multi-module, JUnit 5 + AssertJ + Mockito 5.17, sonar-java `CheckVerifier` (java-checks-testkit 8.15), Google Java Format (Spotless, AOSP), Checkstyle. + +## Global Constraints + +- **Spec:** `docs/superpowers/specs/2026-07-06-callstack-heap-perf-harness-design.md`. +- **Before every commit:** `mvn spotless:apply` then `mvn checkstyle:check` must succeed. Apache 2.0 license header required in every new `.java` file (Spotless applies it from the configured header; run `spotless:apply` and commit the result). `@Override` required where overriding; no unused imports; camelCase params. +- **Watch out — `spotless:apply` intermittently truncates `mapper/.../JsonCipherSuites.java`.** If `git status` after formatting shows that file changed and you did not touch it, restore it (`git checkout -- mapper/src/main/resources/**/JsonCipherSuites.json` / the generated file) before committing. Do not commit a truncated `JsonCipherSuites`. +- **Never weaken or delete an assertion to make a build pass.** The perf test's thresholds are tuned once against observed numbers (Task 4) and then fixed. +- **The perf test must never gate CI** — it is `@Tag("performance")` and excluded from the default build. Only `mvn test` staying green (perf skipped) is required for CI; the perf test is run manually. +- **Determinism:** the perf test asserts only on `CallContextStats` counts (object-variant counts in the call stack), never on measured heap MB or wall-time. Heap/time are printed, never asserted. +- **DetectionLocation for engine tests (if needed):** `new DetectionLocation("testfile", 1, 1, List.of("test"), () -> "TEST")`. + +--- + +### Task 1: `CallContextStats` record + counting logic + +Add the immutable stats record and its pure counting factory in the `engine` module. This is the data the perf test asserts on. The factory is the only non-trivial logic, so it is what the unit test targets. + +**Files:** +- Create: `engine/src/main/java/com/ibm/engine/callstack/CallContextStats.java` +- Test: `engine/src/test/java/com/ibm/engine/callstack/CallContextStatsTest.java` + +**Interfaces:** +- Consumes: existing `CallContext` (sealed), `RetainedCall` (`@Nonnull T tree()`), `DetachedCall` (`tree()` returns `null`), all in `com.ibm.engine.callstack`. +- Produces: + - `public record CallContextStats(int retainedWithTree, int detached, int total, int buckets)` + - `public static final CallContextStats CallContextStats.EMPTY` + - `public double CallContextStats.detachedRatio()` — `detached/total`, or `1.0` when `total == 0`. + - `public static CallContextStats CallContextStats.from(Collection>> buckets)` + +- [ ] **Step 1: Write the failing test** + +Create `engine/src/test/java/com/ibm/engine/callstack/CallContextStatsTest.java` (license header added by `spotless:apply` in Step 4): + +```java +package com.ibm.engine.callstack; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.within; +import static org.mockito.Mockito.mock; + +import com.ibm.engine.language.IScanContext; +import java.util.List; +import org.junit.jupiter.api.Test; + +class CallContextStatsTest { + + @Test + @SuppressWarnings("unchecked") + void countsRetainedAndDetachedAcrossBuckets() { + IScanContext ctx = mock(IScanContext.class); + RetainedCall retained1 = new RetainedCall<>("t1", ctx, null); + RetainedCall retained2 = new RetainedCall<>("t2", ctx, null); + DetachedCall detached = mock(DetachedCall.class); + + List>> buckets = + List.of(List.of(retained1, detached), List.of(retained2)); + + CallContextStats stats = CallContextStats.from(buckets); + + assertThat(stats.retainedWithTree()).isEqualTo(2); + assertThat(stats.detached()).isEqualTo(1); + assertThat(stats.total()).isEqualTo(3); + assertThat(stats.buckets()).isEqualTo(2); + assertThat(stats.detachedRatio()).isCloseTo(1.0d / 3.0d, within(1e-9)); + } + + @Test + void emptyHasZeroCountsAndFullRatio() { + assertThat(CallContextStats.EMPTY.total()).isZero(); + assertThat(CallContextStats.EMPTY.buckets()).isZero(); + assertThat(CallContextStats.EMPTY.detachedRatio()).isEqualTo(1.0d); + } +} +``` + +Note: `DetachedCall` is a `record` (final); Mockito 5.17's default inline mock maker mocks it fine. The counting checks `instanceof DetachedCall` first and never calls a method on the mock, so no stubbing is needed. + +- [ ] **Step 2: Run test to verify it fails** + +Run: `mvn test -pl engine -Dtest=CallContextStatsTest` +Expected: FAIL — compilation error, `CallContextStats` does not exist. + +- [ ] **Step 3: Create the record + factory** + +Create `engine/src/main/java/com/ibm/engine/callstack/CallContextStats.java`: + +```java +package com.ibm.engine.callstack; + +import java.util.Collection; +import java.util.List; +import javax.annotation.Nonnull; + +/** + * Immutable snapshot of the {@link CallStackAgent}'s recorded-call population, used by the + * performance/heap harness to assert that calls were detached (ASTs released) at {@code leaveFile}. + * + * @param retainedWithTree number of {@link RetainedCall}s still pinning a live AST tree + * @param detached number of tree-free {@link DetachedCall}s + * @param total {@code retainedWithTree + detached} + * @param buckets number of hash buckets in the call stack + */ +public record CallContextStats(int retainedWithTree, int detached, int total, int buckets) { + + /** A zero-valued snapshot (used as the language-agnostic default). */ + public static final CallContextStats EMPTY = new CallContextStats(0, 0, 0, 0); + + /** Fraction of recorded calls that are detached; {@code 1.0} when nothing was recorded. */ + public double detachedRatio() { + return total == 0 ? 1.0d : (double) detached / (double) total; + } + + /** Counts retained-with-tree vs. detached calls across all call-stack buckets. */ + @Nonnull + public static CallContextStats from( + @Nonnull Collection>> buckets) { + int retainedWithTree = 0; + int detached = 0; + for (List> bucket : buckets) { + for (CallContext call : bucket) { + if (call instanceof DetachedCall) { + detached++; + } else if (call instanceof RetainedCall retained && retained.tree() != null) { + retainedWithTree++; + } + } + } + return new CallContextStats( + retainedWithTree, detached, retainedWithTree + detached, buckets.size()); + } +} +``` + +- [ ] **Step 4: Format, then run the test to verify it passes** + +Run: `mvn spotless:apply -pl engine && mvn test -pl engine -Dtest=CallContextStatsTest` +Expected: PASS (2 tests). If `git status` shows an unrelated `JsonCipherSuites` change, restore it (see Global Constraints). + +- [ ] **Step 5: Checkstyle + commit** + +```bash +mvn checkstyle:check -pl engine +git add engine/src/main/java/com/ibm/engine/callstack/CallContextStats.java \ + engine/src/test/java/com/ibm/engine/callstack/CallContextStatsTest.java +git commit -m "feat(engine): CallContextStats accessor for call-stack retention" +``` + +--- + +### Task 2: Expose stats through `CallStackAgent → Handler → ILanguageSupport → JavaLanguageSupport` + +Thread a `callContextStats()` accessor down the existing chain so a test can read the populated stats after a scan via `JavaAggregator.getLanguageSupport()`. Default interface method returns `EMPTY` (Go/Python untouched); `JavaLanguageSupport` overrides to delegate to its `Handler`. + +**Files:** +- Modify: `engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java` +- Modify: `engine/src/main/java/com/ibm/engine/detection/Handler.java` +- Modify: `engine/src/main/java/com/ibm/engine/language/ILanguageSupport.java` +- Modify: `engine/src/main/java/com/ibm/engine/language/java/JavaLanguageSupport.java` +- Test: `engine/src/test/java/com/ibm/engine/language/java/JavaLanguageSupportStatsTest.java` + +**Interfaces:** +- Consumes: `CallContextStats` and `CallContextStats.from(...)` (Task 1); existing `CallStackAgent.invokedCallStack` (private `ConcurrentMap>>`); `Handler.callStackAgent`; `JavaLanguageSupport.handler`; `LanguageSupporter.javaLanguageSupporter()`. +- Produces: + - `public @Nonnull CallContextStats CallStackAgent.callContextStats()` + - `public @Nonnull CallContextStats Handler.callContextStats()` + - `default @Nonnull CallContextStats ILanguageSupport.callContextStats()` → `CallContextStats.EMPTY` + - `@Override public @Nonnull CallContextStats JavaLanguageSupport.callContextStats()` → `handler.callContextStats()` + +- [ ] **Step 1: Write the failing test** + +Create `engine/src/test/java/com/ibm/engine/language/java/JavaLanguageSupportStatsTest.java`: + +```java +package com.ibm.engine.language.java; + +import static org.assertj.core.api.Assertions.assertThat; + +import com.ibm.engine.callstack.CallContextStats; +import com.ibm.engine.language.ILanguageSupport; +import com.ibm.engine.language.LanguageSupporter; +import org.junit.jupiter.api.Test; +import org.sonar.plugins.java.api.JavaCheck; +import org.sonar.plugins.java.api.JavaFileScannerContext; +import org.sonar.plugins.java.api.semantic.Symbol; +import org.sonar.plugins.java.api.tree.Tree; + +class JavaLanguageSupportStatsTest { + + @Test + void freshSupportReportsEmptyStatsThroughTheChain() { + ILanguageSupport support = + LanguageSupporter.javaLanguageSupporter(); + + CallContextStats stats = support.callContextStats(); + + assertThat(stats).isNotNull(); + assertThat(stats.total()).isZero(); + assertThat(stats.retainedWithTree()).isZero(); + assertThat(stats.detached()).isZero(); + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `mvn test -pl engine -Dtest=JavaLanguageSupportStatsTest` +Expected: FAIL — compilation error, `ILanguageSupport.callContextStats()` does not exist. + +- [ ] **Step 3: Add the accessor to `CallStackAgent`** + +In `engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java`, add this method (e.g. directly after `detachCallsForFile`, around line 75). Same package as `CallContextStats`, so no import needed: + +```java + /** Read-only snapshot of the recorded-call population (retained-with-tree vs. detached). */ + @Nonnull + public CallContextStats callContextStats() { + return CallContextStats.from(invokedCallStack.values()); + } +``` + +- [ ] **Step 4: Add the delegating accessor to `Handler`** + +In `engine/src/main/java/com/ibm/engine/detection/Handler.java`, add the import `import com.ibm.engine.callstack.CallContextStats;` (with the other `com.ibm.engine.callstack.*` imports) and this method (near `detachCallsForFile`, around line 90): + +```java + @Nonnull + public CallContextStats callContextStats() { + return this.callStackAgent.callContextStats(); + } +``` + +- [ ] **Step 5: Add the default method to `ILanguageSupport`** + +In `engine/src/main/java/com/ibm/engine/language/ILanguageSupport.java`, add the import `import com.ibm.engine.callstack.CallContextStats;` and, next to the `notifyLeaveFile` default (around line 155), add: + +```java + /** + * Read-only snapshot of the language's recorded-call population, for the performance/heap + * harness. Languages that do not accumulate a call stack return {@link CallContextStats#EMPTY}. + * + * @return the current call-context stats; never {@code null} + */ + @Nonnull + default CallContextStats callContextStats() { + return CallContextStats.EMPTY; + } +``` + +- [ ] **Step 6: Override in `JavaLanguageSupport`** + +In `engine/src/main/java/com/ibm/engine/language/java/JavaLanguageSupport.java`, add the import `import com.ibm.engine.callstack.CallContextStats;` and, next to the `notifyLeaveFile` override (around line 148), add: + +```java + @Override + @Nonnull + public CallContextStats callContextStats() { + return this.handler.callContextStats(); + } +``` + +- [ ] **Step 7: Format, then run the test to verify it passes** + +Run: `mvn spotless:apply -pl engine && mvn test -pl engine -Dtest=JavaLanguageSupportStatsTest` +Expected: PASS (1 test). + +- [ ] **Step 8: Regression + checkstyle + commit** + +Run: `mvn test -pl engine` (whole engine suite still green), then `mvn checkstyle:check -pl engine`. + +```bash +git add engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java \ + engine/src/main/java/com/ibm/engine/detection/Handler.java \ + engine/src/main/java/com/ibm/engine/language/ILanguageSupport.java \ + engine/src/main/java/com/ibm/engine/language/java/JavaLanguageSupport.java \ + engine/src/test/java/com/ibm/engine/language/java/JavaLanguageSupportStatsTest.java +git commit -m "feat(engine): expose callContextStats through Handler/ILanguageSupport" +``` + +--- + +### Task 3: `CryptoCorpusGenerator` (synthetic cross-file corpus) + +A test-only helper in the `java` module that writes N wrapper/caller unit pairs into a directory. Each unit is a `WrapperK` (crypto factory method taking an `algo` parameter → forces a method hook) and a `CallerK` (invokes the wrapper cross-file with a string literal and a field constant → detachable recorded calls). Distinct class names per unit keep call sites distinct so retention grows with corpus size. Uses only JDK `javax.crypto`/`java.security` APIs (JCA, 100% supported) so the corpus compiles with the system compiler and resolves in sonar-java's semantic model. + +**Files:** +- Create: `java/src/test/java/com/ibm/plugin/perf/CryptoCorpusGenerator.java` +- Test: `java/src/test/java/com/ibm/plugin/perf/CryptoCorpusGeneratorTest.java` + +**Interfaces:** +- Produces: `static List CryptoCorpusGenerator.generate(Path root, int units) throws IOException` — writes `2*units` files under `root/perf/` and returns their paths (wrapper then caller, per unit). + +- [ ] **Step 1: Write the failing test** + +Create `java/src/test/java/com/ibm/plugin/perf/CryptoCorpusGeneratorTest.java`: + +```java +package com.ibm.plugin.perf; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; +import javax.tools.JavaCompiler; +import javax.tools.ToolProvider; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +class CryptoCorpusGeneratorTest { + + @Test + void generatesTwoFilesPerUnitThatCompile(@TempDir Path tmp) throws Exception { + List files = CryptoCorpusGenerator.generate(tmp, 3); + + assertThat(files).hasSize(6); + assertThat(files).allSatisfy(p -> assertThat(Files.exists(p)).isTrue()); + + JavaCompiler compiler = ToolProvider.getSystemJavaCompiler(); + List args = new ArrayList<>(List.of("-d", tmp.resolve("out").toString())); + files.forEach(p -> args.add(p.toAbsolutePath().toString())); + int rc = compiler.run(null, null, null, args.toArray(new String[0])); + + assertThat(rc).as("generated corpus must compile cleanly").isZero(); + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `mvn test -pl java -Dtest=CryptoCorpusGeneratorTest` +Expected: FAIL — compilation error, `CryptoCorpusGenerator` does not exist. + +- [ ] **Step 3: Create the generator** + +Create `java/src/test/java/com/ibm/plugin/perf/CryptoCorpusGenerator.java`: + +```java +package com.ibm.plugin.perf; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; +import javax.annotation.Nonnull; + +/** + * Generates a synthetic corpus of cross-file crypto wrapper/caller unit pairs for the call-stack + * heap harness. Each unit is a {@code WrapperK} whose factory method takes the algorithm as a + * parameter (forcing a cross-file method hook) and a {@code CallerK} that invokes it with a string + * literal and a field constant (both detachable recorded calls). Distinct class names per unit keep + * call sites distinct so the recorded-call set grows with corpus size, approximating a large project + * without checking one in. Uses only JDK JCA APIs so the corpus compiles and resolves. + */ +final class CryptoCorpusGenerator { + + private CryptoCorpusGenerator() {} + + /** A JCA factory rotated across units so multiple detection rules fire. */ + private record Api(@Nonnull String call, @Nonnull String algo) {} + + private static final List APIS = + List.of( + new Api("javax.crypto.Cipher.getInstance(algo)", "AES"), + new Api("javax.crypto.KeyGenerator.getInstance(algo)", "AES"), + new Api("java.security.MessageDigest.getInstance(algo)", "SHA-256")); + + @Nonnull + static List generate(@Nonnull Path root, int units) throws IOException { + Path pkg = Files.createDirectories(root.resolve("perf")); + List files = new ArrayList<>(); + for (int i = 0; i < units; i++) { + Api api = APIS.get(i % APIS.size()); + + Path wrapper = pkg.resolve("Wrapper" + i + ".java"); + Files.writeString(wrapper, wrapperSource(i, api)); + files.add(wrapper); + + Path caller = pkg.resolve("Caller" + i + ".java"); + Files.writeString(caller, callerSource(i, api)); + files.add(caller); + } + return files; + } + + @Nonnull + private static String wrapperSource(int i, @Nonnull Api api) { + return "package perf;\n\n" + + "public class Wrapper" + i + " {\n" + + " public Object make(String algo) throws Exception {\n" + + " return " + api.call() + ";\n" + + " }\n" + + "}\n"; + } + + @Nonnull + private static String callerSource(int i, @Nonnull Api api) { + return "package perf;\n\n" + + "public class Caller" + i + " {\n" + + " static final String ALGO = \"" + api.algo() + "\";\n" + + " void run() throws Exception {\n" + + " new Wrapper" + i + "().make(\"" + api.algo() + "\");\n" + + " new Wrapper" + i + "().make(ALGO);\n" + + " }\n" + + "}\n"; + } +} +``` + +- [ ] **Step 4: Format, then run the test to verify it passes** + +Run: `mvn spotless:apply -pl java && mvn test -pl java -Dtest=CryptoCorpusGeneratorTest` +Expected: PASS (1 test). + +- [ ] **Step 5: Checkstyle + commit** + +```bash +mvn checkstyle:check -pl java +git add java/src/test/java/com/ibm/plugin/perf/CryptoCorpusGenerator.java \ + java/src/test/java/com/ibm/plugin/perf/CryptoCorpusGeneratorTest.java +git commit -m "test(java): synthetic cross-file crypto corpus generator" +``` + +--- + +### Task 4: `CallStackHeapPerfTest` + surefire tag exclusion + threshold tuning + +Add the `@Tag("performance")` harness that ties it together, exclude the tag from the default build, tune the assertion thresholds against observed numbers, and validate that the assertion actually discriminates (fails when detach is neutralized). + +**Files:** +- Modify: `pom.xml` (parent — surefire `excludedGroups` via overridable property) +- Create: `java/src/test/java/com/ibm/plugin/perf/CallStackHeapPerfTest.java` + +**Interfaces:** +- Consumes: `CryptoCorpusGenerator.generate(Path, int)` (Task 3); `JavaAggregator.getLanguageSupport().callContextStats()` (Task 2) returning `CallContextStats`; `TestBase` (`asserts(...)` abstract, `report(Tree, List)` overridable → controls issues raised); `CheckVerifier` (`onFiles(Collection)`, `withClassPath(Collection)`, `withChecks(JavaFileScanner...)`, `verifyNoIssues()`). + +- [ ] **Step 1: Make surefire exclude the `performance` tag (overridable)** + +In the parent `pom.xml`, add a property (in the existing `` block) that defaults the exclusion on: + +```xml + performance +``` + +and give the `maven-surefire-plugin` (currently declared with no ``, around line 192) a configuration that reads it: + +```xml + + org.apache.maven.plugins + maven-surefire-plugin + 3.5.4 + + ${excludedGroups} + + +``` + +Effect: `mvn test` excludes `performance` (property from the pom). To run the perf test, a CLI `-DexcludedGroups=` overrides the pom property to empty (CLI system properties win over pom ``), re-including it. + +- [ ] **Step 2: Verify the default build skips tagged tests (no perf test yet — use a temporary probe)** + +Create a throwaway probe to prove exclusion works before writing the real test. Create `java/src/test/java/com/ibm/plugin/perf/TagExclusionProbe.java`: + +```java +package com.ibm.plugin.perf; + +import org.junit.jupiter.api.Tag; +import org.junit.jupiter.api.Test; + +class TagExclusionProbe { + @Tag("performance") + @Test + void taggedTestMustBeExcludedByDefault() { + throw new AssertionError("this @Tag(performance) test must NOT run in the default build"); + } +} +``` + +Run (default build must NOT execute it, so it must stay green): +`mvn test -pl java -Dtest=TagExclusionProbe` +Expected: BUILD SUCCESS with "No tests were executed" (or 0 run) — the tag is excluded. + +Then confirm it DOES run when exclusion is cleared, and fails as written: +`mvn test -pl java -DexcludedGroups= -Dtest=TagExclusionProbe` +Expected: FAIL with the `AssertionError` above — proving `-DexcludedGroups=` re-includes tagged tests. + +Delete the probe: `rm java/src/test/java/com/ibm/plugin/perf/TagExclusionProbe.java` + +- [ ] **Step 3: Commit the surefire change** + +```bash +mvn spotless:apply +git add pom.xml +git commit -m "build: exclude @Tag(performance) tests from default surefire run" +``` + +- [ ] **Step 4: Write the perf harness test** + +Create `java/src/test/java/com/ibm/plugin/perf/CallStackHeapPerfTest.java`. Starting thresholds are intentionally loose (`detachedRatio >= 0.5`, `retainedWithTree <= units`); Step 6 tightens them against observed numbers. + +```java +package com.ibm.plugin.perf; + +import static org.assertj.core.api.Assertions.assertThat; + +import com.ibm.engine.callstack.CallContextStats; +import com.ibm.engine.detection.DetectionStore; +import com.ibm.mapper.model.INode; +import com.ibm.plugin.JavaAggregator; +import com.ibm.plugin.TestBase; +import com.ibm.rules.issue.Issue; +import java.io.File; +import java.lang.management.ManagementFactory; +import java.lang.management.MemoryMXBean; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; +import javax.annotation.Nonnull; +import javax.tools.JavaCompiler; +import javax.tools.ToolProvider; +import org.junit.jupiter.api.Tag; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import org.sonar.java.checks.verifier.CheckVerifier; +import org.sonar.plugins.java.api.JavaCheck; +import org.sonar.plugins.java.api.JavaFileScannerContext; +import org.sonar.plugins.java.api.semantic.Symbol; +import org.sonar.plugins.java.api.tree.Tree; + +/** + * Manual heap/perf harness for the call-stack AST-detach mechanism. Generates a synthetic + * cross-file crypto corpus (scale via {@code -Dperf.corpus.files}, default 200), scans it in-process + * through {@link CheckVerifier} (compiled to a classpath dir so callee types resolve and cross-file + * detections fire), then asserts the recorded calls were detached (ASTs released) at {@code + * leaveFile}. Heap delta and wall-time are printed for manual comparison, never asserted. + * + *

Excluded from the default build via {@code @Tag("performance")}. Run with: + * {@code mvn test -pl java -DexcludedGroups= -Dtest=CallStackHeapPerfTest} (add + * {@code -Dperf.corpus.files=3000} for a heavy soak). This does NOT reproduce the full-project ~7 GB + * keycloak number — that remains the documented manual {@code mvn sonar:sonar} route in + * {@code docs/superpowers/plans/2026-07-05-callstack-hooks-heap-reduction.md}. + */ +@Tag("performance") +class CallStackHeapPerfTest extends TestBase { + + private static final int FILES = Integer.getInteger("perf.corpus.files", 200); + + /** Raise no SonarQube issues, so the harness is decoupled from {@code // Noncompliant} lines. */ + @Override + public List> report(@Nonnull Tree markerTree, @Nonnull List translatedNodes) { + return List.of(); + } + + @Override + public void asserts( + int findingId, + @Nonnull DetectionStore detectionStore, + @Nonnull List nodes) { + // no per-finding assertions; the harness asserts on aggregate CallContextStats below + } + + @Test + void detachesRecordedCallsAtScale(@TempDir Path tmp) throws Exception { + int units = Math.max(1, FILES / 2); + List sources = CryptoCorpusGenerator.generate(tmp, units); + File classes = compileToClasspath(sources); + + MemoryMXBean mem = ManagementFactory.getMemoryMXBean(); + long heapBefore = usedHeapAfterGc(mem); + long start = System.nanoTime(); + + CheckVerifier.newVerifier() + .onFiles(absolutePaths(sources)) + .withClassPath(List.of(classes)) + .withChecks(this) + .verifyNoIssues(); + + long elapsedMs = (System.nanoTime() - start) / 1_000_000L; + long heapAfter = usedHeapAfterGc(mem); + + CallContextStats stats = JavaAggregator.getLanguageSupport().callContextStats(); + + // REPORT (never asserted) + System.out.printf( + "%n[callstack-perf] files=%d units=%d time=%dms heapDeltaMB=%d " + + "retainedWithTree=%d detached=%d total=%d buckets=%d ratio=%.3f%n", + sources.size(), + units, + elapsedMs, + (heapAfter - heapBefore) / (1024L * 1024L), + stats.retainedWithTree(), + stats.detached(), + stats.total(), + stats.buckets(), + stats.detachedRatio()); + + // ASSERT (deterministic gate — object-variant counts, no heap/time dependence) + assertThat(stats.total()) + .as("detections must fire (compiled classpath) or the harness proves nothing") + .isPositive(); + assertThat(stats.detachedRatio()) + .as("most recorded calls must be detached (ASTs released at leaveFile)") + .isGreaterThanOrEqualTo(0.5d); + assertThat(stats.retainedWithTree()) + .as("tree-pinning calls must stay bounded, not grow ~1:1 with detached") + .isLessThanOrEqualTo(units); + } + + private static File compileToClasspath(@Nonnull List sources) throws Exception { + Path out = Files.createTempDirectory("perf-corpus-classes"); + JavaCompiler compiler = ToolProvider.getSystemJavaCompiler(); + List args = new ArrayList<>(List.of("-d", out.toString())); + for (Path s : sources) { + args.add(s.toAbsolutePath().toString()); + } + int rc = compiler.run(null, null, null, args.toArray(new String[0])); + if (rc != 0) { + throw new IllegalStateException("corpus compilation failed rc=" + rc); + } + return out.toFile(); + } + + @Nonnull + private static List absolutePaths(@Nonnull List sources) { + List paths = new ArrayList<>(sources.size()); + for (Path s : sources) { + paths.add(s.toAbsolutePath().toString()); + } + return paths; + } + + private static long usedHeapAfterGc(@Nonnull MemoryMXBean mem) { + System.gc(); + try { + Thread.sleep(50L); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + return mem.getHeapMemoryUsage().getUsed(); + } +} +``` + +Note: `report(...)` overrides `JavaInventoryRule.report(Tree, List)` which returns `List>` (`java/src/main/java/com/ibm/plugin/rules/JavaInventoryRule.java:49`), so the import above is `com.ibm.rules.issue.Issue`. Returning `List.of()` raises no issues, so `verifyNoIssues()` passes regardless of detections. + +- [ ] **Step 5: Run the perf test to verify it passes on this branch (detach live)** + +Run: `mvn spotless:apply -pl java && mvn test -pl java -DexcludedGroups= -Dtest=CallStackHeapPerfTest` +Expected: PASS. Capture the printed `[callstack-perf] ...` line. Sanity-check it: `total` positive (detections fired), `detached` is the large majority, `retainedWithTree` small. + +If `total == 0`: types did not resolve — verify `withClassPath(classes)` is present and the corpus compiled (the `compileToClasspath` step threw nothing). If `onFiles` rejects the absolute temp paths, generate under a module-relative dir instead (e.g. `Path root = Path.of("target", "perf-corpus");` cleaned at start) and pass module-relative paths. + +- [ ] **Step 6: Tighten thresholds to the observed numbers** + +Using the captured line, tighten the two assertions so they still pass comfortably on this branch but leave no room for a regression to slip through: +- Set `detachedRatio` lower bound just below the observed ratio (e.g. observed 0.98 → assert `>= 0.9`). +- Set the `retainedWithTree` bound to a small multiple of the observed value, still well under `units` and NOT proportional to `total` (e.g. observed 3 with units 100 → assert `<= 20`). + +Edit the two `assertThat` bounds accordingly. Re-run Step 5's command; expected PASS. + +- [ ] **Step 7: Validate the assertion discriminates (neutralize detach → must FAIL)** + +This proves the test is worth committing. Temporarily make detach a no-op — comment out the body of `CallStackAgent.detachCallsForFile` (or of `JavaLanguageSupport.notifyLeaveFile`) so no call is ever swapped to its detached form: + +Run: `mvn test -pl java -DexcludedGroups= -Dtest=CallStackHeapPerfTest` +Expected: **FAIL** — `retainedWithTree` now grows with the corpus (≈ `total`) and `detachedRatio` collapses toward 0, tripping both assertions. Capture the failing line to confirm. + +**Revert the neutralization** (`git checkout -- engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java`) and re-run Step 5 to confirm PASS again. The neutralization is a local sanity check and is **NOT committed**. + +- [ ] **Step 8: Confirm the default build still skips it, then commit** + +Run: `mvn test -pl java -Dtest=CallStackHeapPerfTest` (default build) +Expected: BUILD SUCCESS, 0 tests run (tag excluded) — CI-safe. + +```bash +mvn checkstyle:check -pl java +git add java/src/test/java/com/ibm/plugin/perf/CallStackHeapPerfTest.java +git commit -m "test(java): manual call-stack heap/perf harness (@Tag performance)" +``` + +--- + +## Self-Review + +**Spec coverage:** +- Self-contained JUnit, no docker/network → Tasks 3–4 (generator + in-process `CheckVerifier`). ✓ +- Not keycloak; synthetic corpus, scale knob → Task 3 generator, `-Dperf.corpus.files` (Task 4). ✓ +- Types must resolve → Task 4 `compileToClasspath` + `withClassPath` (mirrors `CrossFileHookDetachTest`). ✓ +- Wrapper/caller cross-file pattern with literal + field-constant (detachable) args → Task 3 templates. ✓ (Note: the design's `new byte[]` retained case is intentionally omitted from the generator — the primary signal is a high detach ratio + bounded tree-pinning; the retained path is already covered functionally by `CrossFileHookDetachTest`. Recorded in the spec's assertion section.) +- Structural assert (`retainedWithTree` bounded, `detachedRatio` high) via `callContextStats()` → Tasks 1–2 accessor, Task 4 asserts. ✓ +- Heap/time report-only, never asserted → Task 4 `System.out.printf`, no assertion on `heapDelta`/`elapsedMs`. ✓ +- Small `callContextStats()` accessor, not reflection → Tasks 1–2. ✓ +- Manual only, `@Tag("performance")` + surefire `excludedGroups` → Task 4 Steps 1–3, 8. ✓ +- Placement (`com.ibm.plugin.perf`, engine `callstack`/`detection`/`language`) → matches approved layout. ✓ +- Harness self-validation (passes with detach, fails without) → Task 4 Step 7. ✓ +- Pointer to the manual keycloak `mvn sonar:sonar` route, not automated → Task 4 test javadoc. ✓ + +**Placeholder scan:** No TBD/TODO. The two threshold values are concrete starting numbers (`0.5`, `units`) with an explicit, evidence-based tightening step (Task 4 Step 6) — not placeholders. All imports are exact (`Issue` verified as `com.ibm.rules.issue.Issue`). + +**Type consistency:** `callContextStats()` returns `CallContextStats` at every layer (`CallStackAgent`, `Handler`, `ILanguageSupport`, `JavaLanguageSupport`). `CallContextStats.from(Collection>>)` matches `invokedCallStack.values()`'s type (`Collection>>`). `generate(Path, int) -> List` consumed consistently in Tasks 3 and 4. `detachedRatio()`, `retainedWithTree()`, `detached()`, `total()`, `buckets()` accessor names consistent between Task 1 definition and Task 4 use. From 77bff0a9dd40ec310cedaf994bee191a7028f6c9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Mon, 6 Jul 2026 09:16:47 +0200 Subject: [PATCH 19/35] feat(engine): CallContextStats accessor for call-stack retention MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- .../engine/callstack/CallContextStats.java | 63 +++++++++++++++++++ .../callstack/CallContextStatsTest.java | 58 +++++++++++++++++ 2 files changed, 121 insertions(+) create mode 100644 engine/src/main/java/com/ibm/engine/callstack/CallContextStats.java create mode 100644 engine/src/test/java/com/ibm/engine/callstack/CallContextStatsTest.java diff --git a/engine/src/main/java/com/ibm/engine/callstack/CallContextStats.java b/engine/src/main/java/com/ibm/engine/callstack/CallContextStats.java new file mode 100644 index 00000000..77c8f45a --- /dev/null +++ b/engine/src/main/java/com/ibm/engine/callstack/CallContextStats.java @@ -0,0 +1,63 @@ +/* + * Sonar Cryptography Plugin + * Copyright (C) 2026 PQCA + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.ibm.engine.callstack; + +import java.util.Collection; +import java.util.List; +import javax.annotation.Nonnull; + +/** + * Immutable snapshot of the {@link CallStackAgent}'s recorded-call population, used by the + * performance/heap harness to assert that calls were detached (ASTs released) at {@code leaveFile}. + * + * @param retainedWithTree number of {@link RetainedCall}s still pinning a live AST tree + * @param detached number of tree-free {@link DetachedCall}s + * @param total {@code retainedWithTree + detached} + * @param buckets number of hash buckets in the call stack + */ +public record CallContextStats(int retainedWithTree, int detached, int total, int buckets) { + + /** A zero-valued snapshot (used as the language-agnostic default). */ + public static final CallContextStats EMPTY = new CallContextStats(0, 0, 0, 0); + + /** Fraction of recorded calls that are detached; {@code 1.0} when nothing was recorded. */ + public double detachedRatio() { + return total == 0 ? 1.0d : (double) detached / (double) total; + } + + /** Counts retained-with-tree vs. detached calls across all call-stack buckets. */ + @Nonnull + public static CallContextStats from( + @Nonnull Collection>> buckets) { + int retainedWithTree = 0; + int detached = 0; + for (List> bucket : buckets) { + for (CallContext call : bucket) { + if (call instanceof DetachedCall) { + detached++; + } else if (call instanceof RetainedCall retained && retained.tree() != null) { + retainedWithTree++; + } + } + } + return new CallContextStats( + retainedWithTree, detached, retainedWithTree + detached, buckets.size()); + } +} diff --git a/engine/src/test/java/com/ibm/engine/callstack/CallContextStatsTest.java b/engine/src/test/java/com/ibm/engine/callstack/CallContextStatsTest.java new file mode 100644 index 00000000..c386f23a --- /dev/null +++ b/engine/src/test/java/com/ibm/engine/callstack/CallContextStatsTest.java @@ -0,0 +1,58 @@ +/* + * Sonar Cryptography Plugin + * Copyright (C) 2026 PQCA + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.ibm.engine.callstack; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.within; +import static org.mockito.Mockito.mock; + +import com.ibm.engine.language.IScanContext; +import java.util.List; +import org.junit.jupiter.api.Test; + +class CallContextStatsTest { + + @Test + @SuppressWarnings("unchecked") + void countsRetainedAndDetachedAcrossBuckets() { + IScanContext ctx = mock(IScanContext.class); + RetainedCall retained1 = new RetainedCall<>("t1", ctx, null); + RetainedCall retained2 = new RetainedCall<>("t2", ctx, null); + DetachedCall detached = mock(DetachedCall.class); + + List>> buckets = + List.of(List.of(retained1, detached), List.of(retained2)); + + CallContextStats stats = CallContextStats.from(buckets); + + assertThat(stats.retainedWithTree()).isEqualTo(2); + assertThat(stats.detached()).isEqualTo(1); + assertThat(stats.total()).isEqualTo(3); + assertThat(stats.buckets()).isEqualTo(2); + assertThat(stats.detachedRatio()).isCloseTo(1.0d / 3.0d, within(1e-9)); + } + + @Test + void emptyHasZeroCountsAndFullRatio() { + assertThat(CallContextStats.EMPTY.total()).isZero(); + assertThat(CallContextStats.EMPTY.buckets()).isZero(); + assertThat(CallContextStats.EMPTY.detachedRatio()).isEqualTo(1.0d); + } +} From d2aa0fcd2924f4fcd68a39641abb42a0a9bebf63 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Mon, 6 Jul 2026 09:18:38 +0200 Subject: [PATCH 20/35] feat(engine): expose callContextStats through Handler/ILanguageSupport MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- .../ibm/engine/callstack/CallStackAgent.java | 6 +++ .../com/ibm/engine/detection/Handler.java | 6 +++ .../ibm/engine/language/ILanguageSupport.java | 12 +++++ .../language/java/JavaLanguageSupport.java | 7 +++ .../java/JavaLanguageSupportStatsTest.java | 47 +++++++++++++++++++ 5 files changed, 78 insertions(+) create mode 100644 engine/src/test/java/com/ibm/engine/language/java/JavaLanguageSupportStatsTest.java diff --git a/engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java b/engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java index fbb814b8..42ded8e7 100644 --- a/engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java +++ b/engine/src/main/java/com/ibm/engine/callstack/CallStackAgent.java @@ -73,6 +73,12 @@ public void detachCallsForFile(@Nonnull InputFile inputFile) { } } + /** Read-only snapshot of the recorded-call population (retained-with-tree vs. detached). */ + @Nonnull + public CallContextStats callContextStats() { + return CallContextStats.from(invokedCallStack.values()); + } + /** Records a call (retained or detached) and notifies live hook subscriptions. */ public void add(@Nonnull CallContext callContext) { final Optional keyOptional = keyOf(callContext); diff --git a/engine/src/main/java/com/ibm/engine/detection/Handler.java b/engine/src/main/java/com/ibm/engine/detection/Handler.java index 370b36c1..01546496 100644 --- a/engine/src/main/java/com/ibm/engine/detection/Handler.java +++ b/engine/src/main/java/com/ibm/engine/detection/Handler.java @@ -21,6 +21,7 @@ import com.ibm.common.IObserver; import com.ibm.engine.callstack.CallContext; +import com.ibm.engine.callstack.CallContextStats; import com.ibm.engine.callstack.CallStackAgent; import com.ibm.engine.hooks.HookDetectionObservable; import com.ibm.engine.hooks.HookRepository; @@ -90,6 +91,11 @@ public void detachCallsForFile(@Nonnull InputFile inputFile) { this.callStackAgent.detachCallsForFile(inputFile); } + @Nonnull + public CallContextStats callContextStats() { + return this.callStackAgent.callContextStats(); + } + public void subscribeToCallStackAgent(@Nonnull IObserver> listener) { this.callStackAgent.subscribe(listener); } diff --git a/engine/src/main/java/com/ibm/engine/language/ILanguageSupport.java b/engine/src/main/java/com/ibm/engine/language/ILanguageSupport.java index a0e26939..820f5def 100644 --- a/engine/src/main/java/com/ibm/engine/language/ILanguageSupport.java +++ b/engine/src/main/java/com/ibm/engine/language/ILanguageSupport.java @@ -19,6 +19,7 @@ */ package com.ibm.engine.language; +import com.ibm.engine.callstack.CallContextStats; import com.ibm.engine.detection.DetectionStore; import com.ibm.engine.detection.EnumMatcher; import com.ibm.engine.detection.IBaseMethodVisitorFactory; @@ -155,4 +156,15 @@ default int parameterIndexOf(@Nonnull T methodDefinition, @Nonnull T methodParam default void notifyLeaveFile(@Nonnull org.sonar.api.batch.fs.InputFile inputFile) { // no-op by default } + + /** + * Read-only snapshot of the language's recorded-call population, for the performance/heap + * harness. Languages that do not accumulate a call stack return {@link CallContextStats#EMPTY}. + * + * @return the current call-context stats; never {@code null} + */ + @Nonnull + default CallContextStats callContextStats() { + return CallContextStats.EMPTY; + } } diff --git a/engine/src/main/java/com/ibm/engine/language/java/JavaLanguageSupport.java b/engine/src/main/java/com/ibm/engine/language/java/JavaLanguageSupport.java index ce197f16..1ad83ad9 100644 --- a/engine/src/main/java/com/ibm/engine/language/java/JavaLanguageSupport.java +++ b/engine/src/main/java/com/ibm/engine/language/java/JavaLanguageSupport.java @@ -19,6 +19,7 @@ */ package com.ibm.engine.language.java; +import com.ibm.engine.callstack.CallContextStats; import com.ibm.engine.detection.DetectionStore; import com.ibm.engine.detection.EnumMatcher; import com.ibm.engine.detection.Handler; @@ -149,6 +150,12 @@ public void notifyLeaveFile(@Nonnull org.sonar.api.batch.fs.InputFile inputFile) this.handler.detachCallsForFile(inputFile); } + @Override + @Nonnull + public CallContextStats callContextStats() { + return this.handler.callContextStats(); + } + @Override public boolean isDetachableCall(@Nonnull Tree tree) { if (!(tree instanceof MethodInvocationTree invocation)) { diff --git a/engine/src/test/java/com/ibm/engine/language/java/JavaLanguageSupportStatsTest.java b/engine/src/test/java/com/ibm/engine/language/java/JavaLanguageSupportStatsTest.java new file mode 100644 index 00000000..3b6b45b8 --- /dev/null +++ b/engine/src/test/java/com/ibm/engine/language/java/JavaLanguageSupportStatsTest.java @@ -0,0 +1,47 @@ +/* + * Sonar Cryptography Plugin + * Copyright (C) 2026 PQCA + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.ibm.engine.language.java; + +import static org.assertj.core.api.Assertions.assertThat; + +import com.ibm.engine.callstack.CallContextStats; +import com.ibm.engine.language.ILanguageSupport; +import com.ibm.engine.language.LanguageSupporter; +import org.junit.jupiter.api.Test; +import org.sonar.plugins.java.api.JavaCheck; +import org.sonar.plugins.java.api.JavaFileScannerContext; +import org.sonar.plugins.java.api.semantic.Symbol; +import org.sonar.plugins.java.api.tree.Tree; + +class JavaLanguageSupportStatsTest { + + @Test + void freshSupportReportsEmptyStatsThroughTheChain() { + ILanguageSupport support = + LanguageSupporter.javaLanguageSupporter(); + + CallContextStats stats = support.callContextStats(); + + assertThat(stats).isNotNull(); + assertThat(stats.total()).isZero(); + assertThat(stats.retainedWithTree()).isZero(); + assertThat(stats.detached()).isZero(); + } +} From 7cc54ba9dcf1c3696dc0a73e61b268731d3ce705 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Mon, 6 Jul 2026 09:19:45 +0200 Subject: [PATCH 21/35] test(java): synthetic cross-file crypto corpus generator MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- .../plugin/perf/CryptoCorpusGenerator.java | 103 ++++++++++++++++++ .../perf/CryptoCorpusGeneratorTest.java | 49 +++++++++ 2 files changed, 152 insertions(+) create mode 100644 java/src/test/java/com/ibm/plugin/perf/CryptoCorpusGenerator.java create mode 100644 java/src/test/java/com/ibm/plugin/perf/CryptoCorpusGeneratorTest.java diff --git a/java/src/test/java/com/ibm/plugin/perf/CryptoCorpusGenerator.java b/java/src/test/java/com/ibm/plugin/perf/CryptoCorpusGenerator.java new file mode 100644 index 00000000..7884133d --- /dev/null +++ b/java/src/test/java/com/ibm/plugin/perf/CryptoCorpusGenerator.java @@ -0,0 +1,103 @@ +/* + * Sonar Cryptography Plugin + * Copyright (C) 2026 PQCA + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.ibm.plugin.perf; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; +import javax.annotation.Nonnull; + +/** + * Generates a synthetic corpus of cross-file crypto wrapper/caller unit pairs for the call-stack + * heap harness. Each unit is a {@code WrapperK} whose factory method takes the algorithm as a + * parameter (forcing a cross-file method hook) and a {@code CallerK} that invokes it with a string + * literal and a field constant (both detachable recorded calls). Distinct class names per unit keep + * call sites distinct so the recorded-call set grows with corpus size, approximating a large + * project without checking one in. Uses only JDK JCA APIs so the corpus compiles and resolves. + */ +final class CryptoCorpusGenerator { + + private CryptoCorpusGenerator() {} + + /** A JCA factory rotated across units so multiple detection rules fire. */ + private record Api(@Nonnull String call, @Nonnull String algo) {} + + private static final List APIS = + List.of( + new Api("javax.crypto.Cipher.getInstance(algo)", "AES"), + new Api("javax.crypto.KeyGenerator.getInstance(algo)", "AES"), + new Api("java.security.MessageDigest.getInstance(algo)", "SHA-256")); + + @Nonnull + static List generate(@Nonnull Path root, int units) throws IOException { + Path pkg = Files.createDirectories(root.resolve("perf")); + List files = new ArrayList<>(); + for (int i = 0; i < units; i++) { + Api api = APIS.get(i % APIS.size()); + + Path wrapper = pkg.resolve("Wrapper" + i + ".java"); + Files.writeString(wrapper, wrapperSource(i, api)); + files.add(wrapper); + + Path caller = pkg.resolve("Caller" + i + ".java"); + Files.writeString(caller, callerSource(i, api)); + files.add(caller); + } + return files; + } + + @Nonnull + private static String wrapperSource(int i, @Nonnull Api api) { + return "package perf;\n\n" + + "public class Wrapper" + + i + + " {\n" + + " public Object make(String algo) throws Exception {\n" + + " return " + + api.call() + + ";\n" + + " }\n" + + "}\n"; + } + + @Nonnull + private static String callerSource(int i, @Nonnull Api api) { + return "package perf;\n\n" + + "public class Caller" + + i + + " {\n" + + " static final String ALGO = \"" + + api.algo() + + "\";\n" + + " void run() throws Exception {\n" + + " new Wrapper" + + i + + "().make(\"" + + api.algo() + + "\");\n" + + " new Wrapper" + + i + + "().make(ALGO);\n" + + " }\n" + + "}\n"; + } +} diff --git a/java/src/test/java/com/ibm/plugin/perf/CryptoCorpusGeneratorTest.java b/java/src/test/java/com/ibm/plugin/perf/CryptoCorpusGeneratorTest.java new file mode 100644 index 00000000..ae698643 --- /dev/null +++ b/java/src/test/java/com/ibm/plugin/perf/CryptoCorpusGeneratorTest.java @@ -0,0 +1,49 @@ +/* + * Sonar Cryptography Plugin + * Copyright (C) 2026 PQCA + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.ibm.plugin.perf; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; +import javax.tools.JavaCompiler; +import javax.tools.ToolProvider; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +class CryptoCorpusGeneratorTest { + + @Test + void generatesTwoFilesPerUnitThatCompile(@TempDir Path tmp) throws Exception { + List files = CryptoCorpusGenerator.generate(tmp, 3); + + assertThat(files).hasSize(6); + assertThat(files).allSatisfy(p -> assertThat(Files.exists(p)).isTrue()); + + JavaCompiler compiler = ToolProvider.getSystemJavaCompiler(); + List args = new ArrayList<>(List.of("-d", tmp.resolve("out").toString())); + files.forEach(p -> args.add(p.toAbsolutePath().toString())); + int rc = compiler.run(null, null, null, args.toArray(new String[0])); + + assertThat(rc).as("generated corpus must compile cleanly").isZero(); + } +} From cca8a5cd3a92297c2f50d10c611460fdc541ea42 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Mon, 6 Jul 2026 09:20:52 +0200 Subject: [PATCH 22/35] build: exclude @Tag(performance) tests from default surefire run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- pom.xml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/pom.xml b/pom.xml index 61484a52..7c1a806e 100644 --- a/pom.xml +++ b/pom.xml @@ -40,6 +40,9 @@ 17 UTF-8 + + performance + 2.9.1 10.15.0 @@ -193,6 +196,9 @@ org.apache.maven.plugins maven-surefire-plugin 3.5.4 + + ${excludedGroups} + com.diffplug.spotless From 7f03c6273c944a40dee33d4eda592552bd292c86 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Mon, 6 Jul 2026 09:25:18 +0200 Subject: [PATCH 23/35] test(java): manual call-stack heap/perf harness (@Tag performance) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- .../plugin/perf/CallStackHeapPerfTest.java | 166 ++++++++++++++++++ 1 file changed, 166 insertions(+) create mode 100644 java/src/test/java/com/ibm/plugin/perf/CallStackHeapPerfTest.java diff --git a/java/src/test/java/com/ibm/plugin/perf/CallStackHeapPerfTest.java b/java/src/test/java/com/ibm/plugin/perf/CallStackHeapPerfTest.java new file mode 100644 index 00000000..f61d92e7 --- /dev/null +++ b/java/src/test/java/com/ibm/plugin/perf/CallStackHeapPerfTest.java @@ -0,0 +1,166 @@ +/* + * Sonar Cryptography Plugin + * Copyright (C) 2026 PQCA + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.ibm.plugin.perf; + +import static org.assertj.core.api.Assertions.assertThat; + +import com.ibm.engine.callstack.CallContextStats; +import com.ibm.engine.detection.DetectionStore; +import com.ibm.mapper.model.INode; +import com.ibm.plugin.JavaAggregator; +import com.ibm.plugin.TestBase; +import com.ibm.rules.issue.Issue; +import java.io.File; +import java.lang.management.ManagementFactory; +import java.lang.management.MemoryMXBean; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; +import javax.annotation.Nonnull; +import javax.tools.JavaCompiler; +import javax.tools.ToolProvider; +import org.junit.jupiter.api.Tag; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import org.sonar.java.checks.verifier.CheckVerifier; +import org.sonar.plugins.java.api.JavaCheck; +import org.sonar.plugins.java.api.JavaFileScannerContext; +import org.sonar.plugins.java.api.semantic.Symbol; +import org.sonar.plugins.java.api.tree.Tree; + +/** + * Manual heap/perf harness for the call-stack AST-detach mechanism. Generates a synthetic + * cross-file crypto corpus (scale via {@code -Dperf.corpus.files}, default 200), scans it + * in-process through {@link CheckVerifier} (compiled to a classpath dir so callee types resolve and + * cross-file detections fire), then asserts the recorded calls were detached (ASTs released) at + * {@code leaveFile}. Heap delta and wall-time are printed for manual comparison, never asserted. + * + *

Excluded from the default build via {@code @Tag("performance")}. Run with: {@code mvn test -pl + * java -DexcludedGroups= -Dtest=CallStackHeapPerfTest} (add {@code -Dperf.corpus.files=3000} for a + * heavy soak). This does NOT reproduce the full-project ~7 GB keycloak number — that remains the + * documented manual {@code mvn sonar:sonar} route in {@code + * docs/superpowers/plans/2026-07-05-callstack-hooks-heap-reduction.md}. + */ +@Tag("performance") +class CallStackHeapPerfTest extends TestBase { + + private static final int FILES = Integer.getInteger("perf.corpus.files", 200); + + /** + * Raise no SonarQube issues, so the harness is decoupled from {@code // Noncompliant} lines. + */ + @Override + public List> report( + @Nonnull Tree markerTree, @Nonnull List translatedNodes) { + return List.of(); + } + + @Override + public void asserts( + int findingId, + @Nonnull DetectionStore detectionStore, + @Nonnull List nodes) { + // no per-finding assertions; the harness asserts on aggregate CallContextStats below + } + + @Test + void detachesRecordedCallsAtScale(@TempDir Path tmp) throws Exception { + int units = Math.max(1, FILES / 2); + List sources = CryptoCorpusGenerator.generate(tmp, units); + File classes = compileToClasspath(sources); + + MemoryMXBean mem = ManagementFactory.getMemoryMXBean(); + long heapBefore = usedHeapAfterGc(mem); + long start = System.nanoTime(); + + CheckVerifier.newVerifier() + .onFiles(absolutePaths(sources)) + .withClassPath(List.of(classes)) + .withChecks(this) + .verifyNoIssues(); + + long elapsedMs = (System.nanoTime() - start) / 1_000_000L; + long heapAfter = usedHeapAfterGc(mem); + + CallContextStats stats = JavaAggregator.getLanguageSupport().callContextStats(); + + // REPORT (never asserted) + System.out.printf( + "%n[callstack-perf] files=%d units=%d time=%dms heapDeltaMB=%d " + + "retainedWithTree=%d detached=%d total=%d buckets=%d ratio=%.3f%n", + sources.size(), + units, + elapsedMs, + (heapAfter - heapBefore) / (1024L * 1024L), + stats.retainedWithTree(), + stats.detached(), + stats.total(), + stats.buckets(), + stats.detachedRatio()); + + // ASSERT (deterministic gate — object-variant counts, no heap/time dependence) + assertThat(stats.total()) + .as("detections must fire (compiled classpath) or the harness proves nothing") + .isPositive(); + // Observed on feat/callstack-ast-detach (200 files): ratio=1.000, retainedWithTree=0. + // With detach disabled these collapse to ratio~0 and retainedWithTree~total, so the + // margins below stay comfortably true here yet fail hard on any regression. + assertThat(stats.detachedRatio()) + .as("most recorded calls must be detached (ASTs released at leaveFile)") + .isGreaterThanOrEqualTo(0.9d); + assertThat(stats.retainedWithTree()) + .as("tree-pinning calls must stay bounded, not grow ~1:1 with detached") + .isLessThanOrEqualTo(10); + } + + private static File compileToClasspath(@Nonnull List sources) throws Exception { + Path out = Files.createTempDirectory("perf-corpus-classes"); + JavaCompiler compiler = ToolProvider.getSystemJavaCompiler(); + List args = new ArrayList<>(List.of("-d", out.toString())); + for (Path s : sources) { + args.add(s.toAbsolutePath().toString()); + } + int rc = compiler.run(null, null, null, args.toArray(new String[0])); + if (rc != 0) { + throw new IllegalStateException("corpus compilation failed rc=" + rc); + } + return out.toFile(); + } + + @Nonnull + private static List absolutePaths(@Nonnull List sources) { + List paths = new ArrayList<>(sources.size()); + for (Path s : sources) { + paths.add(s.toAbsolutePath().toString()); + } + return paths; + } + + private static long usedHeapAfterGc(@Nonnull MemoryMXBean mem) { + System.gc(); + try { + Thread.sleep(50L); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + return mem.getHeapMemoryUsage().getUsed(); + } +} From 5a5df34734f82937bf3124aa5c96f04c3313251b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Mon, 6 Jul 2026 11:06:03 +0200 Subject: [PATCH 24/35] docs: performance & heap testing guide (self-contained harness + Keycloak scan) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- README.md | 2 + docs/PERFORMANCE_TESTING.md | 298 ++++++++++++++++++++++++++++++++++++ 2 files changed, 300 insertions(+) create mode 100644 docs/PERFORMANCE_TESTING.md diff --git a/README.md b/README.md index c3e542cf..4055c9a1 100644 --- a/README.md +++ b/README.md @@ -293,6 +293,8 @@ Run with `go run gen_package.go`, then delete the script. If you encounter difficulties or unexpected results while installing the plugin with SonarQube, or when trying to scan a repository, please check out our guide [*Testing your configuration and troubleshooting*](docs/TROUBLESHOOTING.md) to run our plugin with step-by-step instructions. +To measure the plugin's runtime performance and heap usage — including a full end-to-end scan of a large project (Keycloak) — see [*Performance & Heap Testing*](docs/PERFORMANCE_TESTING.md). + ## Contribution Guidelines If you'd like to contribute to Sonar Cryptography Plugin, please take a look at our diff --git a/docs/PERFORMANCE_TESTING.md b/docs/PERFORMANCE_TESTING.md new file mode 100644 index 00000000..e957517c --- /dev/null +++ b/docs/PERFORMANCE_TESTING.md @@ -0,0 +1,298 @@ +# Performance & Heap Testing + +This guide explains how to measure the plugin's runtime performance and heap usage, +primarily by scanning a large real-world project (**Keycloak**) end-to-end through +SonarQube. It exists to validate memory-sensitive engine work such as the call-stack +AST-detach fix (see `docs/superpowers/plans/2026-07-05-callstack-hooks-heap-reduction.md`). + +There are two levels of testing: + +| Level | What it is | When to use | +|---|---|---| +| **A. Self-contained JUnit harness** | `CallStackHeapPerfTest` — generates a synthetic cross-file corpus, scans it in-process, asserts detach invariants. No Docker, no network. | Quick, deterministic regression check. Runs in seconds–minutes. | +| **B. Keycloak end-to-end scan** | Full `mvn sonar:sonar` of Keycloak against a local SonarQube with the plugin installed. | Realistic heap/time numbers on a large project. Manual, ~10–15 min + setup. | + +Start with **A** for a fast signal; use **B** to get true, large-project numbers. + +--- + +## A. Self-contained JUnit harness (fast) + +A `@Tag("performance")` test excluded from the default build. It generates N synthetic +cross-file crypto wrapper/caller pairs, compiles them to a classpath directory (so types +resolve and detections fire), scans them with `CheckVerifier`, and asserts that recorded +calls were detached (ASTs released) at `leaveFile`. + +```bash +# default (~200 files, a few seconds) +mvn test -pl java -DexcludedGroups= -Dtest=CallStackHeapPerfTest + +# heavy soak (scale the corpus) +mvn test -pl java -DexcludedGroups= -Dtest=CallStackHeapPerfTest -Dperf.corpus.files=3000 +``` + +It prints a line like: + +``` +[callstack-perf] files=200 units=100 time=2325ms heapDeltaMB=83 \ + retainedWithTree=0 detached=367 total=367 buckets=2 ratio=1.000 +``` + +- **`retainedWithTree`** — recorded calls still pinning a live AST. Must stay ~0. +- **`detached` / `ratio`** — calls converted to tree-free records. Ratio must stay high. +- **`heapDeltaMB`** — reported only, **never asserted** (a coarse whole-JVM number; at this + synthetic scale it is *not* a reliable proxy for AST-pinning savings — the generated files + are tiny, so pinned ASTs cost little in bytes). Use Keycloak (Part B) for real byte numbers. + +The assertions (`ratio >= 0.9`, `retainedWithTree <= 10`) fail hard if AST-detaching regresses. + +> **Why not CI?** It is `@Tag("performance")` and excluded from the default build via the +> surefire `performance` config in the root `pom.xml`. +> `mvn test` skips it; `-DexcludedGroups=` re-includes it. + +--- + +## B. Keycloak end-to-end scan (real numbers) + +### Prerequisites + +- **Docker** + **Docker Compose** (for the local SonarQube + PostgreSQL). +- **JDK 17** to build this plugin; **the JDK Keycloak requires** to build Keycloak + (currently **JDK 21** for `main`). +- **Maven 3.9+**. +- A full **JDK** (not just a JRE) on `PATH` for `jcmd` (used to sample heap — see Step 6). +- ~10 GB free RAM and a few GB disk. The scanner engine is capped at 6 GB in Step 5. + +All commands below assume the plugin repo root is the current directory unless stated. + +--- + +### Step 1 — Clone Keycloak + +```bash +cd ~/Downloads # or any working directory outside this repo +git clone --depth 1 https://github.com/keycloak/keycloak.git keycloak-main +cd keycloak-main +``` + +`--depth 1` is fine — we only need the source tree, not history. + +--- + +### Step 2 — Build Keycloak (REQUIRED — compile so types resolve) + +**This is the most important step.** The heap term only appears when detections actually +fire, and detections only fire when **types resolve**. A source-only scan (no compiled +classes / `sonar.java.binaries`) resolves nothing → 0 detections → the call-stack term is 0 +and the measurement is meaningless. + +Build Keycloak so every module produces `target/classes`: + +```bash +# from the keycloak-main directory; uses Keycloak's Maven wrapper +./mvnw -q clean install -DskipTests -DskipITs +``` + +This is heavy (many modules) and needs Keycloak's required JDK. When it finishes you should +have compiled classes: + +```bash +find . -type d -path '*target/classes' | wc -l # expect dozens of dirs +``` + +If you only want a subset, you must still compile the modules you intend to scan (and their +dependencies), e.g. `./mvnw -pl services -am -DskipTests clean install`. + +--- + +### Step 3 — Build the plugin and start SonarQube with it installed + +SonarQube loads plugins **at startup**, so the JAR must be in place and the container +(re)started for changes to take effect. + +**3a. Build the plugin JAR from your branch:** + +```bash +# from the plugin repo root +mvn clean package -DskipTests +ls -la sonar-cryptography-plugin/target/sonar-cryptography-plugin-*.jar # the artifact +``` + +> If `mvn` reformats/regenerates `mapper/.../JsonCipherSuites.java` (a known Spotless/fetch +> quirk), restore it before committing: `git checkout -- mapper/src/main/java/com/ibm/mapper/mapper/ssl/json/JsonCipherSuites.java`. + +**3b. Deploy the JAR into the compose-mounted plugins directory:** + +```bash +rm -f .SonarQube/plugins/sonar-cryptography-plugin-*.jar +cp sonar-cryptography-plugin/target/sonar-cryptography-plugin-*.jar .SonarQube/plugins/ +``` + +(Do **not** copy the `*-sources.jar` or `original-*.jar`.) + +**3c. Start SonarQube + PostgreSQL.** `docker-compose.yaml` uses `user: "${UID}"`, and in +zsh `UID` is a read-only variable that is *not* exported — if it is blank the container runs +as root and Elasticsearch/temp dirs get root-owned, causing permission crashes. Provide `UID` +explicitly via a `.env` file (Compose reads it automatically): + +```bash +echo "UID=$(id -u)" > .env +docker compose up -d +``` + +Wait until it reports UP: + +```bash +curl -s http://localhost:9000/api/system/status # {"status":"UP", ...} +``` + +> If a previous run started as root and left root-owned volumes, wipe and retry: +> `docker compose down -v && echo "UID=$(id -u)" > .env && docker compose up -d`. + +**3d. Verify the running instance loaded YOUR plugin** (the deployed JAR must contain your +changes, and SonarQube must have started *after* you copied it): + +```bash +# the plugin's registered timestamp should be AFTER your JAR's build time +TOKEN= # from Step 4 +curl -s -u "$TOKEN:" http://localhost:9000/api/plugins/installed \ + | python3 -c "import sys,json;[print(p['key'],p['version']) for p in json.load(sys.stdin)['plugins'] if 'crypto' in p['key']]" +# confirm the JAR carries your code: +unzip -l .SonarQube/plugins/sonar-cryptography-plugin-*.jar | grep CallContextStats +``` + +If you rebuild the plugin later, repeat 3a–3b then `docker compose restart sonarqube`. + +--- + +### Step 4 — Create an analysis token + +In the SonarQube UI (`http://localhost:9000`, default login `admin`/`admin`, change on first +login): **My Account → Security → Generate Token** (type: *Global Analysis Token*). Copy it. + +```bash +export TOKEN=sqp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx +``` + +--- + +### Step 5 — Run the scan with heap instrumentation + +Run from the **Keycloak** directory (so `cbom.json` and scanner work dirs land there, not in +this repo). + +**Important — the scanner forks a separate JVM.** `sonar-maven-plugin` downloads and runs the +**SonarScanner Engine** in a child JVM under `~/.sonar/cache/...` — this is where the plugin, +`CallStackAgent`, and the heap actually live. It does **not** inherit `MAVEN_OPTS`. To cap and +instrument the *right* JVM, pass `-Dsonar.scanner.javaOpts`. + +```bash +cd ~/Downloads/keycloak-main + +mvn org.sonarsource.scanner.maven:sonar-maven-plugin:sonar \ + -Dsonar.projectKey=keycloak \ + -Dsonar.projectName='keycloak' \ + -Dsonar.host.url=http://localhost:9000 \ + -Dsonar.token=$TOKEN \ + -Dsonar.scanner.javaOpts="-Xms512m -Xmx6g -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp" +``` + +- `-Xmx6g` is deliberate: the pre-fix code drove this scan past **7 GB and never finished**, + so completing under 6 GB is itself a pass/fail signal. +- A heap dump is written to `/tmp` if it OOMs, for post-mortem analysis. + +> Avoid embedding `-Xlog:gc*:file=...` in `sonar.scanner.javaOpts`; the multi-colon argument +> is mangled when passed through the scanner. Sample heap externally instead (Step 6). + +--- + +### Step 6 — Sample the engine heap during the scan + +While the scan runs, sample the **engine** JVM's heap with `jcmd` (from a full JDK — the +scanner's bundled JRE has no `jcmd`). Run this in a second terminal *after* the scan reaches +its analysis phase: + +```bash +JCMD=$(command -v jcmd) # must be a JDK jcmd +OUT=/tmp/keycloak-heap-samples.csv +echo "epoch,rss_mb,heap_used_mb,heap_total_mb,meta_used_mb" > "$OUT" +while true; do + E=$(pgrep -f sonar-scanner-engine-shaded | head -1) + [ -z "$E" ] && break # engine gone -> scan finished + RSS=$(ps -o rss= -p "$E" 2>/dev/null | tr -d ' ') + INFO=$("$JCMD" "$E" GC.heap_info 2>/dev/null) + HU=$(echo "$INFO" | grep -oE 'used [0-9]+K' | head -1 | grep -oE '[0-9]+') + HT=$(echo "$INFO" | grep -oE 'total [0-9]+K' | head -1 | grep -oE '[0-9]+') + MU=$(echo "$INFO" | grep Metaspace | grep -oE 'used [0-9]+K' | grep -oE '[0-9]+') + [ -n "$RSS" ] && echo "$(date +%s),$((RSS/1024)),$((${HU:-0}/1024)),$((${HT:-0}/1024)),$((${MU:-0}/1024))" >> "$OUT" + sleep 15 +done + +# peak heap used / committed / RSS across the run: +tail -n +2 "$OUT" | awk -F, 'BEGIN{u=0;c=0;r=0} + $3>u{u=$3} $4>c{c=$4} $2>r{r=$2} + END{printf "peak heap_used=%d MB peak committed=%d MB peak RSS=%d MB samples=%d\n",u,c,r,NR}' +``` + +--- + +### Step 7 — Interpret the results + +Confirm the scan actually exercised the plugin: + +```bash +grep -E "ANALYSIS SUCCESSFUL|CBOM was successfully generated" +``` + +A CBOM (`cbom.json` in the scan CWD) means detections fired, so the call-stack term was +exercised. + +**What good looks like (with the AST-detach fix, measured on Keycloak `main`, SonarQube 26.1, +93 modules / ~8200 files):** + +| Metric | Pre-fix (old) | With AST-detach fix | +|---|---|---| +| Outcome | ~7 GB **and climbing, did not finish** | **ANALYSIS SUCCESSFUL** in ~11 min | +| Under `-Xmx6g` | would OOM | completed, no OOM | +| Peak heap used | 7 GB+ (linear, no plateau) | ~4.6 GB (oscillating, GC reclaims) | + +- **Healthy:** `heap_used` **oscillates** — rises then drops as G1 reclaims — and the scan + completes. The post-GC floor may grow modestly but the run finishes under the cap. +- **Regression:** `heap_used` climbs **monotonically** toward the cap and OOMs (heap dump in + `/tmp`), or the scan never finishes. That signals AST pinning (or another unbounded term) + has returned. + +Note the residual ~4.6 GB is *not* the call-stack AST term (that is eliminated — see the +synthetic harness's `retainedWithTree=0`); it is SonarQube's baseline analysis cost plus the +CBOM nodes accumulated for the whole scan (`JavaAggregator.detectedNodes`). `jcmd`'s +`heap_used` includes uncollected garbage, so the true retained set sits at the GC *floors*, +below the sampled peak. + +--- + +### Cleanup + +```bash +# stop SonarQube (keep data) # or: down -v to wipe volumes +docker compose down + +# remove scan artifacts from the Keycloak dir +rm -f ~/Downloads/keycloak-main/cbom.json +rm -rf ~/Downloads/keycloak-main/.scannerwork +``` + +--- + +## Gotchas quick reference + +- **0 detections / term is 0** → Keycloak wasn't compiled. Rebuild (Step 2); the scan needs + resolvable types (`sonar.java.binaries`). +- **Heap cap/logging seem ignored** → you set `MAVEN_OPTS`. The analysis runs in the forked + scanner-engine JVM; use `-Dsonar.scanner.javaOpts` (Step 5). +- **`jcmd` not found / can't attach** → the scanner's bundled JRE lacks `jcmd`; use a full JDK's + `jcmd` (`GC.heap_info` attaches across compatible versions). +- **Container permission crashes / runs as root** → `UID` was unset (zsh read-only). Create + `.env` with `UID=$(id -u)` and, if it already ran as root, `docker compose down -v` first. +- **Plugin changes not reflected** → rebuild JAR, copy into `.SonarQube/plugins/`, then + `docker compose restart sonarqube` (plugins load at startup). Verify with the + `api/plugins/installed` check in Step 3d. From 1c6c49af6027cf4fccd1aca1731aef6a85299fef Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Mon, 6 Jul 2026 11:27:43 +0200 Subject: [PATCH 25/35] docs: design for call-stack heap attribution & trim (H1+H2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- ...-callstack-heap-attribution-trim-design.md | 183 ++++++++++++++++++ 1 file changed, 183 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-06-callstack-heap-attribution-trim-design.md diff --git a/docs/superpowers/specs/2026-07-06-callstack-heap-attribution-trim-design.md b/docs/superpowers/specs/2026-07-06-callstack-heap-attribution-trim-design.md new file mode 100644 index 00000000..79f74fb0 --- /dev/null +++ b/docs/superpowers/specs/2026-07-06-callstack-heap-attribution-trim-design.md @@ -0,0 +1,183 @@ +# Call-stack Heap: Attribution & Trim — Design + +**Date:** 2026-07-06 +**Status:** Approved (brainstorming) — ready for `writing-plans` +**Scope:** `engine` + Java language support + perf harness/docs. Heap track of the performance backlog. + +## Summary + +The AST-detach work (`feat/callstack-ast-detach`) removed the dominant heap term on large +scans — whole-file AST pinning by the call-stack (the measured ~7 GB on Keycloak). A smaller +residual remains: the post-GC heap floor still grows over a scan (observed ~1.6 → ~3.4 GB). +That growth was *hypothesized* to be `JavaAggregator.detectedNodes`, but it was never +attributed, and the now-uncapped detached call-stack population is an equally plausible +source. + +This design does two things, in order: + +1. **H1 — Attribute the floor (the gate).** Measure what actually grows the post-detach + floor, split across three buckets: retained CBOM `INode`s, detached call-stack records, + and hooks. The result decides how much of H2 to build and whether `detectedNodes` earns + its own future spec. +2. **H2 — Trim the call-stack.** Land the *deferred* Phase-1 work from + `docs/superpowers/plans/2026-07-05-callstack-hooks-heap-reduction.md`: an eligibility + filter that stops recording library calls (Tasks 1–2, done regardless), and — **only if + H1 shows the detached population is still material** — a retention cap (Task 5). + +Measure first, so we don't trim a term that no longer dominates. + +## Background / Why now + +The original heap plan phased the work as: Phase 1 (trim + bound what enters the call-stack) +→ Phase 2 (detach recorded calls from the AST). In practice **Phase 2 shipped first** and +some of Phase 1 shipped alongside it: + +| Original Phase-1 task | Status | +|---|---| +| Task 3 — drop redundant `visitedTreeObjects` | **Done** | +| Task 4 — key-indexed subscription lookup | **Done** | +| Task 1–2 — `isHookEligibleCall` filter (skip library calls) | **Not done** | +| Task 5 — retention caps / overflow WARN | **Not done** | +| Phase 2 — AST detach | **Done** (out of order) | + +Because detach landed, the old Task-6 measurement (pre-detach, ~7 GB, AST-pinning-dominated) +no longer describes the current heap. H1 re-measures the *post-detach* residual; H2 finishes +the deferred trim. + +### Load-bearing facts (verified in the original plan, still hold post-detach) + +- The call-stack (`CallStackAgent.invokedCallStack`) is project-lifetime: one record per + distinct recorded invocation, released only by the end-of-scan `ScannerManager.reset()`. + Measured growth is **linear and unbounded** in project size (~179k records at ~63M calls + on Keycloak). Detach shrank each record's footprint (no AST pin) but **not the count**. +- Method hooks are created only from **user methods with a source declaration** + (`JavaDetectionEngine`, `declaration() != null`). A hook's matcher encodes a **specific + user-class FQN** and matches only `invokedObjectType.is(userClassFQN) AND name AND params`. + ⇒ A recorded **library** call (library owner type, no source declaration) can never match + any method hook. Recording it is pure waste. **Enum** accesses match a separate `EnumHook` + path and must stay recorded. +- Cross-file resolution replays the *entire accumulated* call-stack (a hook created in file B + resolves a call in file A), so retention **cannot** be scoped per file. This is why the + trim is an eligibility filter (never record the useless calls) rather than per-file + clearing. + +## H1 — Attribute the floor + +**Problem.** The perf harness (`CallStackHeapPerfTest`) prints `heapDeltaMB` and +`CallContextStats` (retained-with-tree vs. detached counts), but nothing attributes retained +*bytes* to a source. We can't currently tell call-stack from `detectedNodes` from hooks. + +**Approach — two signals:** + +1. **In-process object counts (cheap, always-on).** Extend the end-of-scan reporting to emit, + alongside `CallContextStats`: + - `JavaAggregator.getDetectedNodes().size()` (retained CBOM node count), + - detached call-stack record count (already in `CallContextStats.detached()`), + - hook-subscription counts (`HookRepository` / `HookDetectionObservable`). + + Surface these in two places: the `CallStackHeapPerfTest` report line, and a single DEBUG + log at end-of-scan (in `ScannerManager` or the aggregator) so they appear on a real + Keycloak run too. These are counts, not bytes — they size *populations*, not footprint. + +2. **True byte attribution (manual, decisive).** On a constrained-heap Keycloak scan using the + existing `jmap -histo:live` protocol in `docs/PERFORMANCE_TESTING.md`, attribute retained + bytes across three buckets: + - CBOM model: `com.ibm.mapper.model.*` `INode` subclasses (Algorithm, Key, Property, …), + - Call-stack: `DetachedCall`, `ArgSnapshot`, `ResolvedSnapshotValue`, + - Residual `Tree`/hooks: any `org.sonar.*` Tree still retained + hook structures. + +**Deliverable.** A short attribution table appended to `docs/PERFORMANCE_TESTING.md` (or a +dated note) plus a one-paragraph **decision**: +- Residual floor is **call-stack-dominated** → build H2's cap (Task 5) as well as the filter. +- Residual floor is **`detectedNodes`-dominated** → build only H2's filter (still a lossless + trim + CPU assist); open a *separate* spec for `detectedNodes` (dedup / incremental + emission). That spec is explicitly **out of scope here**. +- Residual floor is **hooks-dominated** → open a hooks-pruning spec (was Tier-3 item H3). + +H1 has no acceptance test beyond "the numbers are captured and the decision is written." + +## H2 — Trim the call-stack + +### Eligibility filter (Tasks 1–2) — build regardless of H1 + +Add a language-layer predicate that mirrors the existing `isDetachableCall` sibling: + +- **`engine/.../language/ILanguageSupport.java` (or `ILanguageTranslation`)** — add + `default boolean isHookEligibleCall(T tree) { return true; }`. Keeping it a defaulted method + leaves Python/Go on record-all (unchanged behavior). +- **`engine/.../language/java/JavaLanguageSupport.java`** — implement for Java: `true` for + enum accesses (must stay recorded for `EnumHook`) and for method invocations whose + `methodSymbol().declaration() != null`; `false` otherwise (library calls). DEBUG-log skips. +- **`engine/.../language/java/JavaDetectionEngine.java`** — gate at the **top of + `recordCall` (`:135`)**, i.e. *before* `isDetachableCall`/`buildDetachedCall`. Returning + early for library calls means they are neither recorded **nor** put through the expensive + translate + argument-snapshot in `buildDetachedCall` — a free assist to the C2 throughput + finding (that snapshot currently runs per root-rule per invocation). + +**Losslessness.** A library call cannot match any method hook (matcher requires a specific +user-class owner). The single bounded edge — a user class that *overrides* a library method +and is hooked via that override — is documented and DEBUG-logged. Everything else is a strict +no-op change to detections. + +### Retention cap (Task 5) — build only if H1 says so + +If H1 shows the detached population is a material heap term: +- Per-name-bucket cap **and** a global `invokedCallStack` cap, with sensible defaults, + overridable via a plugin property. +- On overflow, stop recording new calls for that bucket and emit a **single** WARN naming the + method + cap, so the (bounded, measurable) detection loss is explicit. + +If H1 shows the call-stack is *not* a material term post-detach, defer the cap with a written +note (the filter alone suffices, and adds no loss). + +## Components & data flow + +``` +recordCall (JavaDetectionEngine:135) + └─ isHookEligibleCall(invocation)? ← NEW gate (Task 1–2) + false → return (skip record + skip buildDetachedCall) [library call] + true → isDetachableCall? → buildDetachedCall (unchanged) + addRecordedCall → CallStackAgent.add + └─ (optional) cap check ← NEW (Task 5, conditional) +``` + +Reporting/attribution (H1) reads existing accessors: `callContextStats()`, +`JavaAggregator.getDetectedNodes()`, hook repositories — no data-flow changes, read-only. + +## Testing + +- **Cross-file regression guard (reuse existing).** `CrossFileHookDetachTest` and + `IsDetachableCallTest` already exercise cross-file hook resolution and the + detach-eligibility predicate. Add a sibling assertion (or a focused test) proving a + cross-file detection whose call site is a **user** method still resolves *with the filter + on*, and that a purely-library call is correctly skipped. This is the guard CI otherwise + lacks (all rule tests are single-file). +- **Filter unit coverage.** Table-style test of `isHookEligibleCall` on: library invocation + (false), user-declared invocation (true), enum access (true), the override edge (true, + documented). +- **Regression.** `mvn test -pl engine` and `mvn test -pl java` stay green. +- **Perf validation.** Re-run `CallStackHeapPerfTest` (`-Dtest=CallStackHeapPerfTest + -DexcludedGroups=`) and confirm detached counts drop by the library-call fraction; re-run + the Keycloak `jmap` attribution from H1 and record the floor delta. + +## Constraints + +- Java 17; changes confined to `engine` + Java language support + `java` test/perf + docs. +- Apache 2.0 header in every new `.java` file. `mvn spotless:apply` before commit (restore + `JsonCipherSuites` if Spotless truncates it — known issue). +- Keep the generic engine language-agnostic: eligibility goes through the language layer, not + into generic engine code. Python/Go remain on the `true` default. + +## Out of scope + +- `JavaAggregator.detectedNodes` dedup / incremental CBOM emission — H1 only decides whether + it earns its own spec. +- C1 root pre-filtering and the other CPU/throughput items — separate "throughput track" spec. +- Hooks-subscription pruning (H3) — only opened if H1 finds hooks material. + +## Sequencing + +1. H1 in-process counts + one-shot Keycloak `jmap` attribution → write the decision. +2. H2 eligibility filter (Tasks 1–2) + cross-file/unit tests. +3. H2 cap (Task 5) **iff** H1 said call-stack is material. +4. Re-measure; record the floor delta. From 80c58c0dc49d163380e61301e43e00347dff51f2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Mon, 6 Jul 2026 11:35:19 +0200 Subject: [PATCH 26/35] docs: task-by-task plan for call-stack heap attribution (H1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- .../2026-07-06-callstack-heap-attribution.md | 453 ++++++++++++++++++ 1 file changed, 453 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-06-callstack-heap-attribution.md diff --git a/docs/superpowers/plans/2026-07-06-callstack-heap-attribution.md b/docs/superpowers/plans/2026-07-06-callstack-heap-attribution.md new file mode 100644 index 00000000..b87fda9a --- /dev/null +++ b/docs/superpowers/plans/2026-07-06-callstack-heap-attribution.md @@ -0,0 +1,453 @@ +# Call-stack Heap Attribution (H1) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Attribute the post-detach heap floor (observed ~1.6 → ~3.4 GB over a scan) to its actual sources — retained CBOM `INode`s vs. detached call-stack records vs. hooks — so a written decision can gate the deferred call-stack trim (H2). + +**Architecture:** Purely additive observability. A small immutable summary record composes the three population *counts* already reachable in-process; `OutputFileJob` logs it once at end-of-scan; the perf harness prints the CBOM-node count alongside its existing `CallContextStats` line. True *byte* attribution is a documented manual `jmap -histo:live` runbook on a constrained-heap Keycloak scan. No engine API changes, no detection-behavior changes. + +**Tech Stack:** Java 17, Maven multi-module, SLF4J, JUnit 5 + AssertJ, SonarQube PostJob API, `jmap`. + +## Global Constraints + +- Java 17; changes confined to `sonar-cryptography-plugin` (main + test), `java` test/perf, and `docs/`. No `engine` or language-support API changes in H1. +- Apache 2.0 license header in every new `.java` file — copy verbatim from a neighbor (e.g. `sonar-cryptography-plugin/src/main/java/com/ibm/plugin/OutputFileJob.java` lines 1–19), updating the year to 2026 for new files. +- Run `mvn spotless:apply` before every commit. If Spotless truncates `JsonCipherSuites`, restore it before committing (known issue). +- `mvn test -pl sonar-cryptography-plugin` and `mvn test -pl java` stay green throughout. +- H2 (eligibility filter + retention cap) is explicitly **out of scope** — this plan ends with the attribution decision that decides whether/how H2 proceeds. Do not add the filter or cap here. + +## Scope Check + +This is a single, self-contained observability + measurement deliverable (one subsystem: end-of-scan reporting). It does not need decomposition. + +## File Structure + +``` +sonar-cryptography-plugin/src/main/java/com/ibm/plugin/ + HeapAttributionSummary.java (CREATE) immutable record of the three population counts + a format() string + OutputFileJob.java (MODIFY) build the summary and DEBUG-log it once, before reset() + ScannerManager.java (MODIFY) heapAttribution() — read the counts from the aggregators/lang-support +sonar-cryptography-plugin/src/test/java/com/ibm/plugin/ + HeapAttributionSummaryTest.java (CREATE) unit test for the record + format() + ScannerManagerTest.java (CREATE) test heapAttribution() reflects added nodes +java/src/test/java/com/ibm/plugin/perf/ + CallStackHeapPerfTest.java (MODIFY) add detectedNodes count to the printed report line +docs/ + PERFORMANCE_TESTING.md (MODIFY) jmap attribution runbook + results-table template + decision rule +``` + +**Responsibilities:** +- `HeapAttributionSummary` — pure value + formatting; the single testable unit, no statics. +- `ScannerManager.heapAttribution()` — the one place that reads the static aggregators / language support and constructs the summary. +- `OutputFileJob` — end-of-scan trigger; logs the summary. +- `CallStackHeapPerfTest` — fast in-process signal for local iteration. +- `PERFORMANCE_TESTING.md` — the decisive manual byte-attribution procedure + where the H1 decision is recorded. + +--- + +## Task 1: `HeapAttributionSummary` record + `format()` + +**Files:** +- Create: `sonar-cryptography-plugin/src/main/java/com/ibm/plugin/HeapAttributionSummary.java` +- Test: `sonar-cryptography-plugin/src/test/java/com/ibm/plugin/HeapAttributionSummaryTest.java` + +**Interfaces:** +- Consumes: nothing. +- Produces: `record HeapAttributionSummary(int detectedNodes, int detachedCalls, int totalCalls, int callStackBuckets)` with instance method `String format()`. + +- [ ] **Step 1: Write the failing test** + +Create `sonar-cryptography-plugin/src/test/java/com/ibm/plugin/HeapAttributionSummaryTest.java` (copy the license header from `OutputFileJob.java`, year 2026): + +```java +package com.ibm.plugin; + +import static org.assertj.core.api.Assertions.assertThat; + +import org.junit.jupiter.api.Test; + +class HeapAttributionSummaryTest { + + @Test + void formatContainsAllFourPopulationCounts() { + HeapAttributionSummary summary = new HeapAttributionSummary(1200, 179000, 630000, 15800); + String line = summary.format(); + assertThat(line) + .contains("detectedNodes=1200") + .contains("detachedCalls=179000") + .contains("totalCalls=630000") + .contains("callStackBuckets=15800"); + } + + @Test + void formatIsStableForZeroPopulations() { + assertThat(new HeapAttributionSummary(0, 0, 0, 0).format()) + .isEqualTo( + "[heap-attribution] detectedNodes=0 detachedCalls=0 " + + "totalCalls=0 callStackBuckets=0"); + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `mvn test -pl sonar-cryptography-plugin -Dtest=HeapAttributionSummaryTest` +Expected: FAIL — `HeapAttributionSummary` does not exist (compilation error). + +- [ ] **Step 3: Write minimal implementation** + +Create `sonar-cryptography-plugin/src/main/java/com/ibm/plugin/HeapAttributionSummary.java` (license header from `OutputFileJob.java`, year 2026): + +```java +package com.ibm.plugin; + +import javax.annotation.Nonnull; + +/** + * End-of-scan snapshot of the three heap populations whose relative size decides the H1 + * attribution: retained CBOM nodes, detached call-stack records, and the call-stack bucket count. + * Counts only — byte attribution is the manual {@code jmap} runbook in {@code + * docs/PERFORMANCE_TESTING.md}. + * + * @param detectedNodes retained CBOM {@code INode}s (Java aggregator) + * @param detachedCalls tree-free {@code DetachedCall}s in the call stack + * @param totalCalls total recorded calls (detached + retained-with-tree) + * @param callStackBuckets number of hash buckets in the call stack + */ +public record HeapAttributionSummary( + int detectedNodes, int detachedCalls, int totalCalls, int callStackBuckets) { + + @Nonnull + public String format() { + return "[heap-attribution] detectedNodes=" + + detectedNodes + + " detachedCalls=" + + detachedCalls + + " totalCalls=" + + totalCalls + + " callStackBuckets=" + + callStackBuckets; + } +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `mvn test -pl sonar-cryptography-plugin -Dtest=HeapAttributionSummaryTest` +Expected: PASS (2 tests). + +- [ ] **Step 5: Format and commit** + +```bash +mvn spotless:apply -pl sonar-cryptography-plugin +git add sonar-cryptography-plugin/src/main/java/com/ibm/plugin/HeapAttributionSummary.java \ + sonar-cryptography-plugin/src/test/java/com/ibm/plugin/HeapAttributionSummaryTest.java +git commit -m "feat(plugin): heap-attribution summary record for scan-floor analysis" +``` + +--- + +## Task 2: `ScannerManager.heapAttribution()` + end-of-scan log + +**Files:** +- Modify: `sonar-cryptography-plugin/src/main/java/com/ibm/plugin/ScannerManager.java` +- Modify: `sonar-cryptography-plugin/src/main/java/com/ibm/plugin/OutputFileJob.java:39-55` +- Test: `sonar-cryptography-plugin/src/test/java/com/ibm/plugin/ScannerManagerTest.java` + +**Interfaces:** +- Consumes: `HeapAttributionSummary(int, int, int, int)` from Task 1; `JavaAggregator.getDetectedNodes()`, `JavaAggregator.getLanguageSupport().callContextStats()` (returns `com.ibm.engine.callstack.CallContextStats` with `detached()`, `total()`, `buckets()`), `JavaAggregator.addNodes(List)`, `JavaAggregator.reset()`. +- Produces: `ScannerManager.heapAttribution()` returning `HeapAttributionSummary`. + +- [ ] **Step 1: Write the failing test** + +Create `sonar-cryptography-plugin/src/test/java/com/ibm/plugin/ScannerManagerTest.java` (license header from `OutputFileJob.java`, year 2026): + +```java +package com.ibm.plugin; + +import static org.assertj.core.api.Assertions.assertThat; + +import com.ibm.mapper.model.INode; +import com.ibm.mapper.model.algorithms.AES; +import com.ibm.mapper.utils.DetectionLocation; +import java.util.List; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +class ScannerManagerTest { + + @BeforeEach + @AfterEach + void clearAggregators() { + JavaAggregator.reset(); + } + + @Test + void heapAttributionReportsZeroPopulationsOnAFreshScanner() { + HeapAttributionSummary summary = new ScannerManager(null).heapAttribution(); + assertThat(summary.detectedNodes()).isZero(); + assertThat(summary.totalCalls()).isZero(); + assertThat(summary.callStackBuckets()).isZero(); + } + + @Test + void heapAttributionCountsRetainedDetectedNodes() { + DetectionLocation location = + new DetectionLocation("Test.java", 1, 0, List.of("aes"), () -> "test"); + JavaAggregator.addNodes(List.of(new AES(128, location))); + + HeapAttributionSummary summary = new ScannerManager(null).heapAttribution(); + assertThat(summary.detectedNodes()).isEqualTo(1); + } +} +``` + +Note: `new AES(int keyLength, DetectionLocation)` and the `DetectionLocation(String, int, int, List, IBundle)` record are verified real; `IBundle` is a functional interface, so `() -> "test"` supplies it (matching the `mapper` module's own test fixtures). The assertion (`detectedNodes() == 1`) is what matters, not which node type. + +- [ ] **Step 2: Run test to verify it fails** + +Run: `mvn test -pl sonar-cryptography-plugin -Dtest=ScannerManagerTest` +Expected: FAIL — `ScannerManager.heapAttribution()` does not exist (compilation error). + +- [ ] **Step 3: Add `heapAttribution()` to `ScannerManager`** + +In `ScannerManager.java`, add these imports near the existing ones: + +```java +import com.ibm.engine.callstack.CallContextStats; +``` + +Add this method after `getAggregatedNodes()` (around line 70): + +```java + /** + * Snapshot of the heap populations whose relative size attributes the post-detach scan-floor + * growth: retained CBOM nodes vs. detached call-stack records (see the H1 attribution in {@code + * docs/PERFORMANCE_TESTING.md}). Java is the measured heap driver; Python/Go call stacks are + * not included. + */ + @Nonnull + public HeapAttributionSummary heapAttribution() { + int detectedNodes = JavaAggregator.getDetectedNodes().size(); + CallContextStats stats = JavaAggregator.getLanguageSupport().callContextStats(); + return new HeapAttributionSummary( + detectedNodes, stats.detached(), stats.total(), stats.buckets()); + } +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `mvn test -pl sonar-cryptography-plugin -Dtest=ScannerManagerTest` +Expected: PASS (2 tests). + +- [ ] **Step 5: Wire the log into `OutputFileJob`** + +In `OutputFileJob.java`, replace the body of `execute` (lines 39–55) so the attribution logs once, before `reset()`, regardless of whether results exist: + +```java + @Override + public void execute(PostJobContext postJobContext) { + final String cbomFilename = + postJobContext + .config() + .get(Constants.CBOM_OUTPUT_NAME) + .orElse(Constants.CBOM_OUTPUT_NAME_DEFAULT); + ScannerManager scannerManager = new ScannerManager(new CBOMOutputFileFactory()); + if (scannerManager.hasResults()) { + final File cbom = new File(cbomFilename + ".json"); + scannerManager.getOutputFile().saveTo(cbom); + LOGGER.info("CBOM was successfully generated '{}'.", cbom.getAbsolutePath()); + scannerManager.getStatistics().print(LOGGER::info); + } else { + LOGGER.info("No cryptography assets were detected. CBOM will not be generated."); + } + LOGGER.debug(scannerManager.heapAttribution().format()); + scannerManager.reset(); + } +``` + +- [ ] **Step 6: Run the plugin module tests** + +Run: `mvn test -pl sonar-cryptography-plugin` +Expected: PASS (existing `OutputFileJobTest`, `PluginTest`, `JavaFileCheckRegistrarTest` plus the two new tests). + +- [ ] **Step 7: Format and commit** + +```bash +mvn spotless:apply -pl sonar-cryptography-plugin +git add sonar-cryptography-plugin/src/main/java/com/ibm/plugin/ScannerManager.java \ + sonar-cryptography-plugin/src/main/java/com/ibm/plugin/OutputFileJob.java \ + sonar-cryptography-plugin/src/test/java/com/ibm/plugin/ScannerManagerTest.java +git commit -m "feat(plugin): log heap-attribution populations at end of scan" +``` + +--- + +## Task 3: Add CBOM-node count to the perf harness report line + +**Files:** +- Modify: `java/src/test/java/com/ibm/plugin/perf/CallStackHeapPerfTest.java:103-118` + +**Interfaces:** +- Consumes: `JavaAggregator.getDetectedNodes()` (already imported in this test), the existing `CallContextStats stats` local. +- Produces: nothing (test-only reporting). + +- [ ] **Step 1: Add the detectedNodes count to the printed report** + +In `CallStackHeapPerfTest.detachesRecordedCallsAtScale`, the report block currently prints `retainedWithTree/detached/total/buckets/ratio`. Add the retained CBOM-node count so the harness shows both heap populations side by side. Replace the `System.out.printf(...)` call (lines 106–117) with: + +```java + System.out.printf( + "%n[callstack-perf] files=%d units=%d time=%dms heapDeltaMB=%d " + + "detectedNodes=%d " + + "retainedWithTree=%d detached=%d total=%d buckets=%d ratio=%.3f%n", + sources.size(), + units, + elapsedMs, + (heapAfter - heapBefore) / (1024L * 1024L), + JavaAggregator.getDetectedNodes().size(), + stats.retainedWithTree(), + stats.detached(), + stats.total(), + stats.buckets(), + stats.detachedRatio()); +``` + +(No new import — `com.ibm.plugin.JavaAggregator` is already imported at line 27.) + +- [ ] **Step 2: Run the perf harness and eyeball the new field** + +Run: `mvn test -pl java -DexcludedGroups= -Dtest=CallStackHeapPerfTest` +Expected: PASS; the printed line now includes `detectedNodes=` with `n > 0`. The existing assertions (`total()` positive, `detachedRatio >= 0.9`, `retainedWithTree <= 10`) still hold. + +- [ ] **Step 3: Commit** + +```bash +git add java/src/test/java/com/ibm/plugin/perf/CallStackHeapPerfTest.java +git commit -m "test(java): report detectedNodes count in call-stack perf harness" +``` + +--- + +## Task 4: Byte-attribution runbook + decision in `PERFORMANCE_TESTING.md` + +**Files:** +- Modify: `docs/PERFORMANCE_TESTING.md` + +**Interfaces:** none (documentation). + +- [ ] **Step 1: Append the attribution section** + +Add a new top-level section to `docs/PERFORMANCE_TESTING.md` (after the existing Keycloak end-to-end section). Paste it verbatim: + +````markdown +--- + +## C. Post-detach floor attribution (H1) + +AST-detach removed the dominant AST-pinning heap term, but the post-GC floor still grows over a +scan (observed ~1.6 → ~3.4 GB). This procedure attributes that residual to one of three +populations so we know whether the call-stack still needs trimming (H2) or whether retained CBOM +nodes dominate (a separate follow-up). + +### Two signals + +**1. In-process population counts (cheap).** At end of scan the plugin logs, at DEBUG: + +``` +[heap-attribution] detectedNodes= detachedCalls= totalCalls= callStackBuckets= +``` + +Enable DEBUG for the plugin (e.g. `-Dsonar.log.level=DEBUG` on the scanner, or the analysis +`sonar.verbose=true`) and read the line from the scanner log. The fast local proxy is +`CallStackHeapPerfTest`, whose report line now also prints `detectedNodes=`. + +Counts size the *populations*, not their bytes — a small count of heavy objects can still +dominate. Use them to spot which population grows, then confirm bytes with the histogram below. + +**2. Byte attribution via `jmap` (decisive).** During a constrained-heap Keycloak scan (see +section B), capture a live histogram near the end of analysis: + +```bash +# find the scanner JVM pid (the surefire/scanner java process running the analysis) +jps -l +# live histogram (forces a GC first), sorted by retained bytes +jmap -histo:live > histo.txt +``` + +Bucket the top entries of `histo.txt` into the three sources: + +| Bucket | Classes to sum in `histo.txt` | +|---|---| +| Retained CBOM nodes | `com.ibm.mapper.model.**` (e.g. `Algorithm`, `Key`, `Property`, `MessageDigest`, …) and their child `HashMap`/`HashMap$Node` share | +| Detached call-stack | `com.ibm.engine.callstack.DetachedCall`, `...callstack.ArgSnapshot`, `...callstack.ResolvedSnapshotValue` | +| Residual Tree / hooks | `org.sonar.**Tree*` still live + `com.ibm.engine.hooks.**` | + +Sample two or three histograms as the scan progresses to see which bucket *grows* (the floor is +about accumulation, not a one-time cost). + +### Decision (fill in after measuring) + +| Population | count (log) | approx. retained bytes (jmap) | +|---|---|---| +| Retained CBOM nodes | | | +| Detached call-stack | | | +| Residual Tree / hooks | | | + +Record the outcome here, then route H2 accordingly: + +- **Call-stack dominates** → proceed with H2: eligibility filter **and** retention cap. +- **CBOM nodes dominate** → build only H2's eligibility filter (still a lossless trim + CPU + assist); open a separate spec for `detectedNodes` dedup / incremental emission. +- **Hooks dominate** → open a hooks-subscription-pruning spec. + +> Note on H2's eligibility filter: the predicate cannot be derived from `methodSymbol().declaration()` +> — cross-file *user* calls resolve via `sonar.java.binaries` and have a null declaration, exactly +> like library calls (see the comment in `JavaLanguageSupport.isDetachableCall`). The discriminator +> must be pinned empirically against the `crossfile/` fixtures before the filter is written. +```` + +- [ ] **Step 2: Commit** + +```bash +git add docs/PERFORMANCE_TESTING.md +git commit -m "docs: post-detach floor attribution runbook (H1) + H2 decision rule" +``` + +--- + +## Task 5: Full-suite regression + measurement run + +**Files:** none (verification). + +- [ ] **Step 1: Run the two touched module suites** + +Run: `mvn test -pl sonar-cryptography-plugin && mvn test -pl java` +Expected: both green (the `java` default run excludes `@Tag("performance")`, so `CallStackHeapPerfTest` does not run here). + +- [ ] **Step 2: Capture the in-process signal** + +Run: `mvn test -pl java -DexcludedGroups= -Dtest=CallStackHeapPerfTest` +Expected: report line prints `detectedNodes=` and `detached=`; note both. + +- [ ] **Step 3: Run the decisive measurement** + +Follow `docs/PERFORMANCE_TESTING.md` section C on a Keycloak scan: read the `[heap-attribution]` +DEBUG line and capture at least two `jmap -histo:live` snapshots. Fill in the decision table and +write the one-paragraph outcome + H2 routing in the doc. + +- [ ] **Step 4: Commit the recorded decision** + +```bash +git add docs/PERFORMANCE_TESTING.md +git commit -m "docs: record H1 floor-attribution results and H2 routing decision" +``` + +--- + +## Self-Review notes + +- **Spec coverage:** H1's two signals (in-process counts + jmap byte attribution) → Tasks 1–4; the written decision + routing → Task 4 template and Task 5. H2 filter/cap deliberately excluded per the H1-only scope decision; the doc note preserves the corrected-predicate finding so H2 isn't rebuilt on the wrong premise. +- **No placeholders:** the only unfilled content is the runtime measurement table (Task 4/5), which is inherent to a measurement deliverable — the *procedure*, commands, and jmap class buckets are fully concrete. +- **Type consistency:** `HeapAttributionSummary(int detectedNodes, int detachedCalls, int totalCalls, int callStackBuckets)` is used identically in Tasks 1–2; `CallContextStats.detached()/total()/buckets()` match the record defined in `engine/.../callstack/CallContextStats.java`. From 10f5f793748068af2405f0f69e220de9f5587ba2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Mon, 6 Jul 2026 11:35:36 +0200 Subject: [PATCH 27/35] docs: correct H2 eligibility-predicate premise in heap-trim spec MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- ...6-07-06-callstack-heap-attribution-trim-design.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/specs/2026-07-06-callstack-heap-attribution-trim-design.md b/docs/superpowers/specs/2026-07-06-callstack-heap-attribution-trim-design.md index 79f74fb0..6414edbb 100644 --- a/docs/superpowers/specs/2026-07-06-callstack-heap-attribution-trim-design.md +++ b/docs/superpowers/specs/2026-07-06-callstack-heap-attribution-trim-design.md @@ -106,8 +106,16 @@ Add a language-layer predicate that mirrors the existing `isDetachableCall` sibl `default boolean isHookEligibleCall(T tree) { return true; }`. Keeping it a defaulted method leaves Python/Go on record-all (unchanged behavior). - **`engine/.../language/java/JavaLanguageSupport.java`** — implement for Java: `true` for - enum accesses (must stay recorded for `EnumHook`) and for method invocations whose - `methodSymbol().declaration() != null`; `false` otherwise (library calls). DEBUG-log skips. + enum accesses (must stay recorded for `EnumHook`) and for calls whose invoked-object owner + type *could* be a user class; `false` for library calls. **Predicate premise correction:** + the discriminator is **not** `methodSymbol().declaration() != null` — cross-file *user* calls + resolve their callee via `sonar.java.binaries` and have a null declaration too (see the + comment in `JavaLanguageSupport.isDetachableCall`, `:165-172`), so that predicate would drop + exactly the cross-file detections we must keep. There may be no clean intrinsic AST/symbol + signal separating a binary-resolved user class from a library class; the real discriminator + must be **pinned empirically** against the `crossfile/` fixtures (a characterization test) + before the filter body is written, and may end up coarser (e.g. a library-package check). + DEBUG-log skips. - **`engine/.../language/java/JavaDetectionEngine.java`** — gate at the **top of `recordCall` (`:135`)**, i.e. *before* `isDetachableCall`/`buildDetachedCall`. Returning early for library calls means they are neither recorded **nor** put through the expensive From f96cb7b1947181fd51fb89beab8ed93c263972b8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Mon, 6 Jul 2026 12:21:39 +0200 Subject: [PATCH 28/35] feat(plugin): heap-attribution summary record for scan-floor analysis MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- .../ibm/plugin/HeapAttributionSummary.java | 49 +++++++++++++++++++ .../plugin/HeapAttributionSummaryTest.java | 46 +++++++++++++++++ 2 files changed, 95 insertions(+) create mode 100644 sonar-cryptography-plugin/src/main/java/com/ibm/plugin/HeapAttributionSummary.java create mode 100644 sonar-cryptography-plugin/src/test/java/com/ibm/plugin/HeapAttributionSummaryTest.java diff --git a/sonar-cryptography-plugin/src/main/java/com/ibm/plugin/HeapAttributionSummary.java b/sonar-cryptography-plugin/src/main/java/com/ibm/plugin/HeapAttributionSummary.java new file mode 100644 index 00000000..116776a8 --- /dev/null +++ b/sonar-cryptography-plugin/src/main/java/com/ibm/plugin/HeapAttributionSummary.java @@ -0,0 +1,49 @@ +/* + * Sonar Cryptography Plugin + * Copyright (C) 2026 PQCA + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.ibm.plugin; + +import javax.annotation.Nonnull; + +/** + * End-of-scan snapshot of the three heap populations whose relative size decides the H1 + * attribution: retained CBOM nodes, detached call-stack records, and the call-stack bucket count. + * Counts only — byte attribution is the manual {@code jmap} runbook in {@code + * docs/PERFORMANCE_TESTING.md}. + * + * @param detectedNodes retained CBOM {@code INode}s (Java aggregator) + * @param detachedCalls tree-free {@code DetachedCall}s in the call stack + * @param totalCalls total recorded calls (detached + retained-with-tree) + * @param callStackBuckets number of hash buckets in the call stack + */ +public record HeapAttributionSummary( + int detectedNodes, int detachedCalls, int totalCalls, int callStackBuckets) { + + @Nonnull + public String format() { + return "[heap-attribution] detectedNodes=" + + detectedNodes + + " detachedCalls=" + + detachedCalls + + " totalCalls=" + + totalCalls + + " callStackBuckets=" + + callStackBuckets; + } +} diff --git a/sonar-cryptography-plugin/src/test/java/com/ibm/plugin/HeapAttributionSummaryTest.java b/sonar-cryptography-plugin/src/test/java/com/ibm/plugin/HeapAttributionSummaryTest.java new file mode 100644 index 00000000..ca7fe283 --- /dev/null +++ b/sonar-cryptography-plugin/src/test/java/com/ibm/plugin/HeapAttributionSummaryTest.java @@ -0,0 +1,46 @@ +/* + * Sonar Cryptography Plugin + * Copyright (C) 2026 PQCA + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.ibm.plugin; + +import static org.assertj.core.api.Assertions.assertThat; + +import org.junit.jupiter.api.Test; + +class HeapAttributionSummaryTest { + + @Test + void formatContainsAllFourPopulationCounts() { + HeapAttributionSummary summary = new HeapAttributionSummary(1200, 179000, 630000, 15800); + String line = summary.format(); + assertThat(line) + .contains("detectedNodes=1200") + .contains("detachedCalls=179000") + .contains("totalCalls=630000") + .contains("callStackBuckets=15800"); + } + + @Test + void formatIsStableForZeroPopulations() { + assertThat(new HeapAttributionSummary(0, 0, 0, 0).format()) + .isEqualTo( + "[heap-attribution] detectedNodes=0 detachedCalls=0 " + + "totalCalls=0 callStackBuckets=0"); + } +} From 7123bb294c3704a6b65566107ea0856a5b48f7cc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Mon, 6 Jul 2026 12:22:51 +0200 Subject: [PATCH 29/35] feat(plugin): log heap-attribution populations at end of scan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- .../java/com/ibm/plugin/OutputFileJob.java | 1 + .../java/com/ibm/plugin/ScannerManager.java | 15 +++++ .../com/ibm/plugin/ScannerManagerTest.java | 57 +++++++++++++++++++ 3 files changed, 73 insertions(+) create mode 100644 sonar-cryptography-plugin/src/test/java/com/ibm/plugin/ScannerManagerTest.java diff --git a/sonar-cryptography-plugin/src/main/java/com/ibm/plugin/OutputFileJob.java b/sonar-cryptography-plugin/src/main/java/com/ibm/plugin/OutputFileJob.java index 3d8a0af1..e43c1439 100644 --- a/sonar-cryptography-plugin/src/main/java/com/ibm/plugin/OutputFileJob.java +++ b/sonar-cryptography-plugin/src/main/java/com/ibm/plugin/OutputFileJob.java @@ -51,6 +51,7 @@ public void execute(PostJobContext postJobContext) { } else { LOGGER.info("No cryptography assets were detected. CBOM will not be generated."); } + LOGGER.debug(scannerManager.heapAttribution().format()); scannerManager.reset(); } } diff --git a/sonar-cryptography-plugin/src/main/java/com/ibm/plugin/ScannerManager.java b/sonar-cryptography-plugin/src/main/java/com/ibm/plugin/ScannerManager.java index 5570021e..17ded630 100644 --- a/sonar-cryptography-plugin/src/main/java/com/ibm/plugin/ScannerManager.java +++ b/sonar-cryptography-plugin/src/main/java/com/ibm/plugin/ScannerManager.java @@ -19,6 +19,7 @@ */ package com.ibm.plugin; +import com.ibm.engine.callstack.CallContextStats; import com.ibm.mapper.model.INode; import com.ibm.output.IOutputFile; import com.ibm.output.IOutputFileFactory; @@ -69,6 +70,20 @@ private List getAggregatedNodes() { return nodes; } + /** + * Snapshot of the heap populations whose relative size attributes the post-detach scan-floor + * growth: retained CBOM nodes vs. detached call-stack records (see the H1 attribution in {@code + * docs/PERFORMANCE_TESTING.md}). Java is the measured heap driver; Python/Go call stacks are + * not included. + */ + @Nonnull + public HeapAttributionSummary heapAttribution() { + int detectedNodes = JavaAggregator.getDetectedNodes().size(); + CallContextStats stats = JavaAggregator.getLanguageSupport().callContextStats(); + return new HeapAttributionSummary( + detectedNodes, stats.detached(), stats.total(), stats.buckets()); + } + public void reset() { JavaAggregator.reset(); PythonAggregator.reset(); diff --git a/sonar-cryptography-plugin/src/test/java/com/ibm/plugin/ScannerManagerTest.java b/sonar-cryptography-plugin/src/test/java/com/ibm/plugin/ScannerManagerTest.java new file mode 100644 index 00000000..a6eeac5f --- /dev/null +++ b/sonar-cryptography-plugin/src/test/java/com/ibm/plugin/ScannerManagerTest.java @@ -0,0 +1,57 @@ +/* + * Sonar Cryptography Plugin + * Copyright (C) 2026 PQCA + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.ibm.plugin; + +import static org.assertj.core.api.Assertions.assertThat; + +import com.ibm.mapper.model.INode; +import com.ibm.mapper.model.algorithms.AES; +import com.ibm.mapper.utils.DetectionLocation; +import java.util.List; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +class ScannerManagerTest { + + @BeforeEach + @AfterEach + void clearAggregators() { + JavaAggregator.reset(); + } + + @Test + void heapAttributionReportsZeroPopulationsOnAFreshScanner() { + HeapAttributionSummary summary = new ScannerManager(null).heapAttribution(); + assertThat(summary.detectedNodes()).isZero(); + assertThat(summary.totalCalls()).isZero(); + assertThat(summary.callStackBuckets()).isZero(); + } + + @Test + void heapAttributionCountsRetainedDetectedNodes() { + DetectionLocation location = + new DetectionLocation("Test.java", 1, 0, List.of("aes"), () -> "test"); + JavaAggregator.addNodes(List.of(new AES(128, location))); + + HeapAttributionSummary summary = new ScannerManager(null).heapAttribution(); + assertThat(summary.detectedNodes()).isEqualTo(1); + } +} From 2732d739881b283b5c27c91ea6bb2b6379f67847 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Mon, 6 Jul 2026 12:24:10 +0200 Subject: [PATCH 30/35] test(java): report detectedNodes count in call-stack perf harness MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- .../plans/2026-07-06-callstack-heap-attribution.md | 6 +++++- .../java/com/ibm/plugin/perf/CallStackHeapPerfTest.java | 2 ++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-07-06-callstack-heap-attribution.md b/docs/superpowers/plans/2026-07-06-callstack-heap-attribution.md index b87fda9a..d7e0381c 100644 --- a/docs/superpowers/plans/2026-07-06-callstack-heap-attribution.md +++ b/docs/superpowers/plans/2026-07-06-callstack-heap-attribution.md @@ -319,7 +319,11 @@ In `CallStackHeapPerfTest.detachesRecordedCallsAtScale`, the report block curren - [ ] **Step 2: Run the perf harness and eyeball the new field** Run: `mvn test -pl java -DexcludedGroups= -Dtest=CallStackHeapPerfTest` -Expected: PASS; the printed line now includes `detectedNodes=` with `n > 0`. The existing assertions (`total()` positive, `detachedRatio >= 0.9`, `retainedWithTree <= 10`) still hold. +Expected: PASS; the printed line now includes `detectedNodes=`. Note it reads `detectedNodes=0` +in this harness — the `CheckVerifier`/`TestBase` path constructs rules with `isInventory=false` +(`JavaBaseDetectionRule.java:52`), so `JavaAggregator.addNodes` never fires here; the count is +non-zero only on a real inventory scan (Keycloak). The existing assertions (`total()` positive, +`detachedRatio >= 0.9`, `retainedWithTree <= 10`) still hold. - [ ] **Step 3: Commit** diff --git a/java/src/test/java/com/ibm/plugin/perf/CallStackHeapPerfTest.java b/java/src/test/java/com/ibm/plugin/perf/CallStackHeapPerfTest.java index f61d92e7..b8c352ca 100644 --- a/java/src/test/java/com/ibm/plugin/perf/CallStackHeapPerfTest.java +++ b/java/src/test/java/com/ibm/plugin/perf/CallStackHeapPerfTest.java @@ -105,11 +105,13 @@ void detachesRecordedCallsAtScale(@TempDir Path tmp) throws Exception { // REPORT (never asserted) System.out.printf( "%n[callstack-perf] files=%d units=%d time=%dms heapDeltaMB=%d " + + "detectedNodes=%d " + "retainedWithTree=%d detached=%d total=%d buckets=%d ratio=%.3f%n", sources.size(), units, elapsedMs, (heapAfter - heapBefore) / (1024L * 1024L), + JavaAggregator.getDetectedNodes().size(), stats.retainedWithTree(), stats.detached(), stats.total(), From 38c01befc25d298f35f5efe52d70f7aa11b5b267 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Mon, 6 Jul 2026 12:25:12 +0200 Subject: [PATCH 31/35] docs: post-detach floor attribution runbook (H1) + H2 decision rule MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- docs/PERFORMANCE_TESTING.md | 67 +++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+) diff --git a/docs/PERFORMANCE_TESTING.md b/docs/PERFORMANCE_TESTING.md index e957517c..ac8fc436 100644 --- a/docs/PERFORMANCE_TESTING.md +++ b/docs/PERFORMANCE_TESTING.md @@ -283,6 +283,73 @@ rm -rf ~/Downloads/keycloak-main/.scannerwork --- +## C. Post-detach floor attribution (H1) + +AST-detach removed the dominant AST-pinning heap term, but the post-GC floor still grows over a +scan (observed ~1.6 → ~3.4 GB). This procedure attributes that residual to one of three +populations so we know whether the call-stack still needs trimming (H2) or whether retained CBOM +nodes dominate (a separate follow-up). + +### Two signals + +**1. In-process population counts (cheap).** At end of scan the plugin logs, at DEBUG: + +``` +[heap-attribution] detectedNodes= detachedCalls= totalCalls= callStackBuckets= +``` + +Enable DEBUG for the plugin (e.g. `-Dsonar.log.level=DEBUG` on the scanner, or the analysis +`sonar.verbose=true`) and read the line from the scanner log. The fast local proxy is +`CallStackHeapPerfTest`, whose report line now also prints `detectedNodes=` (note: `0` in that +harness — it runs with `isInventory=false`, so CBOM nodes are not aggregated there; the count is +meaningful only on a real inventory scan). + +Counts size the *populations*, not their bytes — a small count of heavy objects can still +dominate. Use them to spot which population grows, then confirm bytes with the histogram below. + +**2. Byte attribution via `jmap` (decisive).** During a constrained-heap Keycloak scan (see +section B), capture a live histogram near the end of analysis: + +```bash +# find the scanner JVM pid (the surefire/scanner java process running the analysis) +jps -l +# live histogram (forces a GC first), sorted by retained bytes +jmap -histo:live > histo.txt +``` + +Bucket the top entries of `histo.txt` into the three sources: + +| Bucket | Classes to sum in `histo.txt` | +|---|---| +| Retained CBOM nodes | `com.ibm.mapper.model.**` (e.g. `Algorithm`, `Key`, `Property`, `MessageDigest`, …) and their child `HashMap`/`HashMap$Node` share | +| Detached call-stack | `com.ibm.engine.callstack.DetachedCall`, `...callstack.ArgSnapshot`, `...callstack.ResolvedSnapshotValue` | +| Residual Tree / hooks | `org.sonar.**Tree*` still live + `com.ibm.engine.hooks.**` | + +Sample two or three histograms as the scan progresses to see which bucket *grows* (the floor is +about accumulation, not a one-time cost). + +### Decision (fill in after measuring) + +| Population | count (log) | approx. retained bytes (jmap) | +|---|---|---| +| Retained CBOM nodes | | | +| Detached call-stack | | | +| Residual Tree / hooks | | | + +Record the outcome here, then route H2 accordingly: + +- **Call-stack dominates** → proceed with H2: eligibility filter **and** retention cap. +- **CBOM nodes dominate** → build only H2's eligibility filter (still a lossless trim + CPU + assist); open a separate spec for `detectedNodes` dedup / incremental emission. +- **Hooks dominate** → open a hooks-subscription-pruning spec. + +> Note on H2's eligibility filter: the predicate cannot be derived from `methodSymbol().declaration()` +> — cross-file *user* calls resolve via `sonar.java.binaries` and have a null declaration, exactly +> like library calls (see the comment in `JavaLanguageSupport.isDetachableCall`). The discriminator +> must be pinned empirically against the `crossfile/` fixtures before the filter is written. + +--- + ## Gotchas quick reference - **0 detections / term is 0** → Keycloak wasn't compiled. Rebuild (Step 2); the scan needs From 1d0e442b1f9b4834638d7bf86e77cc23b31b9f54 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Mon, 6 Jul 2026 12:51:02 +0200 Subject: [PATCH 32/35] docs: record H1 floor-attribution results (Keycloak) + revised H2 routing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Measured: plugin footprint ~28MB (~1% of ~2.9GB floor); CBOM nodes negligible (~25KB), call-stack dominates plugin term (15.6MB, linear). Floor growth is SonarQube/ECJ baseline, not detectedNodes. Heap track effectively closed; eligibility filter reclassified to CPU/throughput. Signed-off-by: Nicklas Körtge --- docs/PERFORMANCE_TESTING.md | 54 +++++++++++++++++++++++++++---------- 1 file changed, 40 insertions(+), 14 deletions(-) diff --git a/docs/PERFORMANCE_TESTING.md b/docs/PERFORMANCE_TESTING.md index ac8fc436..bd6d2232 100644 --- a/docs/PERFORMANCE_TESTING.md +++ b/docs/PERFORMANCE_TESTING.md @@ -328,20 +328,46 @@ Bucket the top entries of `histo.txt` into the three sources: Sample two or three histograms as the scan progresses to see which bucket *grows* (the floor is about accumulation, not a one-time cost). -### Decision (fill in after measuring) - -| Population | count (log) | approx. retained bytes (jmap) | -|---|---|---| -| Retained CBOM nodes | | | -| Detached call-stack | | | -| Residual Tree / hooks | | | - -Record the outcome here, then route H2 accordingly: - -- **Call-stack dominates** → proceed with H2: eligibility filter **and** retention cap. -- **CBOM nodes dominate** → build only H2's eligibility filter (still a lossless trim + CPU - assist); open a separate spec for `detectedNodes` dedup / incremental emission. -- **Hooks dominate** → open a hooks-subscription-pruning spec. +### Decision — measured 2026-07-06 (Keycloak `main`, 94 compiled modules, SonarQube 26.1, `-Xmx6g`) + +Scan outcome: **ANALYSIS SUCCESSFUL** in ~14 min, CBOM generated (108 detected assets → 68 +components), no OOM; post-GC heap floor oscillated and ended ~2.9–3.4 GB (healthy — G1 reclaims, +no monotonic climb). Attribution from live `GC.class_histogram` (post-full-GC) sampled early / +mid / late in the run: + +| Population | early | mid | late (near end) | +|---|---|---|---| +| Retained CBOM nodes (`com.ibm.mapper.**`) | ~0 | 0.01 MB (484) | **0.02 MB (824 inst)** | +| Detached call-stack (`com.ibm.engine.callstack.**`) | ~0 | 7.6 MB (291k) | **15.6 MB (594k inst)** | +| Hooks (`com.ibm.engine.hooks.**`) | ~0 | 0.001 MB | **~1 KB (1376 inst)** | +| **Plugin total (`com.ibm.**`)** | ~0 | 14.2 MB | **28.4 MB** | +| **Whole heap floor** | 0.05 GB | 1.55 GB | **2.90 GB** | + +The ~2.9 GB floor is overwhelmingly **SonarQube / ECJ baseline**, not plugin state — the top +retained classes are `byte[]` (332 MB), `HashMap$Node` (251 MB), `Object[]` (219 MB), `char[]` +(210 MB), `ArrayList` (176 MB), and sonar-java/ECJ semantic objects (`InternalPosition` 152 MB, +`InternalSyntaxToken` 122 MB, `MethodBinding` 95 MB, `InternalRange` 76 MB). + +**Outcome — the original hypothesis is disproven.** The floor growth (~1.6 → ~3.4 GB) is *not* +`JavaAggregator.detectedNodes`: retained CBOM nodes are **negligible (~25 KB, 824 instances)**. +The entire plugin footprint is **~28 MB (~1 % of the floor)**, and the floor growth tracks +SonarQube's own accumulating semantic model / AST of the module under analysis — which the plugin +cannot reduce. Within the plugin the **call-stack dominates** (15.6 MB vs. 0.02 MB) and grows +linearly/unbounded (291k → 594k records mid→late), but AST-detach keeps each record tiny, so the +absolute heap cost is small. + +**H2 routing (revised by this measurement):** +- **No `detectedNodes` spec.** CBOM-node retention is not a heap problem — drop that candidate. +- **No heap-motivated retention cap.** 594k detached records ≈ 15.6 MB is not a memory risk; the + cap (old Task 5) is unjustified on heap grounds — defer/drop it. +- **Eligibility filter → justify on CPU, not heap.** Skipping library calls still avoids building + their expensive detached form (`buildDetachedCall`), so fold it into the **throughput track + (C1/C2)**, not a heap spec. Net: the heap track is effectively closed by this measurement. + +> Note: signal 1 (the `[heap-attribution]` DEBUG line) does not surface in the Maven scanner +> console even with `sonar.verbose=true` — the scanner does not route plugin `LOGGER.debug` there +> (the INFO `Detected Assets` statistics line does print). The `jmap`/`jcmd` histogram (signal 2) +> is the reliable attribution path; use it directly. > Note on H2's eligibility filter: the predicate cannot be derived from `methodSymbol().declaration()` > — cross-file *user* calls resolve via `sonar.java.binaries` and have a null declaration, exactly From 280b7e98edf0827529d091cf1e888264d9734ed9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Mon, 6 Jul 2026 12:55:37 +0200 Subject: [PATCH 33/35] refactor(java): reorder imports in `JavaDetachedIssueReporter` for consistency MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicklas Körtge --- .../engine/language/java/JavaDetachedIssueReporter.java | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/engine/src/main/java/com/ibm/engine/language/java/JavaDetachedIssueReporter.java b/engine/src/main/java/com/ibm/engine/language/java/JavaDetachedIssueReporter.java index 74b9fdab..d7169476 100644 --- a/engine/src/main/java/com/ibm/engine/language/java/JavaDetachedIssueReporter.java +++ b/engine/src/main/java/com/ibm/engine/language/java/JavaDetachedIssueReporter.java @@ -21,6 +21,9 @@ import com.ibm.engine.callstack.DetachedSyntaxToken; import com.ibm.engine.callstack.IDetachedIssueReporter; +import java.lang.reflect.Field; +import javax.annotation.Nonnull; +import javax.annotation.Nullable; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.sonar.api.batch.fs.InputFile; @@ -30,10 +33,6 @@ import org.sonar.plugins.java.api.JavaCheck; import org.sonar.plugins.java.api.JavaFileScannerContext; -import javax.annotation.Nonnull; -import javax.annotation.Nullable; -import java.lang.reflect.Field; - /** * Raises a SonarQube issue for a detached (tree-free) cross-file detection through {@link * SonarComponents}, which is shared per scan and does not pin any file's AST. It is captured while From 7c20aea8436eefde3540f10624b3ca7684cc040b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Mon, 6 Jul 2026 14:47:52 +0200 Subject: [PATCH 34/35] fix(engine): match detached call-stack keys with hook-context semantics MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit buildDetachedCall snapshotted its invoked-object/parameter IType keys with MatchContext.build(false, rule) (record context), but a DetachedCall is only ever matched in hook context via MethodMatcher.matchKeys. Under isHookContext=false the translation selects subtype-permissive matching (is || isSubtypeOf) instead of the exact is() the live retained-call path uses, so cross-file matches accepted subtypes the same-file path rejects (false positives) and the outcome depended on file visitation order. Snapshot with MatchContext.createForHookContext() so detached matching reproduces the live retained-call path exactly. Also: - gate the heap-attribution debug log behind isDebugEnabled() so the full CallContextStats.from call-stack traversal does not run on every production scan when DEBUG is off. - clarify ILanguageSupport.callContextStats is diagnostics-only and not part of the detection contract. Signed-off-by: Nicklas Körtge --- .../java/com/ibm/engine/language/ILanguageSupport.java | 6 ++++-- .../com/ibm/engine/language/java/JavaDetectionEngine.java | 8 ++++++-- .../src/main/java/com/ibm/plugin/OutputFileJob.java | 4 +++- 3 files changed, 13 insertions(+), 5 deletions(-) diff --git a/engine/src/main/java/com/ibm/engine/language/ILanguageSupport.java b/engine/src/main/java/com/ibm/engine/language/ILanguageSupport.java index 820f5def..4de5f908 100644 --- a/engine/src/main/java/com/ibm/engine/language/ILanguageSupport.java +++ b/engine/src/main/java/com/ibm/engine/language/ILanguageSupport.java @@ -158,8 +158,10 @@ default void notifyLeaveFile(@Nonnull org.sonar.api.batch.fs.InputFile inputFile } /** - * Read-only snapshot of the language's recorded-call population, for the performance/heap - * harness. Languages that do not accumulate a call stack return {@link CallContextStats#EMPTY}. + * Diagnostics-only: a read-only snapshot of the language's recorded-call population, exposed + * for the performance/heap harness and scan-floor attribution. This is not part of the + * detection contract — no detection behaviour depends on it, and languages that do not + * accumulate a call stack simply return {@link CallContextStats#EMPTY}. * * @return the current call-context stats; never {@code null} */ diff --git a/engine/src/main/java/com/ibm/engine/language/java/JavaDetectionEngine.java b/engine/src/main/java/com/ibm/engine/language/java/JavaDetectionEngine.java index 8f1b6705..99e800e4 100644 --- a/engine/src/main/java/com/ibm/engine/language/java/JavaDetectionEngine.java +++ b/engine/src/main/java/com/ibm/engine/language/java/JavaDetectionEngine.java @@ -147,8 +147,12 @@ private void recordCall(@Nonnull MethodInvocationTree invocation) { @Nullable private DetachedCall buildDetachedCall( @Nonnull MethodInvocationTree invocation, @Nonnull IScanContext scanContext) { - final MatchContext matchContext = - MatchContext.build(false, detectionStore.getDetectionRule()); + // A detached call is only ever matched in hook context (MethodMatcher.matchKeys), so its + // type keys must be snapshotted with hook-context semantics (exact type matching) to + // reproduce the live retained-call path, which matches via the hook's isHookContext=true + // MatchContext. Using record-context (isHookContext=false) here would make cross-file + // matching subtype-permissive and diverge from the same-file result. + final MatchContext matchContext = MatchContext.createForHookContext(); final ILanguageTranslation translation = handler.getLanguageSupport().translation(); final Optional invokedType = translation.getInvokedObjectTypeString(matchContext, invocation); diff --git a/sonar-cryptography-plugin/src/main/java/com/ibm/plugin/OutputFileJob.java b/sonar-cryptography-plugin/src/main/java/com/ibm/plugin/OutputFileJob.java index e43c1439..085ceb38 100644 --- a/sonar-cryptography-plugin/src/main/java/com/ibm/plugin/OutputFileJob.java +++ b/sonar-cryptography-plugin/src/main/java/com/ibm/plugin/OutputFileJob.java @@ -51,7 +51,9 @@ public void execute(PostJobContext postJobContext) { } else { LOGGER.info("No cryptography assets were detected. CBOM will not be generated."); } - LOGGER.debug(scannerManager.heapAttribution().format()); + if (LOGGER.isDebugEnabled()) { + LOGGER.debug(scannerManager.heapAttribution().format()); + } scannerManager.reset(); } } From d39f087d5535baa39bfc3714b40ed764760133d4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicklas=20K=C3=B6rtge?= Date: Mon, 6 Jul 2026 15:53:48 +0200 Subject: [PATCH 35/35] test(python,go): cross-file guard tests for shared call-stack bucketing The AST-detach branch narrowed CallStackAgent.onNewHookSubscription to fetch only the method-name-keyed bucket (bucketsToScan) instead of scanning all buckets. That narrowing is shared by every language, so document and guard its safety for Python and Go: - Python: bucketing is exercised within-file only (within-file wrapper-hook tests already pass, proving key alignment). No scan-level cross-file symbol resolution exists, so the narrowing can't drop cross-file detections. Added a two-file CrossFileHookResolveTest, @Disabled like ResolveImportedStructTest. - Go: GoDetectionEngine only populates the call stack and never registers hooks, so onNewHookSubscription/bucketsToScan is unreachable for Go; the GoVerifier harness is single-file only. Added a @Disabled CrossFileHookDetachTest that documents both blockers. Both tests are forward guards: enable them if either language gains cross-file or hook-based resolution. --- .../crossfile/CrossFileHookCaller.go | 8 ++ .../crossfile/CrossFileHookWrapper.go | 10 ++ .../crossfile/CrossFileHookDetachTest.java | 80 ++++++++++++++ .../resolve/CrossFileHookResolveTestFile.py | 7 ++ .../resolve/imports/CrossFileHookWrapper.py | 8 ++ .../resolve/CrossFileHookResolveTest.java | 103 ++++++++++++++++++ 6 files changed, 216 insertions(+) create mode 100644 go/src/test/files/rules/detection/crossfile/CrossFileHookCaller.go create mode 100644 go/src/test/files/rules/detection/crossfile/CrossFileHookWrapper.go create mode 100644 go/src/test/java/com/ibm/plugin/rules/detection/crossfile/CrossFileHookDetachTest.java create mode 100644 python/src/test/files/rules/resolve/CrossFileHookResolveTestFile.py create mode 100644 python/src/test/files/rules/resolve/imports/CrossFileHookWrapper.py create mode 100644 python/src/test/java/com/ibm/plugin/rules/resolve/CrossFileHookResolveTest.java diff --git a/go/src/test/files/rules/detection/crossfile/CrossFileHookCaller.go b/go/src/test/files/rules/detection/crossfile/CrossFileHookCaller.go new file mode 100644 index 00000000..ac480806 --- /dev/null +++ b/go/src/test/files/rules/detection/crossfile/CrossFileHookCaller.go @@ -0,0 +1,8 @@ +package crossfile + +// Caller in a different file from the wrapper. In a hook-based, cross-file world the MD5 usage inside +// HashIt would surface here through the wrapper call. Go registers no such hook today, so this stays +// a plain cross-file call with no detection expected at this site. +func RunHash() [16]byte { + return HashIt([]byte("data")) +} diff --git a/go/src/test/files/rules/detection/crossfile/CrossFileHookWrapper.go b/go/src/test/files/rules/detection/crossfile/CrossFileHookWrapper.go new file mode 100644 index 00000000..46d4cd80 --- /dev/null +++ b/go/src/test/files/rules/detection/crossfile/CrossFileHookWrapper.go @@ -0,0 +1,10 @@ +package crossfile + +import "crypto/md5" + +// Wrapper defined in a SEPARATE file from its caller. Were Go to register a hook on this wrapper's +// parameter (as Java/Python do), that hook would have to resolve the call recorded while analyzing +// the caller file via the shared, whole-scan call stack — the leg the bucketing narrowing touches. +func HashIt(data []byte) [16]byte { + return md5.Sum(data) +} diff --git a/go/src/test/java/com/ibm/plugin/rules/detection/crossfile/CrossFileHookDetachTest.java b/go/src/test/java/com/ibm/plugin/rules/detection/crossfile/CrossFileHookDetachTest.java new file mode 100644 index 00000000..7ebe7fe9 --- /dev/null +++ b/go/src/test/java/com/ibm/plugin/rules/detection/crossfile/CrossFileHookDetachTest.java @@ -0,0 +1,80 @@ +/* + * Sonar Cryptography Plugin + * Copyright (C) 2024 PQCA + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.ibm.plugin.rules.detection.crossfile; + +import com.ibm.engine.detection.DetectionStore; +import com.ibm.engine.language.go.GoScanContext; +import com.ibm.mapper.model.INode; +import com.ibm.plugin.TestBase; +import java.util.List; +import javax.annotation.Nonnull; +import org.junit.jupiter.api.Disabled; +import org.junit.jupiter.api.Test; +import org.sonar.go.symbols.Symbol; +import org.sonar.go.testing.GoVerifier; +import org.sonar.plugins.go.api.Tree; +import org.sonar.plugins.go.api.checks.GoCheck; + +/** + * Cross-file guard for the shared call-stack bucketing optimization, Go edition — documented as + * {@code @Disabled} because it is not constructible today, for two independent reasons found while + * auditing the {@code feat/callstack-ast-detach} branch: + * + *

    + *
  1. Go registers no hooks. {@code GoDetectionEngine} only ever calls {@code + * handler.addCallToCallStack(...)} (it populates the call stack) but never {@code + * addHookToHookRepository(...)} — only the Java and Python engines do. The optimization + * changed {@code CallStackAgent.onNewHookSubscription}/{@code bucketsToScan}, which run + * only when a hook is subscribed. With no Go hooks, that narrowed lookup path is + * unreachable for Go, so the change cannot drop any Go detection. Go also has no cross-scope + * (wrapper) argument resolution, so {@code HashIt([]byte(...))} in {@code + * CrossFileHookCaller.go} would not surface the MD5 inside {@code CrossFileHookWrapper.go} + * even within a single file. + *
  2. The Go test harness is single-file. {@link GoVerifier} parses exactly one source + * ({@code SingleFileVerifier}, one {@code parse} of {@code foo.go}); there is no two-file + * scan entry point to reproduce a genuine cross-file scenario. + *
+ * + *

The fixtures {@code crossfile/CrossFileHookWrapper.go} and {@code + * crossfile/CrossFileHookCaller.go} capture the intended scenario. Enable this test — and add + * multi-file support to {@link GoVerifier} — if Go ever gains hook-based cross-scope resolution; + * only then does the shared bucketing narrowing become reachable for Go and need this guard. + */ +class CrossFileHookDetachTest extends TestBase { + + @Disabled( + "Go registers no hooks (bucketing path unreachable) and GoVerifier is single-file — see" + + " class Javadoc") + @Test + void crossFileHookResolvesAcrossTwoFiles() { + // Intent (not runnable today): analyze CrossFileHookCaller.go + CrossFileHookWrapper.go in + // one scan and assert the MD5 in the wrapper resolves at the caller. GoVerifier has no + // multi-file entry point, so this documents the scenario rather than executing it. + GoVerifier.verify("rules/detection/crossfile/CrossFileHookCaller.go", this); + } + + @Override + public void asserts( + int findingId, + @Nonnull DetectionStore detectionStore, + @Nonnull List nodes) { + // Disabled — no findings are driven. + } +} diff --git a/python/src/test/files/rules/resolve/CrossFileHookResolveTestFile.py b/python/src/test/files/rules/resolve/CrossFileHookResolveTestFile.py new file mode 100644 index 00000000..a4923843 --- /dev/null +++ b/python/src/test/files/rules/resolve/CrossFileHookResolveTestFile.py @@ -0,0 +1,7 @@ +from cryptography.hazmat.primitives.asymmetric import ec +from imports.CrossFileHookWrapper import make_private_key + +# The algorithm literal lives HERE, in the caller module. Cross-file hook resolution must carry it +# into the detection recorded for `generate_private_key` inside CrossFileHookWrapper.py. The issue is +# reported at the resolved value's location (this line), so the Noncompliant marker sits here. +private_key = make_private_key(ec.SECP384R1()) # Noncompliant {{SECP384R1}} diff --git a/python/src/test/files/rules/resolve/imports/CrossFileHookWrapper.py b/python/src/test/files/rules/resolve/imports/CrossFileHookWrapper.py new file mode 100644 index 00000000..48ca43ac --- /dev/null +++ b/python/src/test/files/rules/resolve/imports/CrossFileHookWrapper.py @@ -0,0 +1,8 @@ +from cryptography.hazmat.primitives.asymmetric import ec + + +# Wrapper defined in a SEPARATE module from its caller. The hook that resolves `arg` is created +# while analyzing THIS file; it must match the `make_private_key(...)` call recorded while analyzing +# the caller module. This is the cross-file leg of the call-stack bucketing path. +def make_private_key(arg): + return ec.generate_private_key(arg) diff --git a/python/src/test/java/com/ibm/plugin/rules/resolve/CrossFileHookResolveTest.java b/python/src/test/java/com/ibm/plugin/rules/resolve/CrossFileHookResolveTest.java new file mode 100644 index 00000000..9f8c55d5 --- /dev/null +++ b/python/src/test/java/com/ibm/plugin/rules/resolve/CrossFileHookResolveTest.java @@ -0,0 +1,103 @@ +/* + * Sonar Cryptography Plugin + * Copyright (C) 2024 PQCA + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.ibm.plugin.rules.resolve; + +import com.ibm.engine.detection.DetectionStore; +import com.ibm.engine.detection.Finding; +import com.ibm.engine.utils.DetectionStoreLogger; +import com.ibm.mapper.model.INode; +import com.ibm.plugin.TestBase; +import java.util.List; +import javax.annotation.Nonnull; +import org.junit.jupiter.api.Disabled; +import org.junit.jupiter.api.Test; +import org.sonar.plugins.python.api.PythonCheck; +import org.sonar.plugins.python.api.PythonVisitorContext; +import org.sonar.plugins.python.api.symbols.Symbol; +import org.sonar.plugins.python.api.tree.Tree; +import org.sonar.python.checks.utils.PythonCheckVerifier; + +/** + * Cross-file guard for the shared call-stack bucketing optimization. + * + *

The AST-detach branch narrowed {@code CallStackAgent.onNewHookSubscription} from scanning + * every recorded-call bucket to fetching only the single bucket keyed by the hooked method name's + * hash (see {@code bucketsToScan}). That narrowing is shared by all languages, so it must hold that + * a call recorded while analyzing one file is still found by a hook created while analyzing another + * file in the same scan — Python detaches nothing ({@code notifyLeaveFile} is a no-op here), so the + * whole-scan call stack is exactly where cross-file resolution lives. + * + *

This is the Python analogue of {@code CrossFileHookDetachTest} (Java): {@code + * make_private_key(...)} is defined in {@code imports/CrossFileHookWrapper.py} and called from + * {@code CrossFileHookResolveTestFile.py}; the {@code SECP384R1} literal must resolve across the + * file boundary via the hook on the wrapper's parameter. + * + *

Empirical status (branch {@code feat/callstack-ast-detach}): {@code @Disabled} because + * Python does not currently perform scan-level cross-file symbol resolution — the call {@code + * make_private_key} in the caller module does not link to its {@code def} in the wrapper module, so + * no detection is recorded and nothing is raised (same limitation as the {@code @Disabled} {@code + * ResolveImportedStructTest}). The bucketing narrowing is therefore not a risk for Python: + * it is only ever exercised within a single file, and the within-file wrapper-hook path (identical + * minus the file boundary, e.g. {@code fun7}/{@code fun8} in {@code + * ResolveValuesWithHooksTestFile.py}) still passes on this branch — proving the record-time key + * ({@code getKeyFormT}) stays aligned with the matcher-lookup key. This test stands as a forward + * guard: if Python ever gains cross-file resolution, remove {@code @Disabled} and it must pass, + * catching any bucketing regression that would silently drop cross-file detections. + */ +class CrossFileHookResolveTest extends TestBase { + + public CrossFileHookResolveTest() { + super(ResolveValuesWithHooks.rules()); + } + + @Override + public void asserts( + int findingId, + @Nonnull DetectionStore detectionStore, + @Nonnull List nodes) { + // Verification is driven by the Noncompliant marker in the caller fixture. + } + + @Disabled("cross-file symbol resolution not supported in Python — see class Javadoc") + @Test + void crossFileHookResolvesAcrossTwoModules() { + PythonCheckVerifier.verify( + List.of( + "src/test/files/rules/resolve/CrossFileHookResolveTestFile.py", + "src/test/files/rules/resolve/imports/CrossFileHookWrapper.py"), + this); + } + + @Override + public void update(@Nonnull Finding finding) { + final DetectionStore detectionStore = + finding.detectionStore(); + (new DetectionStoreLogger()) + .print(detectionStore); + detectionStore + .getDetectionValues() + .forEach( + iValue -> + detectionStore + .getScanContext() + .reportIssue( + this, iValue.getLocation(), iValue.asString())); + } +}