Call-stack/hooks state grows unbounded and exhausts the Scanner Engine heap on large scans

[n1ckl0sk0rtge]

Summary

On large scans, the Java call-stack/hooks state accumulated by CallStackAgent grows linearly and without bound with project size and can exhaust the SonarScanner Engine heap. This is a separate heap term from the rule-graph construction OOM in #476 (fixed by #477): that one is ~520k rule objects built up-front; this one is runtime state that keeps growing as more files are analyzed.

Root cause

One static CallStackAgent per language (held via the *Aggregator statics) accumulates state for the whole scan and is released only by the end-of-scan ScannerManager.reset() PostJob:

• CallStackAgent.invokedCallStack — a CallContext(tree, scanContext) is recorded for method-invocation/enum trees encountered during detection traversal and never pruned during a scan.
• AST pinning is the dominant cost: each CallContext retains a SonarQube Tree (whose parent() chain reaches the compilation unit) plus the file's JavaFileScannerContext, so every retained node pins that file's entire AST for the whole scan → effectively large portions of the whole project's AST are held at once.
• CallStackAgent.visitedTreeObjects is a dedup Set that duplicates those tree refs (measured to be 100% redundant — see below).

Hard constraint: cross-file resolution (onNewHookSubscription) replays the entire accumulated call stack so a hook created in file B can resolve a call in file A. Naive per-file clearing silently drops cross-file detections (and CI misses it — all rule tests are single-file), so retention cannot simply be scoped per file.

Measured evidence

keycloak (Keycloak/keycloak main) scanned via mvn sonar:sonar (compiled modules, so types resolve and detections fire), with an instrumented CallStackAgent, SonarQube 26.1:

addCalls	retained CallContexts	buckets	peak used heap
1,147,068	3,265	~500	~0.5 GB
62,770,000	178,818	15,859	~7 GB (scan not yet finished)

• Growth is perfectly linear — ~2.85 retained per 1,000 addCalls, identical slope from 1M→63M calls, no plateau (tree objects are per-file, so distinct recorded call sites keep accumulating with project size).
• At ~63M calls it already held ~179k CallContexts and drove the heap to ~7 GB — enough to OOM a default-heap scanner on a large project.
• visitedTreeObjects.size() equals retainedCallContexts at every sample → the dedup set is a redundant duplicate (~179k entries).

Note: a source-only scan (no sonar.java.binaries/sonar.java.libraries) resolves nothing → no detections fire → addCall is never called → this term is 0. It only manifests once types resolve (compiled project, or sonar.java.libraries supplied). This is why it is easy to miss.

Proposed direction

A phased fix is drafted in docs/superpowers/plans/2026-07-05-callstack-hooks-heap-reduction.md:

• Phase 1 — trim & bound (mostly lossless):
  • Only record hook-eligible calls (skip library calls that can never match a source-declared method hook).
  • Remove the redundant visitedTreeObjects set (dedup within the per-name bucket instead).
  • Key-indexed subscription lookup (stop scanning all buckets in onNewHookSubscription).
  • Per-bucket / global retention caps with a single WARN on overflow (bounded, measurable loss).
  • Add a multi-file CheckVerifier cross-file regression test (today's tests are all single-file) and re-measure heap.
• Phase 2 — structural (deferred, after Phase 1 measurement): replace retained CallContext(tree, scanContext) with a detached call record (eagerly extract the match keys + a self-contained argument-value snapshot) so per-file ASTs become GC-eligible after a file is analyzed, removing the whole-project-AST pin.

Acceptance criteria

• Retained CallContext count and peak heap on the keycloak reference scan reduced substantially (target: no longer linear-unbounded in project size; bounded or GC-eligible per file).
• Cross-file hook detections unchanged (new multi-file regression test passes).
• mvn test -pl engine and mvn test -pl java stay green.
• A representative large scan completes under the default Scanner Engine heap.

Related

• Reduce detection-rule memory footprint (BouncyCastle rule graph OOMs the Scanner Engine) #476 / perf(java): memoize BouncyCastle rule graph to cut construction heap (#476) #477 — the other heap term (rule-graph construction). This issue is complementary, not a duplicate.