Skip to content

perf(java): memoize BouncyCastle rule graph to cut construction heap (#476)#477

Merged
n1ckl0sk0rtge merged 12 commits into
mainfrom
fix/476-rule-graph-memoization
Jul 5, 2026
Merged

perf(java): memoize BouncyCastle rule graph to cut construction heap (#476)#477
n1ckl0sk0rtge merged 12 commits into
mainfrom
fix/476-rule-graph-memoization

Conversation

@n1ckl0sk0rtge

Copy link
Copy Markdown
Contributor

Closes #476.

Problem

JavaInventoryRule builds the Java detection-rule graph via JavaDetectionRules.rules(), which expanded into ~521,000 distinct rule objects — ~99.98% from BouncyCastle. Each X.rules()/X.all() re-materialized its entire dependent subtree at every embed site, and the subtrees compose multiplicatively (BcDerivationFunction → BcMac → BcBlockCipher.all() → BcBlockCipherEngine). Constructing that many DetectionRule/MethodDetectionRule objects can exhaust the SonarScanner Engine heap during rule construction.

Fix

Memoize the no-arg (default-context) rules()/all() accessors across all 84 BouncyCastle rule classes so each subtree is built once and shared by reference. A small thread-safe helper (Memoize.of) caches an immutable snapshot (List.copyOf). The parameterized rules(IDetectionContext) overloads are left building fresh for non-null contexts, but null-short-circuit into the memo so composed no-arg subtrees share automatically — no per-call-site rewiring.

Why sharing is safe: rules are immutable records; match() builds a fresh MatchContext per call (no state on the rule); all traversal state lives in DetectionStore (keyed by store, not rule identity); the engine has no rule-identity visited-set; and the builders copy their input lists, so a shared cached list can't be mutated by a consumer.

Result

Distinct reachable rule objects
Before ~521,017
After 2,563

~200× reduction — well beyond the issue's ≥10× acceptance target.

Verification

  • New RuleGraphMemoizationTest walks the full Java rule graph (following nextDetectionRules() and Parameter.getDetectionRules()), dedups by object identity, and asserts the count stays < 60_000. It fails at the pre-fix ~521k and passes at 2,563.
  • New MemoizeTest covers build-once caching and immutable snapshots.
  • All existing detection-rule tests pass unchanged (behavior preserved).
  • Full multi-module reactor build (mvn clean package) green across all 11 modules.

Notes

  • Scope is BouncyCastle only (JCA/SSL/random contribute ~126 objects and are untouched).
  • Design spec and implementation plan included under docs/superpowers/.

@n1ckl0sk0rtge

Copy link
Copy Markdown
Contributor Author

Follow-up: memoization now enforced automatically across all language modules

The earlier commits on this branch memoized the BouncyCastle subtrees by hand. This latest commit (8dd15e4) finishes the job and makes it self-policing, so no future rule can silently reintroduce the object blow-up.

What changed

  • Finished the stragglers: memoized the remaining 51 JCA/SSL classes (java), and every python (pyca, 16) and go (gocrypto, 23) detection rule. Every static rules() in all three modules now builds its subtree once and shares it by reference.
  • Memoize helper added to the python and go modules, mirroring the existing java one (typed to each module's Tree).
  • New RuleMemoizationEnforcementTest in each module (java/python/go). It classpath-scans every com.ibm.plugin.rules.detection class exposing a static no-arg rules() and asserts rules() == rules() — a memoized method returns the same List identity, a non-memoized one returns a fresh list each call. A new rule that forgets Memoize.of(...) now fails the build. The type system can't police a static method, so CI does.

Why a test instead of an abstract base class

Memoization here lives on static methods, which Java can't force via inheritance. An abstract-base refactor would make it structurally unbypassable but requires converting ~135 classes to instances and updating ~417 call sites (java-only). That trade-off is tracked separately in #478. The behavioral identity check gives the same "every new rule is memoized" guarantee at a fraction of the churn, and works uniformly across java/python/go.

Verification

  • Full suite green: java 160 / python 49 / go 40, 0 failures.
  • RuleGraphMemoizationTest footprint unchanged at 2563 distinct rule objects (well under the 60k guard; pre-fix was ~521k).

Note

Python's 6 resolve/Resolve* classes also expose rules() but are single-rule, test-only fixtures outside the detection package, so they're intentionally left out of scope. Easy to include if reviewers prefer full coverage.

@n1ckl0sk0rtge n1ckl0sk0rtge self-assigned this Jul 5, 2026
@n1ckl0sk0rtge n1ckl0sk0rtge added the enhancement New feature or request label Jul 5, 2026
Signed-off-by: Nicklas Körtge <Nicklas.Koertge1@ibm.com>
Signed-off-by: Nicklas Körtge <Nicklas.Koertge1@ibm.com>
Signed-off-by: Nicklas Körtge <Nicklas.Koertge1@ibm.com>
…476)

Signed-off-by: Nicklas Körtge <Nicklas.Koertge1@ibm.com>
Signed-off-by: Nicklas Körtge <Nicklas.Koertge1@ibm.com>
Signed-off-by: Nicklas Körtge <Nicklas.Koertge1@ibm.com>
Signed-off-by: Nicklas Körtge <Nicklas.Koertge1@ibm.com>
Signed-off-by: Nicklas Körtge <Nicklas.Koertge1@ibm.com>
Signed-off-by: Nicklas Körtge <Nicklas.Koertge1@ibm.com>
Signed-off-by: Nicklas Körtge <Nicklas.Koertge1@ibm.com>
Signed-off-by: Nicklas Körtge <Nicklas.Koertge1@ibm.com>
Finish the #476 memoization work and make it self-policing for every
future detection rule in all three language modules.

- Memoize the remaining static rules() subtrees: 51 JCA/SSL classes
  (java) plus every python (pyca) and go (gocrypto) detection rule, so
  shared subtrees are built once and referenced everywhere.
- Add a Memoize helper to the python and go modules, mirroring java.
- Add RuleMemoizationEnforcementTest to each module: it classpath-scans
  every com.ibm.plugin.rules.detection class exposing a static no-arg
  rules() and asserts rules() == rules() (same identity => memoized).
  A new rule that forgets Memoize.of(...) now fails the build, since the
  type system cannot police a static method.

Full suite green (java 160, python 49, go 40); RuleGraphMemoizationTest
footprint unchanged at 2563 distinct objects.

Signed-off-by: Nicklas Körtge <Nicklas.Koertge1@ibm.com>
@n1ckl0sk0rtge n1ckl0sk0rtge force-pushed the fix/476-rule-graph-memoization branch from 8dd15e4 to 36e697f Compare July 5, 2026 16:55
@n1ckl0sk0rtge n1ckl0sk0rtge marked this pull request as ready for review July 5, 2026 16:56
@n1ckl0sk0rtge n1ckl0sk0rtge requested a review from a team as a code owner July 5, 2026 16:56
@n1ckl0sk0rtge n1ckl0sk0rtge merged commit b4ac1c0 into main Jul 5, 2026
2 checks passed
@n1ckl0sk0rtge n1ckl0sk0rtge deleted the fix/476-rule-graph-memoization branch July 5, 2026 17:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reduce detection-rule memory footprint (BouncyCastle rule graph OOMs the Scanner Engine)

1 participant