perf(java): memoize BouncyCastle rule graph to cut construction heap (#476)#477
Conversation
Follow-up: memoization now enforced automatically across all language modulesThe earlier commits on this branch memoized the BouncyCastle subtrees by hand. This latest commit ( What changed
Why a test instead of an abstract base classMemoization here lives on static methods, which Java can't force via inheritance. An abstract-base refactor would make it structurally unbypassable but requires converting ~135 classes to instances and updating ~417 call sites (java-only). That trade-off is tracked separately in #478. The behavioral identity check gives the same "every new rule is memoized" guarantee at a fraction of the churn, and works uniformly across java/python/go. Verification
NotePython's 6 |
Signed-off-by: Nicklas Körtge <Nicklas.Koertge1@ibm.com>
Signed-off-by: Nicklas Körtge <Nicklas.Koertge1@ibm.com>
Signed-off-by: Nicklas Körtge <Nicklas.Koertge1@ibm.com>
…476) Signed-off-by: Nicklas Körtge <Nicklas.Koertge1@ibm.com>
Signed-off-by: Nicklas Körtge <Nicklas.Koertge1@ibm.com>
Signed-off-by: Nicklas Körtge <Nicklas.Koertge1@ibm.com>
Signed-off-by: Nicklas Körtge <Nicklas.Koertge1@ibm.com>
Signed-off-by: Nicklas Körtge <Nicklas.Koertge1@ibm.com>
Signed-off-by: Nicklas Körtge <Nicklas.Koertge1@ibm.com>
Signed-off-by: Nicklas Körtge <Nicklas.Koertge1@ibm.com>
Signed-off-by: Nicklas Körtge <Nicklas.Koertge1@ibm.com>
Finish the #476 memoization work and make it self-policing for every future detection rule in all three language modules. - Memoize the remaining static rules() subtrees: 51 JCA/SSL classes (java) plus every python (pyca) and go (gocrypto) detection rule, so shared subtrees are built once and referenced everywhere. - Add a Memoize helper to the python and go modules, mirroring java. - Add RuleMemoizationEnforcementTest to each module: it classpath-scans every com.ibm.plugin.rules.detection class exposing a static no-arg rules() and asserts rules() == rules() (same identity => memoized). A new rule that forgets Memoize.of(...) now fails the build, since the type system cannot police a static method. Full suite green (java 160, python 49, go 40); RuleGraphMemoizationTest footprint unchanged at 2563 distinct objects. Signed-off-by: Nicklas Körtge <Nicklas.Koertge1@ibm.com>
8dd15e4 to
36e697f
Compare
Closes #476.
Problem
JavaInventoryRulebuilds the Java detection-rule graph viaJavaDetectionRules.rules(), which expanded into ~521,000 distinct rule objects — ~99.98% from BouncyCastle. EachX.rules()/X.all()re-materialized its entire dependent subtree at every embed site, and the subtrees compose multiplicatively (BcDerivationFunction → BcMac → BcBlockCipher.all() → BcBlockCipherEngine). Constructing that manyDetectionRule/MethodDetectionRuleobjects can exhaust the SonarScanner Engine heap during rule construction.Fix
Memoize the no-arg (default-context)
rules()/all()accessors across all 84 BouncyCastle rule classes so each subtree is built once and shared by reference. A small thread-safe helper (Memoize.of) caches an immutable snapshot (List.copyOf). The parameterizedrules(IDetectionContext)overloads are left building fresh for non-null contexts, but null-short-circuit into the memo so composed no-arg subtrees share automatically — no per-call-site rewiring.Why sharing is safe: rules are immutable
records;match()builds a freshMatchContextper call (no state on the rule); all traversal state lives inDetectionStore(keyed by store, not rule identity); the engine has no rule-identity visited-set; and the builders copy their input lists, so a shared cached list can't be mutated by a consumer.Result
~200× reduction — well beyond the issue's ≥10× acceptance target.
Verification
RuleGraphMemoizationTestwalks the full Java rule graph (followingnextDetectionRules()andParameter.getDetectionRules()), dedups by object identity, and asserts the count stays< 60_000. It fails at the pre-fix ~521k and passes at 2,563.MemoizeTestcovers build-once caching and immutable snapshots.mvn clean package) green across all 11 modules.Notes
docs/superpowers/.