Entroly 0.18.0: Building a 5,300-line Rust Engine for AI Context Optimization

[juyterman1000]

Entroly is an open-source context optimization engine for AI coding tools. The core is a single Rust crate (entroly-core, ~5,300 lines) exposed to Python via PyO3/maturin. v0.18.0 shipped this week with gzip-compressed index persistence, 415 Rust unit tests, and several architectural lessons worth sharing.

This post covers the Rust-specific implementation details: what worked, what surprised us, and the patterns that emerged from building a high-performance data plane that Python calls into via FFI.

This article was written with AI assistance.

---

Architecture: Why Rust for the Hot Path

Entroly's Python layer handles orchestration (MCP server, HTTP proxy, CLI). The Rust core owns everything latency-sensitive: fragment scoring and selection (PRISM reinforcement learning), SimHash-based near-duplicate detection, entropy-gated semantic caching (EGSC), submodular knapsack optimization with (1-1/e) approximation guarantee, and index persistence (gzip serialization/deserialization).

The boundary is a single #[pyclass] struct with ~40 #[pymethods]. PyO3's GIL management is straightforward once you accept that every method returning a PyObject needs Python::with_gil.

Pattern 1: REINFORCE with EMA Baseline (PRISM)

The core optimization loop uses policy-gradient RL to learn which scoring weights produce useful context selections. Four scalar weights with REINFORCE + an EMA baseline work remarkably well. The EMA baseline is critical — without it, gradient variance makes the weights oscillate wildly. With it, convergence happens in ~50 feedback events.

Pattern 2: SimHash for Near-Duplicate Detection

AI coding tools frequently re-ingest the same file with minor edits. We use SimHash (Charikar's locality-sensitive hash) to detect near-duplicates in O(1). Two fragments are near-duplicates if their Hamming distance is <= 3 bits (out of 64). The 3-bit threshold was tuned empirically.

Pattern 3: Gzip Persistence with Backward-Compat Magic-Byte Detection

v0.18.0 fixed a latent bug where persist_index wrote plain JSON to a file named index.json.gz. The fix uses flate2::write::GzEncoder for actual compression, and load_index detects the format by inspecting the first two bytes (0x1f 0x8b = gzip, 0x7b = legacy plain JSON). The #[cfg(unix)] guard for set_permissions(0o600) is essential — the index file contains ingested source code and must not be world-readable.

Pattern 4: Entropy-Gated Semantic Caching (EGSC)

The cache uses Thompson Sampling for admission control. Each cache entry has a Beta distribution quality model. The frequency sketch is a Count-Min Sketch (4 rows x 256 columns) that estimates query popularity without storing the queries themselves.

Pattern 5: Submodular Knapsack with Greedy (1-1/e) Guarantee

Context selection is a submodular maximization problem under a token budget constraint. We use the standard greedy algorithm with lazy evaluations. The marginal gain function combines four PRISM-weighted signals plus optional resonance and causal bonuses.

Testing: 415 Rust Tests

All 415 tests pass in 1.6 seconds. Coverage includes: core engine, EGSC cache (gauntlet, distribution shift, mutation stress, DAG-aware eviction), SimHash dedup, channel coding, causal graph, and utilization scoring.

Numbers

On Entroly's own codebase (433 files, 1M tokens): Indexing 0.66s, Token savings at 32K budget 97.0%, Optimization latency 22ms, 415/415 Rust tests pass, 965/965 Python tests pass.

Links: GitHub | PyPI | Apache-2.0