mem_evolve

mem_evolve automatically evolves Linux page-cache eviction policies using LLM-driven code generation. Policies are written as eBPF programs against the cache_ext kernel (a custom 6.6.8 build from the SOSP 2025 cache_ext paper). A Python coordinator mutates policies across parallel search branches, evaluates each candidate on a cgroup-isolated benchmark, and feeds kernel-level probe measurements back to the LLM so the next mutation is informed by real performance data.

Paper

MemEvolve: Evolving Page Cache Policies with LLMs Dhruv Desai, Yashwanth Ranjan Singaravel, Michael Swift University of Wisconsin–Madison 📄 MemEvolve_Thesis.pdf

The Linux page cache relies on general-purpose, mostly LRU-derived eviction strategies that are not tuned to any particular workload. MemEvolve is an offline framework that uses an LLM to generate candidate eviction policies directly as native eBPF code, searches over them with breadth-first evolution, and deploys the winners in a running kernel through cache_ext — so evolved policies run in a real kernel, not a simulator. The evolution loop follows the iterative-refinement-with-execution-feedback pattern (Self-Refine, Reflexion, Voyager): each candidate's kernel-level probe feedback — cgroup refault counts, I/O byte counts, and policy-declared counters — is fed back into the next round's prompt so the model refines its proposals rather than guessing blindly. The central finding is that specialization, not universality, is where LLM-driven policy synthesis earns its keep: policies evolved for a specific workload consistently beat the general-purpose baselines cache_ext ships with.

Key ideas:

• Native-code policy synthesis with real deployment — the LLM writes eBPF, and winners load into a live kernel via cache_ext (no rebuild, no simulator).
• Kernel-probe feedback channel — cgroup-level refault counts, I/O bytes, and counters dumped on exit are threaded back into the next round's prompt as the gradient that drives mutation.
• Three-role split on different timescales — a planner (runs once, seeds K branches), a mutator (hot path, one focused edit per branch per round), and a frontier reviewer (periodically reshapes the population: continue / kill / pivot / spawn), with per-branch chat-history compaction to keep prompts bounded.
• Lightweight distributed evaluator — one branch is pinned to one host per round to keep cross-cgroup cache interference out of the score, making the inner loop tractable in wall-clock time on a small cluster.
• Three case studies — a synthetic adversarial workload (scan_thrash), a real production trace (twitter_leveldb_ro), and a novel LLM-driven workload (agent_swarm); the policy-counter feedback channel is what lets the search converge when headroom is small.

How it works

evolve.py  →  evolution.loop.evolve(cfg)
  ├── planner picks K seed policies (LRU, FIFO, S3-FIFO, …) + a focus hint each
  ├── each seed becomes the root of a search branch
  └── for each round, in parallel across branches:
        ├── mutator LLM proposes a new policy from the branch's history + last
        │   round's probe feedback
        ├── the policy is compiled (BPF + userspace loader) and run under a
        │   benchmark cgroup on a worker node
        ├── probes read kernel counters (refaults, bytes read, wallclock, …)
        └── results are recorded in the evolution tree
      every few rounds a "frontier" reviewer reshapes branches
      (continue / kill / pivot / spawn).

Each candidate is measured with workload-agnostic probes that read kernel counters rather than parsing benchmark output:

Probe	Signal
WallclockProbe	total runtime
CgroupIostatProbe	rbytes — bytes the workload had to read from disk
CgroupMemstatProbe	workingset_refault_file — direct refault count
PolicyCountersProbe	optional counters a policy dumps on exit

Repository layout

• cache_policy_evolution/ — the coordinator (Python).
  • evolve.py — CLI entry point; loads a TOML config and runs the loop.
  • evaluator/ — standalone policy evaluator: compile a policy, run it under a cgroup, collect probe results. Usable as a library or python3 -m evaluator.evaluate.
  • evolution/ — the parallel branch search: evolution tree, per-branch LLM chat history, frontier reviewer, seed planner, and the main evolve() loop.
  • worker_server.py — stdlib HTTP daemon that runs on each worker node and evaluates compiled policies sent by the coordinator.
  • targets/ — compilation helpers (split combined .c into BPF + loader).
  • seeds/ — seed policies: fifo.c, lru.c, mru.c, s3_fifo.c, scan_resist.c, noop.c.
  • eval/ — benchmark setup + run scripts (scan-thrash, filebench, YCSB/RocksDB, Twitter/LevelDB).
  • examples/ — small self-contained demos.
  • *.toml — example run configs.
• evolution_analyzer/ — single-file script that renders an evolution-tree JSON into a self-contained interactive HTML page.
• claude_api/ — a small OpenAI-compatible proxy backed by the Claude Code CLI, so the coordinator's OpenAI client can talk to Claude Code locally. Optional/alternative to a LiteLLM proxy — the default configs point at LiteLLM instead (see Quickstart).
• CloudLab provisioning helpers — setup_cloudlab.sh, setup_main_node.sh, start_workers.sh, parse_ssh.sh.
• cache_ext/ — the custom kernel + eBPF runtime, pulled in as a git submodule.

Quickstart

git clone --recurse-submodules <this-repo>
cd evo_cache/cache_policy_evolution
pip install -r requirements.txt

# LLM calls go through a LiteLLM proxy (OpenAI-compatible) by default — see
# [llm.mutator]/[llm.planner] in the TOMLs (api_base, api_key_env). Point
# api_base at your LiteLLM server and export the key it reads:
export LITELLM_MASTER_KEY=...       # or whatever api_key_env names in your TOML

# Run locally (workers = [] in the config → single in-process worker):
python3 evolve.py scan_thrash.toml
python3 evolve.py scan_thrash.toml --rounds 30
python3 evolve.py scan_thrash.toml --resume

To distribute evaluation across nodes, launch a worker daemon on each host and list their URLs in the config:

# on each worker:
python3 worker_server.py --port 8080

# in your TOML:
workers = ["http://host1:8080", "http://host2:8080"]

Requirements

• Must run on the custom 6.6.8-cache-ext kernel (eBPF cache_ext struct_ops). See the cache_ext submodule for build/install scripts.
• System deps: clang-14, bpftool, libbpf, build-essential, libelf-dev.
• Python deps: see cache_policy_evolution/requirements.txt.
• An LLM endpoint reachable from the coordinator — a LiteLLM proxy by default (any OpenAI-compatible endpoint works via llm_adapter; direct Anthropic access is also supported by setting provider = "anthropic" in the TOML).

The coordinator needs clang/bpftool (it compiles policies and ships pre-built binaries to workers). Worker nodes only need Python 3, the booted custom kernel, and the benchmark cgroup set up.

See CLAUDE.md for a deeper architecture reference.