Noema

A semantic cache for LLM apps. It embeds each prompt and returns a cached answer when some earlier prompt was close enough in meaning, so "how do I reset my password" hits an entry stored for "how can I reset my password".

Underneath it's a small key-value server that speaks the Redis protocol, so you can also treat it as a stripped-down Redis (strings, lists, hashes, TTL, LRU) and poke at it with redis-cli. Java 21, nothing outside the JDK.

What's in it

The lookup is a nearest-neighbour search over embeddings. The index is a hand-written HNSW graph, the same structure Pinecone and the other vector databases use. Its recall is tested against a brute-force index that checks every vector.

The other half is surviving load. Model calls are slow, so if fifty copies of the same prompt land at once you don't want fifty calls. A single-flight layer runs the first one and hands its result to the other forty-nine.

By default the server points at a real embedding model. If it can reach one (a local Ollama, or OpenAI) it matches by meaning, so "I can't get into my account" hits an entry stored for "how do I sign in" with no words in common. If it can't reach one, it falls back to a built-in feature-hashing embedder and says so on startup. That fallback matches on shared words, not meaning, so it's only there to keep the thing runnable with zero setup, not to pretend it's semantic.

Try it

mvn -q exec:java -Dexec.mainClass=com.miniredis.semantic.SemanticCacheDemo

The demo throws a stampede of identical prompts at the cache, then two rounds of paraphrased questions, one cold and one warm:

Phase 1  stampede: 100 identical prompts fired at once
         -> 1 upstream call, 99 requests coalesced onto it

Phase 2  cold burst: 2000 concurrent requests across 8 clusters
         -> 30 upstream calls, 1794 coalesced onto in-flight calls

Phase 3  steady state: 2000 more concurrent requests (cache now warm)
         hit rate 100.0%, 0 upstream calls, done in 132 ms (p50 0.60 ms, p99 4.6 ms)

Coalescing carries the cold round (2000 requests, 30 calls). Cache hits carry the warm one.

Commands

The cache commands work over RESP and the inline protocol. Default upstream is a simulated model.

Command	Does	Reply
CPUT <text> <value> [ttl]	Cache a value under the meaning of the text	OK
CGET <text> [threshold]	Look up by meaning	value or (nil)
ASK <text>	Return the cached answer, or produce one and cache it	the answer
EMBED <text>	Embed text	array of floats
CSTATS	Hit rate, coalesced count, upstream time saved	array

redis-cli -p 6399 CPUT "how do I reset my password" "Go to Settings > Security > Reset."
redis-cli -p 6399 CGET "how can I reset my password" 0.6
redis-cli -p 6399 ASK  "when will my order arrive"

The key-value side has the usual suspects: SET/GET/INCR/EXPIRE/TTL/TYPE/KEYS, LPUSH/RPUSH/LPOP/LRANGE, HSET/HGET/HGETALL, and so on. Wrong-type operations return WRONGTYPE; quote values with spaces.

Running

mvn test                              # tests (a couple need a live model and skip without one)
mvn package                           # build target/noema.jar
java -jar target/noema.jar 6399       # start the server

Real embeddings

The server calls an OpenAI-compatible /v1/embeddings endpoint. Point it wherever you like:

# local, no key: Ollama with an embedding model
ollama pull nomic-embed-text
java -jar target/noema.jar 6399       # picks up http://localhost:11434 by default

# or OpenAI
export NOEMA_EMBED_URL=https://api.openai.com/v1/embeddings
export NOEMA_EMBED_MODEL=text-embedding-3-small
export NOEMA_EMBED_KEY=sk-...
java -jar target/noema.jar 6399

With no endpoint reachable it uses the hashing fallback and prints a line saying so.

How it fits together

   SemanticCache
     |  embed(text)        -> HashingEmbedder
     |  nearest neighbour  -> HnswIndex      (recall tested vs a brute-force index)
     |  value + TTL + LRU  -> DataStore       (RESP key-value engine)
     |  miss, coalesced    -> SingleFlight -> Producer (a simulated model)

The index and the value store are kept in step: when the store evicts or expires a key it tells the index to drop that vector, so the two never disagree.

Design notes

• HNSW instead of scanning every vector. Searches share a read lock and run in parallel; inserts take the write lock.
• Under load the win isn't the cache hit, it's not making the duplicate call at all. That's what single-flight buys you.
• LRU evicts the coldest of a small random sample rather than tracking a strict order, which keeps a lock off the read path. Redis does the same.
• Expiry is tested by advancing a fake clock, so no test sleeps.

Not done yet

The algorithmic core is here; most of what's left is the envelope around it for a real deployment.

Cache quality and correctness

• Fail loud instead of silently degrading. Today an unreachable model at startup drops to the word-hashing embedder, which quietly turns a semantic cache into a word-matching one — a real deployment should refuse to start rather than serve plausibly-wrong hits.
• Version the index by embedding model. Vectors from two models aren't comparable, so switching models should rebuild or namespace the index, not mix old and new vectors in one graph.
• Per-namespace thresholds and a false-hit eval set. The tests measure whether HNSW finds the true nearest neighbour, not whether returning it was the right call; a global 0.85 is a blunt knob.
• Early refresh (XFetch) so a hot entry rebuilds just before it expires.

Embedding path

• Batching embedding calls, and caching the embeddings themselves.
• Retries, a timeout budget, and a concurrency cap on the model endpoint. Identical prompts already coalesce via single-flight, but a burst of distinct prompts fires one uncoalesced call each.
• A real JSON parser for the embeddings response instead of the hand-rolled field scan.

Engine and durability

• Sets, sorted sets, and append-only-file persistence on the engine side.
• Rebuild the HNSW graph from persisted vectors on startup, so a restart isn't a cold-cache stampede.
• Bound memory, not just entry count — a real embedding is hundreds to thousands of floats per entry.

Operability

• Metrics export (hit rate, latencies, model error rate, eviction rate), health/readiness endpoints, and structured logs. CacheStats is in-process only today.
• AUTH and TLS — the port is currently open read/write to anyone who can reach it.
• A cap on RESP bulk-string length; a client-supplied length allocates unbounded right now.