The reasoning behind the non-obvious choices — the "why" an interviewer would probe.
The single most important decision. Splitting source into fixed character or
line windows (as generic RAG pipelines do) cuts through the middle of
methods, separating a signature from its body and a Javadoc from what it
documents — which wrecks both embedding quality and the usefulness of a
result. JavaSourceParser uses the JavaParser AST to emit one CodeUnit
per real construct (method, constructor, class, interface, enum, record),
each with its signature, Javadoc, exact line range and full source. Every
vector therefore corresponds to something a developer would actually want
returned whole.
For a query like "validate a JWT token", the strongest signal is the method
name and its documentation, not the body. CodeUnit.embeddingText()
concatenates signature + Javadoc + body (truncated to a char budget) so the
most query-relevant text is front-loaded, and oversized classes still fit
the model's context window.
Keyword search is worthless on code with a prose tokenizer: the user types
"validate jwt token" but the code says validateJWTToken. CodeTokenizer
splits camelCase, snake_case and — the tricky part — acronym boundaries
(JWTToken → jwt, token; parseHTML5 → parse, html) with a single
regex, lowercasing and dropping single-character noise. This is what lets
BM25 contribute meaningfully to hybrid search.
Dense and sparse retrieval fail in opposite ways: embeddings nail
paraphrase but can miss exact identifiers; BM25 nails exact tokens but is
blind to synonyms. Fusing them covers both. RRF is chosen over score
normalisation because cosine similarities and BM25 scores are on
incompatible, corpus-dependent scales; RRF uses only ranks
(Σ 1/(k + rank), k = 60), which is scale-free and a robust default. The
same pattern appears in the companion RAG project — consistent technique
across a portfolio is intentional.
VectorStore.search returns List<Scored> (our own record), never a
Qdrant ScoredPoint. This keeps SearchService free of any storage
dependency, so its ranking logic is unit-tested with a trivial fake store.
The whole search core (CodeTokenizer, Bm25Index,
ReciprocalRankFusion, SearchService) depends only on interfaces and
plain data, which is why those tests need neither a model nor a database.
Qdrant holds vectors (plus a small payload) for approximate
nearest-neighbour search. The authoritative CodeUnit records are also
written to a JSONL sidecar via JsonlCorpusRepository. The search service
loads the sidecar to (a) resolve a hit id back to the full unit for display
and (b) build the in-memory BM25 index. Keeping the canonical records in one
streamable, Git-diffable file — rather than reconstructing them from vector
payloads — keeps the two concerns (ANN vs. source of truth) cleanly
separated. JSONL over a single JSON array so it streams and a bad line can't
invalidate the whole file.
Code units have human-readable ids (File.java#method@42) that are useful
in logs and results, but Qdrant point ids must be UUIDs or unsigned ints.
QdrantVectorStore derives a stable UUID with
UUID.nameUUIDFromBytes(id) and stores the original string in the payload
under unit_id, returning it transparently from search. Callers never
see the UUID.
DjlEmbedder downloads a real transformer on first use, which needs network
and time. HashingEmbedder implements the same interface with the hashing
trick (token → dimension, L2-normalised), so EMBEDDER=hashing runs the
entire index → search flow offline. It captures lexical overlap only — not
semantics — and exists for smoke tests, CI, and demonstrating the wiring; it
is the same idea as the fake embedder used in SearchServiceTest, promoted
to a runnable component.
Embedder (via DjlEmbedder) and VectorStore hold native / network
resources. Both are AutoCloseable and the CLI acquires them in
try-with-resources, so the model and gRPC channel are always released even
on failure. HashingEmbedder and the fakes implement close as a no-op.