Search - why?

[ser]

I indexed 40 decks. I use astrolabe nextcloud app to search.

I completely do not understand search algorithm, unfortunately.

By default the search selects "minimum relevance" to 0% - so it finds all indexed documents when I search for something completely out of order, like aaaaaaaaaaaaaaaaaaaaa. It's obvious results are not relevant.

What is sense of providing search results with 0% relevance?

When I set 20% relevance, it stops finding nonsense results, but then another problem arrives - it also stops finding relevant results... For example I had a word "subsidies" in one of my decks. With "minimum relevance" set to 20% this deck is not found when I search for "grants".

How does it aim to work?

[cbcoutinho]

Howdy @ser thanks for the question. The relevance score is by-default the result of the Reciprocal Rank Fusion (RRF) score as calculated by Qdrant during hybrid search. You can read more about the algorithm here if you're curious. For transparency, I've modified the k parameter from Qdrant's RFF defaults of 2->60, which matches what the broader literature recommends for hybrid search. This has consequences for what the maximum RFF score can be for hybrid search: for an entry that is rank=0 in both keyword and semantic search, the RFF score will be 0.033 => 3.3%. An entry that is rank=0 in only one of the collections will have a score of about 0.0167 => 1.67%

On it's own, RFF is a score that is not directly applicable as a relevance score. I've recently added support for reranking models, which are intended to map the results of a hybrid search query to a [0, 1] relevance score; however, this is not required and skipped if no reranking model is configured. Reranking models are not supported by ollama and require either using an API like Cohere's reranking model, or self-hosting a reranking model via Infinity or vllm.

Regarding the results of your semantic search, the calculated similarity between the words subsidy and grant is highly dependent on the embedding model you've chosen. I haven't benchmarked the bekko model so I can't comment on whether the two are semantically similar, but that's another thing you may want to look into.

[ser]

Thanks, my initial thought was also related to the model choice, so I had a quick look into the current rankings

https://huggingface.co/spaces/mteb/leaderboard

and took the leading harrier-oss-v1-0.6b, which gave me... identical results.

So I will definitely need reranking stuff, and I am very open to run infinity service - the only problem is I do not know, how.

Do you have any notes how could I configure mcp server to use infinity as the primary and reranking service?

[cbcoutinho]

@ser — reranking no longer requires an embedding gateway, so the Infinity route you asked about is now a supported configuration. Shipped in v0.179.0 (ghcr.io/cbcoutinho/nextcloud-mcp-server:0.179.0).

Docs: docs/reranking.md — backend comparison, setup for Infinity / vLLM / Cohere, and how to verify it's actually running. Per-variable reference in docs/configuration.md.

Infinity as both embedding and rerank service

This is what you asked for specifically — one process serves both, since it exposes an OpenAI-compatible /embeddings and a Cohere-protocol /rerank. There's an opt-in Compose profile:

docker compose --profile infinity up -d infinity

or directly:

docker run -it --rm -p 7997:7997 michaelf34/infinity:0.0.77-cpu \
  v2 --model-id BAAI/bge-m3 --model-id BAAI/bge-reranker-v2-m3 --port 7997

Then point the MCP server at it:

SEARCH_RERANK_ENABLED=true
SEARCH_RERANK_URL=http://infinity:7997/rerank
SEARCH_RERANK_MODEL=BAAI/bge-reranker-v2-m3   # bare id — no provider prefix

OPENAI_BASE_URL=http://infinity:7997
OPENAI_API_KEY=dummy                          # unused by Infinity; its presence selects the OpenAI-compatible provider
OPENAI_EMBEDDING_MODEL=BAAI/bge-m3

Two things that will bite otherwise:

• SEARCH_RERANK_URL is the full endpoint URL, path included. Infinity serves /rerank, vLLM /v1/rerank, Cohere /v2/rerank — nothing is appended for you, because guessing wrong fails silently (see below).
• The shipped SEARCH_RERANK_MODEL default is local/BAAI/bge-reranker-v2-m3. That local/ prefix is a routing prefix for the managed gateway; a direct Infinity endpoint has no such router and will reject it. Use the bare id. The server warns at startup if you leave the prefix on with a direct URL.

Please test against a local reranker before trusting the numbers

I'd genuinely recommend standing Infinity up and running your own queries through it rather than taking the defaults on faith. A cross-encoder is what turns the score you were looking at into something meaningful — it reads the query and each candidate together, so unlike the RRF fusion score it can say "none of these answer the question". Concretely, check:

curl -s localhost:8000/api/v1/status | jq .rerank_available   # expect: true

then run a search with rerank: true and look at the response fields:

• reranked: true — it ran. false while you asked for it means it was attempted and degraded to retrieval order; check the server log for rerank unavailable, using retrieval order: ….
• relevance_source: cross_encoder_calibrated — the relevance values are now calibrated probabilities you can read as percentages, and min_relevance will behave the way you expected it to in your original question.

Note that after a failure the rerank stage enters a 30-second cooldown and skips outright, so a config fix won't appear to take effect instantly.

Expect it to be slow on CPU

This is the caveat worth setting expectations on. A cross-encoder is a full forward pass per candidate, not a vector lookup — so on a CPU-only box a default 200-candidate pool takes tens of seconds per search. That is not a misconfiguration, it's the shape of the workload.

On CPU, trade depth for latency:

SEARCH_RERANK_POOL_SIZE=50
SEARCH_RERANK_TIMEOUT_SECONDS=60

A smaller pool costs some of the available quality gain — reranking can only reorder what retrieval handed it — but it keeps searches interactive. If the timeout expires the search still returns, just in retrieval order with reranked: false; it never fails.

For anything beyond single-user experimentation, a GPU is the practical answer. Drop the -cpu suffix from the Infinity image and add --gpus all, and a 200-candidate pool becomes sub-second. That matters most in two situations: if you want the full pool depth (which is where most of the ranking improvement lives), or if more than one person is searching at once — SEARCH_RERANK_MAX_CONCURRENCY defaults to 1 precisely because a CPU reranker has no headroom to serve concurrent searches, so on CPU under load requests queue behind each other.

One last note on model choice, since you tried swapping embedding models: BAAI/bge-reranker-v2-m3 is the only model we ship a fitted calibration curve for. Any other cross-encoder will rerank correctly, but reports relevance_source: uncalibrated — the ordering is still right, the number just isn't a probability.

[ser]

Thanks, I was able to run the stack I think correctly, on a GPU infinity with models takes about 4GB VRAM + 3GB RAM. Indexing and search it nearly instant.

But... I still get irrelevant results on nextcloud/astrolabe.

I need to get into the reason, why astrolabe gives me irrelevant results, it shows me completely empty note as the first hit to "grants" and the second is the deck which really contains "subsidies" then it's followed by another 18 irrelevant results.