Skip to content

[Bug]: OpenClaw plugin reports constant score 0.5 for all search results (retrieval_score dropped) #240

Description

@guolvlin-cn

Summary

The OpenClaw plugin (@matrixorigin/thememoria) discards the API's real relevance score and reports a constant 0.5 for every search result. The API itself is fine — this is purely a field-mapping bug in the plugin.

Consequence: any recall/relevance verification done through the OpenClaw memory_search tool is meaningless. Results appear unranked, and there is no way for the caller to tell a strong match from an unrelated one.

While confirming this I also found the limit parameter is ignored by POST /v1/memories/search (secondary issue, see below).

Environment

  • Plugin: @matrixorigin/thememoria 0.4.3 (backend: "api", apiUrl: https://api.thememoria.ai)
  • Host: OpenClaw 2026.8.1, node v26.7.0, macOS 26.6.2 arm64

Root cause

POST /v1/memories/search returns a per-item field named retrieval_score. The plugin never reads it.

openclaw/client.ts parses only a field called confidence, which the search response does not contain, so it is always null:

// openclaw/client.ts:99
confidence:
  typeof value.confidence === "number" && Number.isFinite(value.confidence)
    ? value.confidence
    : null,

openclaw/index.ts then maps that null through a normalizer whose fallback is the literal 0.5:

// openclaw/index.ts:359
function normalizeScore(confidence?: number | null): number {
  if (typeof confidence !== "number" || !Number.isFinite(confidence)) {
    return 0.5;   // <-- every search result lands here
  }
  ...
}

// openclaw/index.ts:401
score: normalizeScore(memory.confidence),

So score is 0.5 for 100% of results, always.

Two separate defects are stacked here:

  1. Wrong source field. Relevance comes from retrieval_score; confidence is a different concept (the response carries initial_confidence, a per-memory property unrelated to the query).
  2. Silent fallback to a plausible-looking constant. 0.5 is indistinguishable from a genuine mid-relevance score, so the failure is invisible. A missing score should surface as undefined/absent, not as a fabricated mid-range number.

Reproduction

Raw API call — real, well-separated scores:

curl -s -X POST "https://api.thememoria.ai/v1/memories/search" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $MEMORIA_API_KEY" \
  -d '{"query":"ripgrep ignore rules hidden files false negative","top_k":10}' \
| jq -r '.[] | "\(.retrieval_score)  \(.memory_id)"'
3.0664807326043806  019f27d3079373728942af6a4bf239ea
2.198475657498051   01a028b26add7b31b9a33faf50cdeb84
2.1921523899072697  01a023e3664b70418058f8dd45b388ec
2.029844700443825   01a05e7a0e33794290a8cfb758e4ecac
1.8586899208618903  01a047e33fdb79e18ef14eb0ab5ee709
1.8210298144414723  01a039550756...
1.7811247370235601  01a01fdc8424...
1.7108728133920021  01a0226805337e92b599c9f529ba7647
1.6108306094871974  019df1b0023a...
1.5904648443268936  01a034579e2e79c1bd431d2020822bc6

Same query through the OpenClaw memory_search tool — identical ordering, all scores flattened:

score: 0.5   (result 1)
score: 0.5   (result 2)
score: 0.5   (result 3)
...
score: 0.5   (result 10)

Note the response item keys confirm no confidence field exists:

['content', 'created_at', 'initial_confidence', 'is_active', 'memory_id',
 'memory_type', 'observed_at', 'retrieval_score', 'session_id',
 'trust_tier', 'user_id']

Impact

This silently breaks any workflow that reasons about relevance:

  • Recall verification is unusable. A common practice after writing a memory is to query it back and require a top-N hit. With a constant score the caller cannot distinguish "the entry is poorly written" from "the ranking signal is gone", and will waste time rewriting content that was never the problem.
  • Threshold filtering is impossible. Any "keep results above X" logic either passes everything or nothing.
  • Callers cannot detect the degradation. 0.5 looks like a legitimate value. Compare with null/absent, which would be an obvious signal.

Suggested fix

In openclaw/client.ts, parse retrieval_score (keeping confidence as a fallback for other endpoints):

retrieval_score:
  typeof value.retrieval_score === "number" && Number.isFinite(value.retrieval_score)
    ? value.retrieval_score
    : null,

In openclaw/index.ts, prefer it and stop fabricating a value:

score: memory.retrieval_score ?? memory.confidence ?? undefined,

Two design points worth considering:

  • retrieval_score is unbounded (observed >3.0) while the plugin's score contract appears to be [0,1]. Either widen the contract or normalize explicitly — but normalization must preserve relative ordering and gaps, not collapse them.
  • Whatever the choice, an unknown score should be reported as absent rather than as a constant. Fabricating 0.5 converts a detectable failure into a silent one, which is what made this bug survive.

Secondary: limit is ignored by /v1/memories/search

Found while reproducing the above. limit has no effect; top_k works.

for L in 3 5 20; do
  n=$(curl -s -X POST "https://api.thememoria.ai/v1/memories/search" \
    -H "Content-Type: application/json" -H "Authorization: Bearer $MEMORIA_API_KEY" \
    -d "{\"query\":\"ripgrep ignore rules hidden files\",\"limit\":$L}" | jq 'length')
  echo "limit=$L -> $n"
done
# limit=3  -> 10
# limit=5  -> 10
# limit=20 -> 10

for L in 3 20; do
  n=$(curl -s -X POST "https://api.thememoria.ai/v1/memories/search" \
    -H "Content-Type: application/json" -H "Authorization: Bearer $MEMORIA_API_KEY" \
    -d "{\"query\":\"ripgrep ignore rules hidden files\",\"top_k\":$L}" | jq 'length')
  echo "top_k=$L -> $n"
done
# top_k=3  -> 3
# top_k=20 -> 20

limit should either be honoured as an alias of top_k or rejected as an unknown parameter. Silently ignoring it while defaulting to 10 is the same failure mode as the score bug: the caller gets a plausible-looking result and no signal that their parameter did nothing.

Happy to split the second one into its own issue if you prefer.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions