MS: neural-symbolic take-max router + W1 writer fixes - #27
Conversation
|
Curious — the recent MS work moves intent through a writer/reader split. When you serve via MCP + HTTP + CLI simultaneously, how does intent state stay consistent across surfaces? Do you treat the graph as a single source of truth with process-level locking, or does each surface hold its own intent buffer that reconciles async? Working on a multi-agent collab layer and trying to figure out the right model here. Especially interested in what happens when an intent crystallizes while a query agent is mid-retrieval — does the reader see the new intent on the next call, or does it get surfaced through some other channel? |
@zsxh1990 Great question, happy to continue in discussions if useful. My point of view (may be wrong) is the design is really depends on tasks. If we are discussing auto-research discovery etc, then we need a hard synchronise and consistent truth source. If we are discussing say social sim, we may need a bit more uncertainty. If I were designing the multi-surface concurrent version, I'd keep the graph as the SoT and avoid per-surface mutable buffers (that turns a single-store problem into multi-master conflict resolution — the hardest thing in distributed systems, for no benefit when co-located). Concretely: ( below are my thinkings however, I speak so claude rephrase it a bit.
This is a bit beyond the scope of this PR but still if you are interested, happy to have you commit multi-agent design and we go review with PR. cc @wsl2000 @matchyc happy to see how @DengMinghua feel. |
The benchmark docs had drifted badly from the technical report — most notably LongMemEval was still listed as "0% (retrieval returns empty context)" when the paper reports 93.0% J-Score overall (N=500). Align all headline numbers to the paper as the source of truth: - LongMemEval: 0% -> 93.0% overall, with per-category J-Scores and the build/answer/judge stack; cross-ref PR #26/#27 for the MS lever. - LoCoMo: 82.8% (gpt-4.1-mini) -> 81.23% overall (paper Table 4 stack, gpt-4o-mini) vs ENGRAM / MemOS / Zep. - MuSiQue F1 -> 58.7 vs HippoRAG 2 49.3. - BABILong -> 85.0 vs ARMT (fine-tuned) 83.8. - ToMi -> 83.5 vs AutoToM 80.2. - Add a canonical "Paper results" section (Tables 3-5 + Fig. 4) to BENCHMARK.md and note that the reported stack is the proactive-substrate configuration, not a per-benchmark tuned ceiling. Claude-Session: https://claude.ai/code/session_01HCToJStvPhZbWGY2ZcBfy7 Co-authored-by: Claude <noreply@anthropic.com>
|
Just as a reference, update paper results to the main #29. |
For count/sum questions the reader runs on the pure no-hint context and we keep the HIGHER of (baseline count, NS count) — NS can only add, never subtract — so the agent complements rather than overrides the reader. Validated offline against the failing fixtures (ns_take_max_sim.py); a full-MS static-projection harness (ns_take_max_static.py) is included. Still gated behind --neural-symbolic (default OFF). MS result numbers are intentionally omitted (mid-tuning, not a final result). - neural_symbolic.py: parse_answer_count + take_max_answer. - run_eval.py: --neural-symbolic-take-max (default ON when --neural-symbolic); count/sum defers the hint and post-processes max(); records take_max_taken. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-skip + category quantities
The per-qid fixability analysis found ~6 MS failures are writer-gaps where the
operand was dropped at write time and lives in no node (so retrieval/reader/NS
cannot reach it). Targeted additions to the W1 _TYPED_ATTR_PROMPT, each validated
$0 on the exact failing raw turns with BOTH gpt-5.4-mini AND the real gpt-4o-mini
writer (all 6 captured correctly; the planned-skip rule correctly emits nothing
for the Tybee phantom):
- ACQUISITIONS / VIEWINGS: emit a got/bought/viewed item as a `name` even if
mentioned once (peace lily, succulent, 1-bedroom condo) -> 3a704032, gpt4_7fce9456.
- ENTITY-ANCHORED VALUES: a duration/quantity context must name its place/entity
("trip to Hawaii", not "family trip planning") -> edced276.
- PLANNED vs DONE: skip merely-planned/aspirational values -> aae3761f Tybee phantom.
- quantity def: capture embedded category counts ("8 edX courses", "25 rare coins")
-> 67e0d0f2, e3038f8c.
Only fires under --extract-typed-attributes (W1, already ON in the MS stack).
Real end-to-end flip still needs a live run (ingestion -> retrieval -> reader).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e (pre-PR)
Per-qid static verification of the 32 MS failures surfaced two cleanliness issues
introduced/found during the writer-fix pass:
1. W1 PLANNED-vs-DONE was too aggressive — it DROPPED real future-framed values
(verified live: Rachel's wedding date + age 32, a FreshMart 15% coupon both
vanished), which would regress currently-correct questions. Changed to
TAG-DON'T-DROP: never drop a stated value; only mark genuinely-undecided
items ("deciding between", "I think I'll go with") as PLANNED in context so a
later count can include/exclude per the question. Re-verified: Rachel/age,
FreshMart 15%, train $25/$60 all survive; the Tybee phantom is emitted tagged
"deciding, not taken" so the reader can exclude it from a completed-trips sum.
2. _extract_bridge_phrases mined sentence-initial interrogatives ("What",
"Which", "Did") as bogus bridge sub-queries (wasting a retrieval slot).
Expanded the stop set; real bridges (Bachelor, "average age of employees in
my department", Hawaii/NYC) are retained, the noise is gone.
Both validated $0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Summary
Builds on the shelved neural-symbolic agent (#26) to make the count/sum lever complementary with the normal reader for LongMemEval MS, plus targeted writer (W1) fixes for writer-gap failures. All changes are gated behind
--neural-symbolic(default OFF). W1 cues are under--extract-typed-attributes(already ON in the MS stack).1. Neural-symbolic take-max complementary router (count/sum)
For count/sum questions the reader runs on the pure no-hint context and we keep the higher of (baseline count, NS count) — NS can only add, never subtract — so the agent complements rather than overrides the reader.
--neural-symbolic-take-max(default ON when--neural-symbolicis set).2. W1 writer cues for writer-gap operands
Some MS failures are writer-gaps where the operand was dropped at write time and lives in no node (retrieval/reader can't reach it). Targeted
_TYPED_ATTR_PROMPTadditions, validated on the failing raw turns with the real gpt-4o-mini writer:3. Cleanups
_extract_bridge_phrasesno longer mines sentence-initial interrogatives ("What"/"Which"/"Did") as bogus retrieval sub-queries.Validation
neural_symbolic_selftest.py(unit fixtures) PASS.🤖 Generated with Claude Code