Skip to content

MS: neural-symbolic take-max router + W1 writer fixes - #27

Merged
wsuli615 merged 3 commits into
OpenUploading:ms-only-iterfrom
wsuli615:ns-ms-fixes
Jul 1, 2026
Merged

MS: neural-symbolic take-max router + W1 writer fixes#27
wsuli615 merged 3 commits into
OpenUploading:ms-only-iterfrom
wsuli615:ns-ms-fixes

Conversation

@wsuli615

@wsuli615 wsuli615 commented Jun 20, 2026

Copy link
Copy Markdown
Contributor

⚠️ Interim work — NOT a final result. MS accuracy numbers are deliberately omitted; they are mid-tuning and unstable run-to-run. A single consolidated MS result will be reported once tuning is complete.

Summary

Builds on the shelved neural-symbolic agent (#26) to make the count/sum lever complementary with the normal reader for LongMemEval MS, plus targeted writer (W1) fixes for writer-gap failures. All changes are gated behind --neural-symbolic (default OFF). W1 cues are under --extract-typed-attributes (already ON in the MS stack).

1. Neural-symbolic take-max complementary router (count/sum)

For count/sum questions the reader runs on the pure no-hint context and we keep the higher of (baseline count, NS count) — NS can only add, never subtract — so the agent complements rather than overrides the reader. --neural-symbolic-take-max (default ON when --neural-symbolic is set).

2. W1 writer cues for writer-gap operands

Some MS failures are writer-gaps where the operand was dropped at write time and lives in no node (retrieval/reader can't reach it). Targeted _TYPED_ATTR_PROMPT additions, validated on the failing raw turns with the real gpt-4o-mini writer:

  • ACQUISITIONS / VIEWINGS → captures got/bought/viewed items.
  • ENTITY-ANCHORED VALUES → a duration's context must name its place.
  • PLANNED / FUTURE STATUS → tag-don't-drop: never drop a stated future value; only mark genuinely-undecided items so a count can exclude a phantom.
  • quantity def now captures embedded category counts.

3. Cleanups

  • _extract_bridge_phrases no longer mines sentence-initial interrogatives ("What"/"Which"/"Did") as bogus retrieval sub-queries.

Validation

  • ruff + pyright clean; neural_symbolic_selftest.py (unit fixtures) PASS.
  • Writer cues + take-max logic validated offline against the failing fixtures.
  • End-to-end MS confirmation is still in progress; no MS scores are reported here.

🤖 Generated with Claude Code

@wsuli615
wsuli615 requested a review from duanyiqun June 20, 2026 22:42
@zsxh1990

Copy link
Copy Markdown

Curious — the recent MS work moves intent through a writer/reader split. When you serve via MCP + HTTP + CLI simultaneously, how does intent state stay consistent across surfaces? Do you treat the graph as a single source of truth with process-level locking, or does each surface hold its own intent buffer that reconciles async?

Working on a multi-agent collab layer and trying to figure out the right model here. Especially interested in what happens when an intent crystallizes while a query agent is mid-retrieval — does the reader see the new intent on the next call, or does it get surfaced through some other channel?

@duanyiqun

duanyiqun commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

Curious — the recent MS work moves intent through a writer/reader split. When you serve via MCP + HTTP + CLI simultaneously, how does intent state stay consistent across surfaces? Do you treat the graph as a single source of truth with process-level locking, or does each surface hold its own intent buffer that reconciles async?

Working on a multi-agent collab layer and trying to figure out the right model here. Especially interested in what happens when an intent crystallizes while a query agent is mid-retrieval — does the reader see the new intent on the next call, or does it get surfaced through some other channel?

@zsxh1990 Great question, happy to continue in discussions if useful.
This repo hasn't been well designed for multi-agent synchronisation/or compete status -- it's more like a proactive always on but massive amount of single agents. Also a serving topology is missing now for massive serve. But I do think multi-agent would be an important aspect.

My point of view (may be wrong) is the design is really depends on tasks. If we are discussing auto-research discovery etc, then we need a hard synchronise and consistent truth source. If we are discussing say social sim, we may need a bit more uncertainty.

If I were designing the multi-surface concurrent version, I'd keep the graph as the SoT and avoid per-surface mutable buffers (that turns a single-store problem into multi-master conflict resolution — the hardest thing in distributed systems, for no benefit when co-located). Concretely:

( below are my thinkings however, I speak so claude rephrase it a bit.

  • Push consistency into the store as an append-only mutation log (CogniFold already has an auditable rewrite log), giving you a natural serialization point + snapshot reads instead of a coarse process lock.
  • Snapshot-consistent reads: a retrieval reads the graph as of its start version. An intent that crystallizes mid-retrieval is not injected into the in-flight read — it becomes visible on the next query. That keeps the race deterministic.
  • For the proactive side, emit a change-stream off the commit log (CQRS-ish): when a write batch mints/updates an INTENT, publish it; surfaces subscribe and decide whether to surface it or re-query. So the reader sees it via the next call (pull) or a notification channel (push) — your choice based on latency needs, but the graph stays the only authority.
  • I'd only reach for per-surface buffers + async reconciliation (CRDTs / vector clocks) if offline/partitioned writes are a hard requirement — otherwise it's a large complexity tax.

This is a bit beyond the scope of this PR but still if you are interested, happy to have you commit multi-agent design and we go review with PR. cc @wsl2000 @matchyc happy to see how @DengMinghua feel.

duanyiqun added a commit that referenced this pull request Jun 22, 2026
The benchmark docs had drifted badly from the technical report — most
notably LongMemEval was still listed as "0% (retrieval returns empty
context)" when the paper reports 93.0% J-Score overall (N=500).

Align all headline numbers to the paper as the source of truth:
- LongMemEval: 0% -> 93.0% overall, with per-category J-Scores and the
  build/answer/judge stack; cross-ref PR #26/#27 for the MS lever.
- LoCoMo: 82.8% (gpt-4.1-mini) -> 81.23% overall (paper Table 4 stack,
  gpt-4o-mini) vs ENGRAM / MemOS / Zep.
- MuSiQue F1 -> 58.7 vs HippoRAG 2 49.3.
- BABILong -> 85.0 vs ARMT (fine-tuned) 83.8.
- ToMi -> 83.5 vs AutoToM 80.2.
- Add a canonical "Paper results" section (Tables 3-5 + Fig. 4) to
  BENCHMARK.md and note that the reported stack is the proactive-substrate
  configuration, not a per-benchmark tuned ceiling.


Claude-Session: https://claude.ai/code/session_01HCToJStvPhZbWGY2ZcBfy7

Co-authored-by: Claude <noreply@anthropic.com>
@duanyiqun

Copy link
Copy Markdown
Contributor

Just as a reference, update paper results to the main #29.
Please check that our different branches may have different performance from the benchmark performance, which may vary, and synchronize new scores to the main if it has whenever higher or lower. cc @wsl2000 check if you want add more comments.

wsuli615 and others added 3 commits June 22, 2026 09:40
For count/sum questions the reader runs on the pure no-hint context and we keep the
HIGHER of (baseline count, NS count) — NS can only add, never subtract — so the agent
complements rather than overrides the reader. Validated offline against the failing
fixtures (ns_take_max_sim.py); a full-MS static-projection harness (ns_take_max_static.py)
is included. Still gated behind --neural-symbolic (default OFF). MS result numbers are
intentionally omitted (mid-tuning, not a final result).

- neural_symbolic.py: parse_answer_count + take_max_answer.
- run_eval.py: --neural-symbolic-take-max (default ON when --neural-symbolic); count/sum
  defers the hint and post-processes max(); records take_max_taken.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-skip + category quantities

The per-qid fixability analysis found ~6 MS failures are writer-gaps where the
operand was dropped at write time and lives in no node (so retrieval/reader/NS
cannot reach it). Targeted additions to the W1 _TYPED_ATTR_PROMPT, each validated
$0 on the exact failing raw turns with BOTH gpt-5.4-mini AND the real gpt-4o-mini
writer (all 6 captured correctly; the planned-skip rule correctly emits nothing
for the Tybee phantom):

- ACQUISITIONS / VIEWINGS: emit a got/bought/viewed item as a `name` even if
  mentioned once (peace lily, succulent, 1-bedroom condo) -> 3a704032, gpt4_7fce9456.
- ENTITY-ANCHORED VALUES: a duration/quantity context must name its place/entity
  ("trip to Hawaii", not "family trip planning") -> edced276.
- PLANNED vs DONE: skip merely-planned/aspirational values -> aae3761f Tybee phantom.
- quantity def: capture embedded category counts ("8 edX courses", "25 rare coins")
  -> 67e0d0f2, e3038f8c.

Only fires under --extract-typed-attributes (W1, already ON in the MS stack).
Real end-to-end flip still needs a live run (ingestion -> retrieval -> reader).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e (pre-PR)

Per-qid static verification of the 32 MS failures surfaced two cleanliness issues
introduced/found during the writer-fix pass:

1. W1 PLANNED-vs-DONE was too aggressive — it DROPPED real future-framed values
   (verified live: Rachel's wedding date + age 32, a FreshMart 15% coupon both
   vanished), which would regress currently-correct questions. Changed to
   TAG-DON'T-DROP: never drop a stated value; only mark genuinely-undecided
   items ("deciding between", "I think I'll go with") as PLANNED in context so a
   later count can include/exclude per the question. Re-verified: Rachel/age,
   FreshMart 15%, train $25/$60 all survive; the Tybee phantom is emitted tagged
   "deciding, not taken" so the reader can exclude it from a completed-trips sum.

2. _extract_bridge_phrases mined sentence-initial interrogatives ("What",
   "Which", "Did") as bogus bridge sub-queries (wasting a retrieval slot).
   Expanded the stop set; real bridges (Bachelor, "average age of employees in
   my department", Hawaii/NYC) are retained, the noise is gone.

Both validated $0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wsuli615
wsuli615 merged commit e2bb67a into OpenUploading:ms-only-iter Jul 1, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants