Neural-symbolic MS agent (experimental, OFF by default, shelved)

[wsuli615]

⚠️ Interim work — NOT a final result. MS accuracy numbers are deliberately omitted; they are mid-tuning and unstable run-to-run. A single consolidated MS result will be reported once tuning is complete.

Summary

Adds the LongMemEval MS count/sum/arithmetic neural-symbolic agent — built, validated, and deliberately shelved (OFF by default). A focused structured-extraction LLM call reads the raw retrieved turns, enumerates operands for 5 families (count / sum / diff / date / age), computes the answer deterministically, and injects it as a RECALL_HINT for the reader to verify.

Gated behind --neural-symbolic (default OFF) / launcher env NEURAL_SYMBOLIC_FLAG — zero effect on any existing run unless explicitly enabled.

Status: shelved OFF pending further tuning

The agent's mechanism works (the reader adopts a concrete enumerated hint; raw-turn reading recovers writer-gap operands), but its blast radius (it fires on a large fraction of count/sum questions, many currently-correct) means the net effect is not yet favorable. It is kept OFF as a tagged, reversible scaffold while the count/sum approach is refined. No MS scores are asserted here.

What's kept (correct, offline-tested)

• 5-family deterministic compute + classifier exclusions.
• Compute fixes (to_number multiplier/sign bugs, SUM dedup, _norm_label over-merge, compare vs→Yes/No).
• $0 harnesses: neural_symbolic_selftest.py, ns_static_analysis.py, ns_smoke_compare.py, neural_symbolic_replay.py.

Safety / docs

• Rule documented in CLAUDE.md (Critical Operational Rules): do NOT enable for benchmark/paper runs without re-validation.
• Rollback tag iter33-ms-pre-symbolic.

🤖 Generated with Claude Code

[duanyiqun]

@claude review

[claude]

Claude encountered an error —— View job

---

I'll analyze this and get back to you.

[duanyiqun]

Code Review — Neural-symbolic MS agent (#26)

TL;DR: Cleanly isolated to benchmarks/longmemeval/, default-OFF is correctly enforced, compute layer is pure and tested, no eval/exec/shell-exec. Mergeable as the intended tagged, reversible, OFF-by-default scaffold. A few small things worth fixing before merge (items 1–4 below).

What this does

A new neural_symbolic.py (884 LOC): one structured-extraction LLM call reads the raw retrieved turns, enumerates operands for 5 families (count / sum / diff / date / age), computes the answer deterministically in Python, and injects it as a RECALL_HINT for the reader to verify. No src/cognifold/ production code is touched.

The PR's own verdict is honest: two paid A/B smokes + a full-MS static projection show it is not net-positive (fires on 76/133; collateral surface 47 ≈ 1.6× the 29 win opportunities; projected 68–76% ≤ the 75.9% baseline), so it lands shelved/OFF as a reversible scaffold (rollback tag iter33-ms-pre-symbolic).

Strengths

• Default-OFF enforced at all three layers: CLI --neural-symbolic default=False; ${NEURAL_SYMBOLIC_FLAG:-} expands to empty when unset (no arg → CLI default); the whole runtime block is guarded by getattr(args, "neural_symbolic", False) and additionally gated on symbolic_result is None, so with the flag off the LLM call and the injection never run.
• No eval/exec/shell execution in the agent; arithmetic uses native operators only and to_number uses float(). Safe by design.
• $0 offline self-test with 39 fixtures (including adversarial mis-route guards and regression cases like "10 minutes"→10, $1.2m→1.2e6). Good coverage of the compute + classifier layer.

Worth addressing

1. Only un-gated change: the new reader-prompt rule TR-only optimization: target the 26 TR failure modes from iter27 baseline #4 (describing the CROSSCHECK block) in run_eval.py is injected on every run, so the OFF path is not byte-identical to baseline. The wording is conditional ("If such a block appears"), so it should be behaviorally inert when no block is present — please confirm.
2. Stale CLI help: the --neural-symbolic help text says extraction runs "on the writer config", but the code actually passes the reader/reasoning config (matching the CHANGELOG). Fix the help string.
3. Broad except Exception: return None in compute() silently swallows all compute-logic errors, degrading to a no-hint fall-through instead of surfacing the bug — makes future debugging harder.
4. Hardcoded paths/URLs: the self-test docs hardcode /tmp/cf-unified/...; the replay harness inlines provider base URLs and default model ids. These belong in config.

Notes (low risk, benchmark context)

5. Extraction prompt interpolates node text and the question verbatim (no escaping) → a prompt-injection surface if ever reused over untrusted memories; ${NEURAL_SYMBOLIC_FLAG:-} is expanded unquoted into the command line. Both acceptable for a benchmark harness.
6. The long comment block above render_neural_symbolic_block describes the reverted "two-directional render" experiment, i.e. history rather than the current lower-bound-floor behavior — consider trimming to avoid confusing future readers.

Verdict: Approve to merge as a default-OFF, tagged, reversible scaffold. Items 1 and 2 are the most worthwhile to clean up first.

🤖 Generated with Claude Code

---

Generated by Claude Code