fix: serialize Lean REPL access and pinpoint search-tactic feedback - #37
Open
KrystianNowakowski wants to merge 2 commits into
Open
fix: serialize Lean REPL access and pinpoint search-tactic feedback#37KrystianNowakowski wants to merge 2 commits into
KrystianNowakowski wants to merge 2 commits into
Conversation
Issue 1 (run-terminating false positives): SearchTacticsDetectedFeedback dumped the entire declaration as its 'Locations', so the prover could not tell which tactic was flagged and stripped out innocent tactics (field_simp, apply) from a compiling proof. Report each offending tactic with its line number instead. The detection regex itself was already correct. Issue 2 (startup crashes, 0 iterations): a file that fails to compile comes back from lean_interact as a LeanError (message only, no declarations), so list_declarations_from_* blew up with an opaque AttributeError on '.declarations'. Detect LeanError and raise the actual Lean compile message.
The single AutoLeanServer subprocess is shared by all concurrent experiment samples, but run() only locked server creation, not command execution. Firing many full-Mathlib FileCommands at one REPL at once races the subprocess and trips its memory backstop, surfacing as spurious LeanErrors (the 0-iteration startup crashes). Add a run lock so commands are serialized; lake builds are unaffected (separate subprocesses).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two experiment-robustness fixes split out of #35 so they can be reviewed and merged
independently. Both surfaced during larger-scale experiment runs and are model-agnostic
(nothing DeepSeek-specific).
Serialize LeanInteract REPL access with a per-command lock
The single
AutoLeanServersubprocess is shared by all concurrent experiment samples, butrun()only locked server creation, not command execution. Firing many full-MathlibFileCommands at one REPL at once races the subprocess and trips its memory backstop,surfacing as spurious
LeanErrors (the 0-iteration startup crashes). A per-command runlock serializes execution; lake builds are unaffected (separate subprocesses).
Pinpoint search-tactic feedback and surface fatal Lean errors
SearchTacticsDetectedFeedbackdumped the entire declaration as its "Locations", so theprover could not tell which tactic was flagged and stripped innocent tactics
(
field_simp,apply) out of a compiling proof. Each offending tactic is now reportedwith its line number. The detection regex itself is unchanged.
lean_interactas aLeanError(messageonly, no declarations), so
list_declarations_from_*blew up with an opaqueAttributeErroron.declarations. It now detectsLeanErrorand raises the actualLean compile message.
Tests
Adds/keeps coverage:
test_lean_interact.py(lock serializes concurrent commands),test_cheat_detection.py(per-line tactic reporting, innocent tactics not flagged),test_lean_parsing.py(fatalLeanError→ clearRuntimeError). Full unit suite green.Note
Medium Risk
Serializing all LeanInteract commands can increase latency under heavy parallelism, but it fixes correctness of shared REPL use; parsing and feedback changes are localized with new unit tests.
Overview
Hardens concurrent experiment runs and prover feedback when Lean parsing or cheat detection fails.
LeanInteract now holds a per-command
asynciolock so only one REPL command runs at a time on the shared subprocess, avoiding races and spuriousLeanErrors when many samples hit the same server.Search-tactic rejection no longer puts the full declaration in
SearchTacticsDetectedFeedback.locations; it lists each flagged tactic asline N: <tactic>so the prover can fix onlysimp?/exact?-style lines instead of rewriting the whole proof.Declaration listing routes REPL results through
_bundle_response, which raisesRuntimeErrorwith Lean’s message when the response is a fatalLeanError(e.g. bad import) instead of failing on missing.declarations.Reviewed by Cursor Bugbot for commit 22fa709. Bugbot is set up for automated code reviews on this repo. Configure here.