fix(reasoning): use regex for plan readiness detection (closes #6204) - #6655
fix(reasoning): use regex for plan readiness detection (closes #6204)#6655Diwak4r wants to merge 1 commit into
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
Included review availability: Your plan provides up to 10 included reviews per hour; 8 remain after this review. 📝 WalkthroughWalkthroughPlanning readiness detection now uses a shared, case-insensitive regex helper that recognizes standalone ChangesPlan readiness detection
Merge Risk: ⚪ Minimal · up to Planning readiness detection now accepts standalone READY responses case-insensitively while excluding NOT READY, avoiding unnecessary refinement attempts. No current merge-blocking risk remains. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@lib/crewai/src/crewai/utilities/reasoning_handler.py`:
- Line 413: Update AgentReasoning._is_plan_ready and its fallback callers to
recognize readiness declarations without treating JSON field names as readiness:
preserve structured parsed ready booleans, exclude matches for the "ready" JSON
key, and require one-or-more whitespace after "not" so both NOT READY and NOT
READY are negative. Add regression coverage for READY, I'm ready!, NOT READY,
NOT READY, and {"ready": false}.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: df5f32be-656a-4247-9186-3174f8a95ecf
📒 Files selected for processing (1)
lib/crewai/src/crewai/utilities/reasoning_handler.py
| response_str, | ||
| [], | ||
| "READY: I am ready to execute the task." in response_str, | ||
| AgentReasoning._is_plan_ready(response_str), |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="lib/crewai/src/crewai/utilities/reasoning_handler.py"
echo "== file exists and line count =="
wc -l "$file"
echo "== outline around relevant symbols =="
ast-grep outline "$file" --match 'AgentReasoning' --view expanded || true
echo "== relevant line slices =="
for range in "90-140" "380-445" "550-625"; do
echo "--- ${file}:${range} ---"
sed -n "${range}p" "$file" | nl -ba -v "${range#*-}"
done
echo "== searches for _is_plan_ready and ready regex =="
rg -n "_is_plan_ready|ready|NOT\s+READY|NOT\ \+READY|ready\b" "$file"Repository: crewAIInc/crewAI
Length of output: 1632
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="lib/crewai/src/crewai/utilities/reasoning_handler.py"
python3 - <<'PY'
from pathlib import Path
file = Path("lib/crewai/src/crewai/utilities/reasoning_handler.py")
text = file.read_text()
for start, end in [(90,140), (240,320), (380,450), (550,630)]:
print(f"--- {file}:{start}-{end} ---")
for i, line in enumerate(text.splitlines(), 1):
if start <= i <= end:
print(f"{i:4d}\t{line}")
print("--- searches ---")
for pat in ["_is_plan_ready", "ready", "NOT", "\\bnot", "parse_planning_response"]:
print(f"-- {pat} --")
for i, line in enumerate(text.splitlines(), 1):
if pat.lower() in line.lower() or pat in line:
print(f"{i:4d}\t{line}")
PY
echo "== behavioral probe from source regex =="
python3 - <<'PY'
import ast
from pathlib import Path
file = Path("lib/crewai/src/crewai/utilities/reasoning_handler.py")
source = file.read_text()
tree = ast.parse(source, filename=str(file))
compiled_re = None
for node in ast.walk(tree):
if isinstance(node, ast.AnnAssign):
annot = node.target.id if isinstance(node.target, ast.Name) else None
if annot == "plan_ready_regex":
compiled_re = ast.literal_eval(node.value)
break
print("compiled_regex_string:", repr(compiled_re))
import re
if compiled_re is not None:
pattern = re.compile(compiled_re, re.IGNORECASE)
cases = [
"READY",
"I'm ready!",
"NOT READY",
"NOT READY",
'{"ready": false}',
'{"ready": true}',
"not ready",
]
for case in cases:
m = pattern.search(case)
print(repr(case), "=>", m.group() if m else None)
PYRepository: crewAIInc/crewAI
Length of output: 16209
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
import re
pattern = re.compile(r"(?<!\bnot\s)\bready\b", re.IGNORECASE)
cases = [
"READY",
"I'm ready!",
"NOT READY",
"NOT READY",
'{"ready": false}',
'{"ready": true}',
"not ready",
"not ready",
"ready",
]
for case in cases:
m = pattern.search(case)
print(f"{case!r} => {m.group()!r if m else None}")
PYRepository: crewAIInc/crewAI
Length of output: 327
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
import re
pattern = re.compile(r"(?<!\bnot\s)\bready\b", re.IGNORECASE)
cases = [
"READY",
"I'm ready!",
"NOT READY",
"NOT READY",
'{"ready": false}',
'{"ready": true}',
"not ready",
"not ready",
"ready",
]
for case in cases:
m = pattern.search(case)
print(repr(case), "=>", m.group() if m else None)
PYRepository: crewAIInc/crewAI
Length of output: 349
Avoid treating every ready token as a readiness declaration.
The fallback paths pass the response text directly into _is_plan_ready, where "{"ready": false}" matches the JSON field name and NOT READY/not ready matches because the lookbehind only handles one whitespace. This can skip refinement for non-ready plans. Preserve parsed structured ready booleans, exclude JSON field names, match one-or-more whitespace after not, and add regression tests for READY, I'm ready!, NOT READY, NOT READY, and {"ready": false}.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/crewai/src/crewai/utilities/reasoning_handler.py` at line 413, Update
AgentReasoning._is_plan_ready and its fallback callers to recognize readiness
declarations without treating JSON field names as readiness: preserve structured
parsed ready booleans, exclude matches for the "ready" JSON key, and require
one-or-more whitespace after "not" so both NOT READY and NOT READY are
negative. Add regression coverage for READY, I'm ready!, NOT READY, NOT READY,
and {"ready": false}.
a1a3b61 to
d126614
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
…ct string match The readiness check was matching only the exact phrase 'READY: I am ready to execute the task.' but the refine plan prompt instructs the model to conclude with just 'READY' or 'NOT READY'. Models responding with standalone 'READY' were falsely detected as NOT READY. Replace the three rigid substring checks with a case- insensitive regex that matches standalone READY while excluding NOT READY via negative lookbehind. Closes crewAIInc#6204
d126614 to
1b06e11
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
Description
The reasoning plan readiness detection was matching only the exact phrase
"READY: I am ready to execute the task."in three places withinreasoning_handler.py. However, the refine plan prompt instructs the model to "Conclude with READY or NOT READY" — a much shorter format. Models responding with standaloneREADYwere falsely detected as NOT READY, causing the system to loop through unnecessary refinement attempts.This affected:
Fix
Replace all three rigid substring checks with a case-insensitive regex that detects standalone
READYwhile excludingNOT READYvia a negative lookbehind:This correctly handles:
READY— standalone ✅READY: I am ready to execute the task.— long form ✅Ready/ready— case variants ✅NOT READY— correctly rejected ❌I'm ready!— in-context usage ✅NOTREADY(no space) — correctly rejected ❌Related Issue
Closes #6204