Skip to content

[M4] Rule-based no-progress detector baseline#48

Open
askmy-stack wants to merge 1 commit into
mainfrom
feat/parallax-22-no-progress-detector
Open

[M4] Rule-based no-progress detector baseline#48
askmy-stack wants to merge 1 commit into
mainfrom
feat/parallax-22-no-progress-detector

Conversation

@askmy-stack

Copy link
Copy Markdown
Owner

Summary

  • Adds NoProgressDetector in baselines/rules/detectors.py, a rule-based Milestone-4 baseline that flags planning-loop / stagnation failures via two model-independent signals: consecutive repeated actions (same tool name + arguments) and flat goal progress over a configurable window.
  • Records goal_progress on each TraceEvent's attributes in TraceCollector.record so detectors can read progress deltas directly from the trace (previously it was only tracked as a parallel list on the collector).
  • Adds runtime/features/progress.py with the underlying repeated_action_step / flat_progress_step helpers, following the pattern of runtime/features/mismatch.py.
  • Wires the detector into all_detectors() (and thus baselines/rules/__init__.py exports), so it automatically participates in ras experiment run --compare.
  • Regenerates experiments/results/tool-semantic-drift-001{.md,-report.json} to show the new detector row (no false alarm on this tool-drift case, as expected — it targets planning loops, not semantic drift).
  • Updates the README baseline results table to include the new detector.

Testing

  • New tests/unit/test_no_progress_detector.py: repeated-action detection, flat-goal-progress detection, no false alarm on non-consecutive repeats / monotonically increasing progress, registration in all_detectors(), and no false alarms on the existing clean/drifted scripted episodes.
  • make ci (ruff lint + format check, mypy, pytest) passes locally — 31/31 tests pass.
  • Manually ran ras experiment run --case tool-semantic-drift-001 --compare and confirmed no_progress appears in both the JSON and Markdown reports.

Fixes #22

Test plan

  • make ci passes
  • ras experiment run --case tool-semantic-drift-001 --compare shows no_progress in detector comparison output
  • Unit tests cover repeated-action and flat-progress detection paths plus false-alarm avoidance

Made with Cursor

Detects planning-loop / stagnation failures via two model-independent
signals: consecutive repeated actions (same name + arguments) and flat
goal progress over a window of steps. Wires the new NoProgressDetector
into all_detectors() so it participates in `ras experiment run
--compare`, and records goal_progress on each trace event so detectors
can read it.

Fixes #22

Co-authored-by: Cursor <cursoragent@cursor.com>
@askmy-stack askmy-stack self-assigned this Jul 19, 2026
@askmy-stack

Copy link
Copy Markdown
Owner Author

@askmy-stack Checklist for review:

  • Detector implemented (NoProgressDetector in baselines/rules/detectors.py) covering both repeated-action and flat-goal-progress signals
  • Unit tests added (tests/unit/test_no_progress_detector.py) — repeated actions, flat progress, no false alarms, registration check
  • Wired into all_detectors() and baselines/rules/__init__.py exports
  • Appears in ras experiment run --compare output (JSON + Markdown reports regenerated)
  • make ci passes locally (lint, typecheck, tests)

Note: --add-reviewer askmy-stack had no effect since askmy-stack is also the PR author — GitHub doesn't allow self-review requests. Assignee was set successfully.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[M4] Rule-based no-progress detector baseline

1 participant