Skip to content

Fix NodeStatus flooding and GUI responsiveness issues - #88

Open
Bwooce wants to merge 2 commits into
dronecan:masterfrom
Bwooce:rate-limiting-only
Open

Fix NodeStatus flooding and GUI responsiveness issues#88
Bwooce wants to merge 2 commits into
dronecan:masterfrom
Bwooce:rate-limiting-only

Conversation

@Bwooce

@Bwooce Bwooce commented Jul 26, 2025

Copy link
Copy Markdown

Summary

  • Fix scheduler feedback loop causing 6000+ NodeStatus messages per second flooding the CAN bus
  • Fix GUI responsiveness by limiting frames processed per spin(0) call to prevent infinite loops
  • Maintain proper periodic timing while preventing callback overlap

Technical Details

  • Scheduler Fix: Prevent callback overlap in periodic scheduler that was causing runaway NodeStatus generation when callbacks took longer than the period
  • GUI Fix: Limit spin(0) to process maximum 100 frames per call to prevent GUI thread blocking

@Bwooce Bwooce changed the title Rate limiting only Rate limiting on NodeStatus messages from DroneCAN GUI Jul 26, 2025
@Bwooce
Bwooce force-pushed the rate-limiting-only branch from f384a34 to 383ce8e Compare July 26, 2025 13:16
@Bwooce Bwooce changed the title Rate limiting on NodeStatus messages from DroneCAN GUI Fix NodeStatus message flooding caused by scheduler feedback loop Jul 26, 2025
@Bwooce

Bwooce commented Jul 27, 2025

Copy link
Copy Markdown
Author

Apologies for the churn here, on further testing there were...more problems. I've attempted to keep this PR seperate from the ACK/NACK change one to facilitate testing/validation that it's not broken the world before the ACK/NACK change gets merged.

@Bwooce Bwooce changed the title Fix NodeStatus message flooding caused by scheduler feedback loop Fix NodeStatus flooding and GUI responsiveness issues Jul 27, 2025
@Bwooce
Bwooce force-pushed the rate-limiting-only branch from a1794f7 to d9acd2e Compare July 27, 2025 10:38
Developer and others added 2 commits October 29, 2025 12:43
When SLCAN ACK/NACK waits were removed to support adapters without
confirmation, the scheduler's timing assumptions broke. The periodic
scheduler would re-register the next NodeStatus event before the
current callback completed, causing a runaway loop when SLCAN frame
transmission included time.sleep() calls.

This fix prevents callback overlap by:
- Adding callback_running flag to track execution state
- Skipping overlapping executions and rescheduling for next period
- Using try/finally to ensure flag is always reset

Resolves 6000+ NodeStatus messages/second flooding issue while
maintaining proper 1Hz timing and DroneCAN compliance.
- Fix periodic scheduler callback overlap preventing NodeStatus flooding
- Add GUI responsiveness protection by limiting frames per spin cycle
- Improve transfer reassembly with proactive cleanup and shorter timeouts
- Fix dictionary iteration bug in transfer cleanup

These changes resolve NodeStatus flooding and GUI freezing issues
while improving frame decode reliability.
@tridge
tridge force-pushed the rate-limiting-only branch from d9acd2e to 7011c14 Compare October 29, 2025 01:43
@tridge

tridge commented Oct 29, 2025

Copy link
Copy Markdown
Member

@Bwooce can you describe how I can reproduce the issue?

@fallenmi fallenmi left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new callback_running guard does not stop the overdue callback loop described in this PR. sched.scheduler.run(blocking=False) executes due actions serially: this code registers the next absolute deadline before callback(), but finally clears the flag before the scheduler examines that event. When a callback takes at least one period, the next event is already overdue, sees callback_running == False, and runs immediately. That repeats within the same poll, so spin(0) can still remain in _poll_scheduler_and_get_next_deadline() and publish NodeStatus back-to-back.

I reproduced this without wall-clock timing using the repository's real Scheduler and a deterministic fake clock: period 1.0 seconds, callback duration 1.1 seconds, and a sentinel on the fifth call. One non-blocking poll invokes callbacks at [1.0, 2.1, 3.2, 4.3, 5.4] before reaching the sentinel on exact base 37f416fa54e61c9f4832ebe614c257fe66072225, exact head 7011c144d9eea60da1564c23a7f16d287c25bca4, and current GitHub merge 97e90c06f1d9c4748d94dbff4385a5c9ae522f73. The head therefore leaves the stated failure unchanged. Please make an overrun advance the queued deadline into the future (or otherwise skip missed ticks) and add a regression asserting that one scheduler poll returns after a slow periodic callback.

The full repository suite passes 58/58 on exact base, head, and current merge, but no existing test exercises periodic scheduler overruns. git diff --check also reports five trailing-whitespace errors in the submitted diff.

AI disclosure: I used OpenAI Codex to inspect the exact revisions, build and run the deterministic scheduler-overrun oracle, run the repository suites, and draft this review. I verified the commit IDs, fake-clock trace, source paths, test output, live merge parents, and conclusion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants