graphs: add Scheduler Health graph set to mavgraphs2.xml - #1709
Open
asikarastallion wants to merge 1 commit into
Open
graphs: add Scheduler Health graph set to mavgraphs2.xml#1709asikarastallion wants to merge 1 commit into
asikarastallion wants to merge 1 commit into
Conversation
## What
Adds three new MAVExplorer graphs under `PM/Scheduler Health/`, built
entirely from the existing `PM` (Performance Monitoring) log message.
No firmware changes, no new log fields, no MAVExplorer/pymavlink code
changes — graph definitions only (single file, `mavgraphs2.xml`).
1. **CPU Load Trend** — `PM.Load*0.1` (Load is logged at 10x the real
percentage). A filtered/averaged metric, useful for spotting a slow
drift toward CPU saturation over a flight rather than single spikes.
2. **Loop Overrun** — `PM.MaxT` plotted against two reference lines
computed dynamically from the loop rate ArduPilot itself measured
for that PM period (`PM.LR`), rather than a hardcoded budget:
- theoretical per-loop budget: `1e6 / PM.LR` µs
- ArduPilot's own "long loop" alarm threshold, 20% over budget:
`1.2e6 / PM.LR` µs (matches the definition used to increment
`PM.NLon`)
The overrun ratio `NLon/NL` (%) is plotted on the secondary axis,
so a single worst-case `MaxT` sample can be read alongside how
often overruns are actually occurring in that interval.
3. **Overrun Root Cause** — overlays `MaxT`/`NLon` with the `I2CC`,
`I2CI`, `SPIC` bus counters (secondary axis), to help separate
I/O-driven overruns from compute-driven ones.
## Relationship to the existing PM/Perf graph
`PM/Perf` (`PM.MaxT PM.NLon:2`) already exists and remains useful as a
quick raw view. This PR doesn't replace it — it adds a purpose-built
diagnostic layer on top: dynamic budget/alarm reference lines (instead
of eyeballing raw MaxT with no context), an overrun-frequency ratio,
and a root-cause overlay. Happy to fold these into `PM/Perf` directly
instead if maintainers prefer one graph over three — flagging this as
an open design question.
## Why a dynamic threshold instead of a fixed budget line
ArduPilot's own scheduler treats 0-20% over budget as normal (that's
literally the definition of when `PM.NLon` increments), so a single
fixed "MaxT - budget" line would flag harmless variance as a problem.
`PM.LR` — the filtered loop rate ArduPilot measured for that period —
is already present on every `PM` message, so the two reference lines
can be computed per-log with no parameter table lookups and no manual
user input, and they stay correct even if the achieved loop rate
drifts from the configured `SCHED_LOOP_RATE`.
(We initially explored reading `SCHED_LOOP_RATE` dynamically via the
log's parameter table instead. `PM.LR` turned out to be a strictly
simpler and more robust source for the same information — no
cross-message lookup needed — so we went with that. Happy to discuss
if there's a reason to prefer the parameter value in some case we
haven't considered.)
## Known limitations (documented inline in the graph descriptions)
- `PM` is normally logged at ~1Hz, so `MaxT` is a single worst-case
sample between messages, not proof of chronic overruns — the
`NLon/NL` ratio is the better signal for frequency.
- The scheduler only times the main loop; delays in other ChibiOS
threads (IO, DMA, logging) are invisible to these graphs.
- Motor arming commonly causes a `MaxT` spike (failsafe checks, EKF
alignment) — expected, not itself a scheduler problem. Not
auto-shaded in this version to keep the PR small; noted in the
description instead.
- The I2C/SPI counters in "Overrun Root Cause" are cumulative, not
rates — look at the slope at the time of the overrun, not the
absolute value.
## Testing
Verified all three expressions evaluate without errors and render
correctly against four real dataflash logs:
- Two quiet logs with `NLon=0` throughout — `MaxT` stays at/under the
alarm line, overrun ratio flat at 0%.
- One log with a real `NLon=1` overrun — `MaxT` spikes to ~3x budget
at boot/arm and the overrun-ratio axis spikes to ~7% at the same
moment, then both settle under budget for the rest of the flight,
consistent with the documented arm-spike behavior.
Only tested against Copter logs so far — the `PM` message and `LR`
field are shared across vehicles via `AP_Scheduler`, but additional
testing on Plane/Rover logs is welcome if anyone has some handy.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: asikarastallion <mserdarsokmen@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds three new MAVExplorer graphs under
PM/Scheduler Health/, built entirely from the existingPM(Performance Monitoring) log message. No firmware changes, no new log fields, no MAVExplorer/pymavlink code changes — graph definitions only (single file,mavgraphs2.xml).CPU Load Trend —
PM.Load*0.1(Load is logged at 10x the real percentage). A filtered/averaged metric, useful for spotting a slow drift toward CPU saturation over a flight rather than single spikes.Loop Overrun —
PM.MaxTplotted against two reference lines computed dynamically from the loop rate ArduPilot itself measured for that PM period (PM.LR), rather than a hardcoded budget:1e6 / PM.LRµs1.2e6 / PM.LRµs (matches the definition used to incrementPM.NLon)The overrun ratio
NLon/NL(%) is plotted on the secondary axis,so a single worst-case
MaxTsample can be read alongside howoften overruns are actually occurring in that interval.
Overrun Root Cause — overlays
MaxT/NLonwith theI2CC,I2CI,SPICbus counters (secondary axis), to help separate I/O-driven overruns from compute-driven ones.Relationship to the existing PM/Perf graph
PM/Perf(PM.MaxT PM.NLon:2) already exists and remains useful as a quick raw view. This PR doesn't replace it — it adds a purpose-built diagnostic layer on top: dynamic budget/alarm reference lines (instead of eyeballing raw MaxT with no context), an overrun-frequency ratio, and a root-cause overlay. Happy to fold these intoPM/Perfdirectly instead if maintainers prefer one graph over three — flagging this as an open design question.Why a dynamic threshold instead of a fixed budget line
ArduPilot's own scheduler treats 0-20% over budget as normal (that's literally the definition of when
PM.NLonincrements), so a single fixed "MaxT - budget" line would flag harmless variance as a problem.PM.LR— the filtered loop rate ArduPilot measured for that period — is already present on everyPMmessage, so the two reference lines can be computed per-log with no parameter table lookups and no manual user input, and they stay correct even if the achieved loop rate drifts from the configuredSCHED_LOOP_RATE.(We initially explored reading
SCHED_LOOP_RATEdynamically via the log's parameter table instead.PM.LRturned out to be a strictly simpler and more robust source for the same information — no cross-message lookup needed — so we went with that. Happy to discuss if there's a reason to prefer the parameter value in some case we haven't considered.)Known limitations (documented inline in the graph descriptions)
PMis normally logged at ~1Hz, soMaxTis a single worst-case sample between messages, not proof of chronic overruns — theNLon/NLratio is the better signal for frequency.MaxTspike (failsafe checks, EKF alignment) — expected, not itself a scheduler problem. Not auto-shaded in this version to keep the PR small; noted in the description instead.Testing
Verified all three expressions evaluate without errors and render correctly against four real dataflash logs:
NLon=0throughout —MaxTstays at/under the alarm line, overrun ratio flat at 0%.NLon=1overrun —MaxTspikes to ~3x budget at boot/arm and the overrun-ratio axis spikes to ~7% at the same moment, then both settle under budget for the rest of the flight, consistent with the documented arm-spike behavior.Only tested against Copter logs so far — the
PMmessage andLRfield are shared across vehicles viaAP_Scheduler, but additional testing on Plane/Rover logs is welcome if anyone has some handy.AI assistance disclosure: This contribution was developed with the assistance of Claude (Anthropic), as noted in the commit trailer. Claude helped analyze the PM log message fields and draft the graph expressions; I reviewed and tested the results against real dataflash logs as described above.