Skip to content

graphs: add Scheduler Health graph set to mavgraphs2.xml - #1709

Open
asikarastallion wants to merge 1 commit into
ArduPilot:masterfrom
asikarastallion:feature/scheduler-health-graphs
Open

graphs: add Scheduler Health graph set to mavgraphs2.xml#1709
asikarastallion wants to merge 1 commit into
ArduPilot:masterfrom
asikarastallion:feature/scheduler-health-graphs

Conversation

@asikarastallion

@asikarastallion asikarastallion commented Jul 23, 2026

Copy link
Copy Markdown

What

Adds three new MAVExplorer graphs under PM/Scheduler Health/, built entirely from the existing PM (Performance Monitoring) log message. No firmware changes, no new log fields, no MAVExplorer/pymavlink code changes — graph definitions only (single file, mavgraphs2.xml).

  1. CPU Load TrendPM.Load*0.1 (Load is logged at 10x the real percentage). A filtered/averaged metric, useful for spotting a slow drift toward CPU saturation over a flight rather than single spikes.

  2. Loop OverrunPM.MaxT plotted against two reference lines computed dynamically from the loop rate ArduPilot itself measured for that PM period (PM.LR), rather than a hardcoded budget:

    • theoretical per-loop budget: 1e6 / PM.LR µs
    • ArduPilot's own "long loop" alarm threshold, 20% over budget: 1.2e6 / PM.LR µs (matches the definition used to increment PM.NLon)

    The overrun ratio NLon/NL (%) is plotted on the secondary axis,
    so a single worst-case MaxT sample can be read alongside how
    often overruns are actually occurring in that interval.

  3. Overrun Root Cause — overlays MaxT/NLon with the I2CC, I2CI, SPIC bus counters (secondary axis), to help separate I/O-driven overruns from compute-driven ones.

Relationship to the existing PM/Perf graph

PM/Perf (PM.MaxT PM.NLon:2) already exists and remains useful as a quick raw view. This PR doesn't replace it — it adds a purpose-built diagnostic layer on top: dynamic budget/alarm reference lines (instead of eyeballing raw MaxT with no context), an overrun-frequency ratio, and a root-cause overlay. Happy to fold these into PM/Perf directly instead if maintainers prefer one graph over three — flagging this as an open design question.

Why a dynamic threshold instead of a fixed budget line

ArduPilot's own scheduler treats 0-20% over budget as normal (that's literally the definition of when PM.NLon increments), so a single fixed "MaxT - budget" line would flag harmless variance as a problem. PM.LR — the filtered loop rate ArduPilot measured for that period — is already present on every PM message, so the two reference lines can be computed per-log with no parameter table lookups and no manual user input, and they stay correct even if the achieved loop rate drifts from the configured SCHED_LOOP_RATE.

(We initially explored reading SCHED_LOOP_RATE dynamically via the log's parameter table instead. PM.LR turned out to be a strictly simpler and more robust source for the same information — no cross-message lookup needed — so we went with that. Happy to discuss if there's a reason to prefer the parameter value in some case we haven't considered.)

Known limitations (documented inline in the graph descriptions)

  • PM is normally logged at ~1Hz, so MaxT is a single worst-case sample between messages, not proof of chronic overruns — the NLon/NL ratio is the better signal for frequency.
  • The scheduler only times the main loop; delays in other ChibiOS threads (IO, DMA, logging) are invisible to these graphs.
  • Motor arming commonly causes a MaxT spike (failsafe checks, EKF alignment) — expected, not itself a scheduler problem. Not auto-shaded in this version to keep the PR small; noted in the description instead.
  • The I2C/SPI counters in "Overrun Root Cause" are cumulative, not rates — look at the slope at the time of the overrun, not the absolute value.

Testing

Verified all three expressions evaluate without errors and render correctly against four real dataflash logs:

  • Two quiet logs with NLon=0 throughout — MaxT stays at/under the alarm line, overrun ratio flat at 0%.
  • One log with a real NLon=1 overrun — MaxT spikes to ~3x budget at boot/arm and the overrun-ratio axis spikes to ~7% at the same moment, then both settle under budget for the rest of the flight, consistent with the documented arm-spike behavior.

Only tested against Copter logs so far — the PM message and LR field are shared across vehicles via AP_Scheduler, but additional testing on Plane/Rover logs is welcome if anyone has some handy.

AI assistance disclosure: This contribution was developed with the assistance of Claude (Anthropic), as noted in the commit trailer. Claude helped analyze the PM log message fields and draft the graph expressions; I reviewed and tested the results against real dataflash logs as described above.

## What

Adds three new MAVExplorer graphs under `PM/Scheduler Health/`, built
entirely from the existing `PM` (Performance Monitoring) log message.
No firmware changes, no new log fields, no MAVExplorer/pymavlink code
changes — graph definitions only (single file, `mavgraphs2.xml`).

1. **CPU Load Trend** — `PM.Load*0.1` (Load is logged at 10x the real
   percentage). A filtered/averaged metric, useful for spotting a slow
   drift toward CPU saturation over a flight rather than single spikes.

2. **Loop Overrun** — `PM.MaxT` plotted against two reference lines
   computed dynamically from the loop rate ArduPilot itself measured
   for that PM period (`PM.LR`), rather than a hardcoded budget:
   - theoretical per-loop budget: `1e6 / PM.LR` µs
   - ArduPilot's own "long loop" alarm threshold, 20% over budget:
     `1.2e6 / PM.LR` µs (matches the definition used to increment
     `PM.NLon`)

   The overrun ratio `NLon/NL` (%) is plotted on the secondary axis,
   so a single worst-case `MaxT` sample can be read alongside how
   often overruns are actually occurring in that interval.

3. **Overrun Root Cause** — overlays `MaxT`/`NLon` with the `I2CC`,
   `I2CI`, `SPIC` bus counters (secondary axis), to help separate
   I/O-driven overruns from compute-driven ones.

## Relationship to the existing PM/Perf graph

`PM/Perf` (`PM.MaxT PM.NLon:2`) already exists and remains useful as a
quick raw view. This PR doesn't replace it — it adds a purpose-built
diagnostic layer on top: dynamic budget/alarm reference lines (instead
of eyeballing raw MaxT with no context), an overrun-frequency ratio,
and a root-cause overlay. Happy to fold these into `PM/Perf` directly
instead if maintainers prefer one graph over three — flagging this as
an open design question.

## Why a dynamic threshold instead of a fixed budget line

ArduPilot's own scheduler treats 0-20% over budget as normal (that's
literally the definition of when `PM.NLon` increments), so a single
fixed "MaxT - budget" line would flag harmless variance as a problem.
`PM.LR` — the filtered loop rate ArduPilot measured for that period —
is already present on every `PM` message, so the two reference lines
can be computed per-log with no parameter table lookups and no manual
user input, and they stay correct even if the achieved loop rate
drifts from the configured `SCHED_LOOP_RATE`.

(We initially explored reading `SCHED_LOOP_RATE` dynamically via the
log's parameter table instead. `PM.LR` turned out to be a strictly
simpler and more robust source for the same information — no
cross-message lookup needed — so we went with that. Happy to discuss
if there's a reason to prefer the parameter value in some case we
haven't considered.)

## Known limitations (documented inline in the graph descriptions)

- `PM` is normally logged at ~1Hz, so `MaxT` is a single worst-case
  sample between messages, not proof of chronic overruns — the
  `NLon/NL` ratio is the better signal for frequency.
- The scheduler only times the main loop; delays in other ChibiOS
  threads (IO, DMA, logging) are invisible to these graphs.
- Motor arming commonly causes a `MaxT` spike (failsafe checks, EKF
  alignment) — expected, not itself a scheduler problem. Not
  auto-shaded in this version to keep the PR small; noted in the
  description instead.
- The I2C/SPI counters in "Overrun Root Cause" are cumulative, not
  rates — look at the slope at the time of the overrun, not the
  absolute value.

## Testing

Verified all three expressions evaluate without errors and render
correctly against four real dataflash logs:
- Two quiet logs with `NLon=0` throughout — `MaxT` stays at/under the
  alarm line, overrun ratio flat at 0%.
- One log with a real `NLon=1` overrun — `MaxT` spikes to ~3x budget
  at boot/arm and the overrun-ratio axis spikes to ~7% at the same
  moment, then both settle under budget for the rest of the flight,
  consistent with the documented arm-spike behavior.

Only tested against Copter logs so far — the `PM` message and `LR`
field are shared across vehicles via `AP_Scheduler`, but additional
testing on Plane/Rover logs is welcome if anyone has some handy.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: asikarastallion <mserdarsokmen@gmail.com>
Copilot AI review requested due to automatic review settings July 23, 2026 22:53

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants