feat: rebase OTAFIX onto adafruit mainline 0.11.0 (#25)

[jamesarich]

Checklist

• PR title specifically describes the change
• tools/build_all.py passes — 17/17 locally, GCC 13.3.Rel1
• Tested on real hardware — nothing here has been flashed yet; every board needs re-validation before this is anything but a draft

---

Description of Change

The long-term half of #25. Branch is adafruit/master (c67f0bc, 0.11.0) with OTAFIX re-applied on top as a thin delta — 10 commits instead of the 65+ oltaco carries. Flash for RAK4631 goes 34500 → 33856 bytes; SRAM 31330 → 31198.

Layers

1. Boards — matrix pruned from mainline's 51 to our 17. Mainline changed the board.h contract (BUTTON_DFU/BUTTON_DFU_OTA replace BUTTON_1/BUTTON_2, BUTTONS_NUMBER gone, PINNUM provided by boards.h); the 14 boards only we ship were translated mechanically. The 3 boards mainline also has (t1000_e, both xiao) keep mainline's board.h and only gain the per-board DFU name in board.mk. board.cmake dropped (CMake was never wired).
2. Display — our screen.c/images.c (size-agnostic, Meshtastic-branded), ST7735 init added beside mainline's ST7789, UF2_VERSION_BASE + submodule versions in the version string.
3. DFU — cherry-picks from oltaco's line, see ledger.
4. Repo identity — CI workflow, AGENTS.md/README/CONTRIBUTING/etc, renovate, changelog. s340 SoftDevice dropped (no ANT boards).

Reconciliation ledger — OTAFIX commit → outcome on the 0.11 base

OTAFIX	Outcome
25454a8 HCI_RX_BUF_QUEUE_SIZE 16	obsolete — mainline has 16
3f7312e OTA resiliency (GPREGRET=0xA8 unconditional)	obsolete — mainline has the refined if (_ota_was_connected) form; default-to-OTA covers the no-app case
557f9cb per-board advertising name	obsolete — mainline guards DEVICE_NAME with #ifndef; board.mk -DDEVICE_NAME works as-is
f9f9f51 display in OTA mode	picked (main.c hunk; screen.c already ours)
afc721a high MTU	obsolete — mainline: ATT MTU 247, DLE, MTU-exchange handling
db8dc4d default to OTA DFU	picked clean. Mainline's opt-in DEFAULT_TO_OTA_DFU block stays but is off; OTAFIX's finer rule (not when UF2/serial/double-reset asked) is the behaviour Android relies on
02fc9bd usb_teardown after OTA	picked by hand — insurance, mainline never inits USB on the OTA path
7e69127 lazy erase	picked with merge — mainline still erases the whole app region up front, so this is a real OTAFIX feature. pstorage_clear(CODE_PAGE_SIZE) now wrapped in mainline's NRF_ERROR_NO_MEM retry loop; non-OTA branch uses mainline's flash_nrf5x_erase
54a9e85 small-packet accumulator	picked (one additive hunk next to prioritize_flash_writes_over_ble())
1d8194f max BLE TX power	picked clean
18758d9 CURRENT.UF2 real size (#20)	picked clean
f41dab2 nrfx 3.14 / tinyusb bump (#19)	not taken — mainline pins nrfx v2.0.0 + older tinyusb; keeping mainline's pins means the linker/nrf_common.ld alias shim is also unneeded. Re-bumping is a separate PR
adafruit backports (#26)	already in the base

Mainline features we now get for free

Flash/notification queue handling (e1ea1c6, the one #25 called too entangled), DFU-mode exit timeout, opt-in dual-bank (DUALBANK_FW=1), opt-in signed firmware (SIGNED_FW=1, tinycrypt submodule), pending-interrupt clear before app jump, GCC 15 clean.

Before un-drafting

• RAK4631: boot, UF2 flash, BLE OTA via Android app, CURRENT.UF2 dump-and-restore (fix: size CURRENT.UF2 to the real installed app, not the max region #20 regression), serial DFU via in-app upgrade
• One display board (T114) — both screens render
• Remaining 15 boards at least boot + one DFU path each
• AGENTS.md gotchas rewritten for this base (nrfx pin, linker shim, CI contexts no longer true as written)
• Decide with oltaco whether this lands upstream in his repo first (Reconcile with adafruit/Adafruit_nRF52_Bootloader mainline (0.11.0+) #25's coordination point)

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 812b8e11-9f1a-439d-a2af-e00e3e59d7f6

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 3 committers have signed the CLA.

✅ jamesarich
❌ oltaco
❌ hathach
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[jamesarich]

RAK4631 hardware pass on 960b301 (bootloader 0.11.0-101-g960b301, S140 6.1.1, app 2.8.0.abd3348):

• bootloader update via UF2 from OTAFIX 2.2 → boots app, INFO_UF2.TXT shows new version
• app boot + USB CDC - ok, config intact
• CURRENT.UF2 dump byte-identical to the dump taken under the old bootloader; restore completes, app boots (fix: size CURRENT.UF2 to the real installed app, not the max region #20 regression check)
• serial DFU with the official firmware-rak4631-2.8.0.abd3348-ota.zip via adafruit-nrfutil (--singlebank, no touch) - 44s, auto-activates, app boots, config intact
• bootloader rollback to master via UF2 and forward again - both fine

Not yet: BLE OTA via the Android app, display boards, the other 15.

Found one real bug while testing: CURRENT.UF2 on this branch is sized to bank_0_size but still starts at 0x1000, so it's 0x25000 (the SoftDevice) short - dump was 1.2M instead of 1.5M after a serial DFU. The #20 pick landed clean onto mainline's ghostfat.c but mainline computes the window differently. Restoring such a dump only "works" while the tail is still in flash. Needs fixing before un-draft.

Side note for anyone repeating this: the 1200-baud touch is handled by the app, and 2.8.0 dev builds ignore it - use meshtastic --enter-dfu.

[jamesarich]

Correction on the CURRENT.UF2 short-dump: ghostfat.c is identical on master, it's pre-existing. bank_0_size is app-only after serial/OTA DFU but SD+app after a UF2 flash (msc_uf2 counts every block written from 0x1000), and current_flash_size() uses it as a window from USER_FLASH_START either way. Fixing on master separately, will merge through.

[jamesarich]

BLE OTA via the Android app (snapshot 29321961, legacy DFU, PRN 5, packet 244) fails on db7c34b, reproducibly: bootloader enters OTA mode and advertises 4631_DFU, stream starts, PRNs arrive at 1220/2440/3660/4880 (~90ms), then nothing at 6100 and the link drops on supervision timeout. Same offset both runs - right after the first 4K page, i.e. the first on-demand page erase of the lazy-erase path. That's my merge of 7e69127 onto mainline's reworked pstorage_raw.c, not the app. Board recovered via UF2. Looking at it.

[jamesarich]

BLE OTA A/B, same RAK4631, same Android snapshot, same firmware-rak4631-2.8.0.abd3348-ota.zip:

bootloader	attempt 1 (app-jump entry)	stream
db7c34b (this branch, lazy erase)	wedges: LEDs solid, no PRN, link timeout	after the DFU-timeout reset: PRNs to 4880 then dead at 6100
bfe39b6 (lazy erase reverted)	wedges the same way	after the reset: 729,528 B in 47 s, activates, app boots
master 33e095d	START answered <1 s, streams immediately	729,528 B in 73 s, activates, app boots

So two things:

• lazy erase as merged stalls at the first on-demand page erase - reverted in bfe39b6. Mainline's upfront erase instead.
• the DFU_MAGIC_OTA_APPJUM (no-reset, SoftDevice already up) entry wedges on this base and is fine on master - regression, still open. Only self-heals because the DFU inactivity timeout resets the chip ~60-90 s later and the app's retry catches the clean advertisement. On master USB stays enumerated through the whole DFU (no chip reset until activate), so app-jump is the normal path, not an edge case.

Ledger updated: 7e69127 lazy erase → reverted, needs a real re-port onto mainline's pstorage/flash-queue model.

Separately, the app's post-update BLE reconnect to the node timed out on the master run even though the node was up on USB - Android side, not this repo.

[jamesarich]

App-jump wedge, what's been eliminated on the bench (each a separate bootloader flash + in-app OTA on the RAK4631):

• lazy erase - reverted, wedge unchanged
• app's USBD IRQ still firing into our vector table - masked USBD_IRQn on _sd_inited, wedge unchanged
• fault handler - it does NVIC_SystemReset(), and USB never drops while wedged, so this is a spin, not a fault
• main.c SD-lifecycle (_sd_inited / mbr_init_sd / SD_EVT_IRQHandler / proc_sd_task / app_sched) - identical to master

Shape: attempt 1 (SoftDevice inherited from the app) connects, START is sent, then silence - ~5 s with lazy erase, ~21 s without (≈ the upfront erase of 729 KB) - link drops on supervision timeout, device stops advertising, LEDs solid, no reset until the DFU inactivity timeout (~60-90 s). After that reset (SD freshly enabled by the bootloader) the same transfer completes. Master streams on attempt 1.

Still-open candidates, all mainline-only on this path: sd_ble_gap_phy_update (2M) in proc_ble on connect; the new DFU inactivity app_timer (dfu_timer_restart() on every packet, VERIFY_SUCCESS-checked); the hvx do { } while (NRF_ERROR_RESOURCES) spin in ble_dfu.c (present on master too, but master's SD config differs); HVN/WRCMD sd_ble_cfg_set calls that fail INVALID_STATE on an inherited stack. Reading won't settle it - debug output here is RTT-only, so the next step is either an SWD probe (halt + PC while wedged, 10 min) or a UART log backend for CFG_DEBUG (~1 h).

[jamesarich]

Why the wedge clears after ~90 s: it's the application's watchdog, not anything in the bootloader. Meshtastic's nRF52 firmware arms the WDT, a WDT can't be stopped, and app-jump keeps it running. The bootloader feeds it from wait_for_events() - so the fact that it fires proves the bootloader is spinning somewhere outside its event loop. The bootloader's own DFU inactivity timeout is 360 s and never gets reached.

[jamesarich]

Correction to the A/B table: master is not clean on app-jump either. It streams on attempt 1 but hangs for the same ~90 s after ACTIVATE_AND_RESET (kernel log: activate-disconnect → app re-enumerates 90 s later, both master runs), while this branch reboots ~1 s after activate and hangs before START instead. Same WDT-rescued spin, different spot. Filed the master half as #35 - whoever gets a probe on this should look at both ends of the app-jump path at once.

[jamesarich]

Correction: most of the app-jump "wedge" was a USB-connected-testing artifact

Retested battery-only (RAK on battery, USB unplugged, stock app 29321961, this branch's bootloader bfe39b6 — no probe): the BLE OTA completes cleanly. Node advertised 4631_DFU, streamed 729528/729528, "Upload complete", app showed Update Successful!, node rebooted healthy.

So the earlier findings on this path need walking back:

• The sd_ble_gap_addr_get → NRF_ERROR_INTERNAL fault and the ~90 s wedge were caught with USB connected (and partly with the perturbing SysTick probe). Battery-only, addr_get succeeds (the device advertises), and the transfer runs. USB power/ground during radio DFU was perturbing the nRF52.
• The deterministic mid-stream drop at 351360/729528 seen earlier was also USB-connected; battery-only did not reproduce it.

What is real, battery-only: the transfer still stalls ~1-2 min at the start and burns the app's first retry cycle(s) before it catches. That is the upfront full-erase of the app region on START_DFU — the cost of the lazy-erase revert (bfe39b6). Lazy erase existed precisely to hide that wait (master answers START in <1 s). So the open item stands: re-port lazy erase onto mainline's pstorage/flash model, or accept the slow-start (which Meshtastic-Android #6812 now surfaces as "Preparing device: erasing flash…" instead of a frozen 0%).

Consequence for the WIP app-jump fix: the mbr_init_sd-on-app-jump change (local branch fix/app-jump-sd-lifecycle, unpushed) is now of uncertain necessity — the fault it targets may have been USB/probe-induced. Not pushing it; will re-evaluate with an SWD probe (ordered) and battery-only repro.

[jamesarich]

Lazy erase re-ported — validated on hardware ✅

Branch feat/lazy-erase-report (off this branch's bfe39b6), commit 7eb7ee6. Replaces the revert with a working re-port.

Approach: keep mainline's pstorage untouched; do the on-demand erase synchronously in dfu_single_bank.c, not via OTAFIX's async pending-packet ring buffer (which is what stalled on this base). Erase only page 0 before START_DFU (so START responds immediately, no ~20 s upfront-erase stall), then erase each later page the first time a data packet targets it — issue pstorage_clear for that page and block pumping proc_soc until the flash-op event lands. proc_soc drains only SOC/flash events, not BLE, so the data path isn't re-entered while waiting — no completion chain to get wrong.

Change is one file, +61 lines. 17/17 boards build.

Hardware, RAK4631, battery-only (USB perturbs BLE OTA — always test on battery): full 729528-byte image streams past the first-page boundary (where the old lazy erase died at ~4880) to completion — "Update Successful". Same node/app that stalled at 4880 with the async lazy erase, and that took ~20 s + a wasted retry with the erase reverted.

Suggest this supersedes the Revert "lazy erase" commit when the rebase firms up: cherry-pick 7eb7ee6 in place of the revert. Leaving it on its own branch for review rather than force-rewriting this one.

(One orthogonal observation from the same testing: attempt 1 after the buttonless reboot consistently fails to connect (~5 s, "device not found") and the transfer succeeds on a later attempt, across all three bootloader variants — so it's the app/BLE reconnect timing, not the bootloader. Not chased here.)

[jamesarich]

Remaining regression isolated: attempt-1 BLE supervision timeout. Parked pending an SWD probe.

Captured with a clean logcat-to-file trace, battery-only, on feat/lazy-erase-report (7eb7ee6):

09:54:21.452  Connection state → Connected
09:54:21.548  Control Point subscribed
09:54:22.136  Starting upload (init=14B, firmware=729528B)...
09:54:27.419  Connection state → Disconnected(reason=Timeout)   ← supervision timeout, no PRN ever arrives

The first post-reboot DFU connection dies on a BLE supervision timeout ~5 s after START_DFU, on every rebased variant (upfront-erase, old lazy erase, new re-port — so it is not erase-related). The SoftDevice stops servicing the link. The transfer then succeeds on a retry once the device comes back (~60-90 s worst case). Master streams attempt 1 clean battery-only, so this is a real rebase-side regression — the last one standing.

Two fix attempts, both falsified on hardware:

• reuse the inherited SD when sd_softdevice_is_enabled (early-return in ble_stack_init) → device comes up USB-DFU only, no BLE
• SD_MBR_COMMAND_INIT_SD when the SD is not actually enabled (helped on master-with-probe earlier) → on this base the device faults during OTA entry and boots straight back to the app, never advertising

Interrupt-level SD-state debugging is beyond blind iteration — needs the SWD probe (ordered). Parking this PR until it arrives. State of the branch when parked:

• rebase-adafruit-0.11 @ bfe39b6 (lazy erase reverted) — as pushed
• feat/lazy-erase-report @ 7eb7ee6 — validated re-port, supersedes the revert when this un-parks
• everything else hardware-verified battery-only: UF2 update, dump/restore, serial DFU, BLE OTA completes (on retry), 17/17 boards build
• test discipline note for whoever picks this up: BLE OTA must be tested with the device on battery — USB-connected runs produce spurious wedges/faults/mid-stream drops that cost this effort most of a day