Download logs over MAVLink FTP, add ArduPilot support

CMakeLists.txt

@@ -33,6 +33,8 @@ include_directories(${SQLite3_INCLUDE_DIRS})
 add_executable(${PROJECT_NAME}
     src/main.cpp
     src/ServerInterface.cpp
+    src/FtpLogFetcher.cpp
+    src/LogEntryLister.cpp
     src/LogLoader.cpp)
 
 target_link_libraries(${PROJECT_NAME}

README.md

@@ -4,9 +4,48 @@ Downloads PX4 log files (.ulg) and uploads them to a local server and optionally
 
 The **config.toml** file is used to configure the program settings.
 
+Works with PX4 (`.ulg`) and ArduPilot (`.BIN`).
+
 ### Behavior
 Downloading and uploading will only occur while the vehicle is not armed. Downloading and uploading operations are performed in separate threads. An sqlite database per server is used to track log file download/upload status.
 
+### Log transport
+The bulk transfer uses **MAVLink FTP** (`FILE_TRANSFER_PROTOCOL`, msg 110), not the classic log protocol (`LOG_DATA`, msg 120).
+
+`LOG_DATA` has no `target_system`/`target_component` fields, so a router in between — mavlink-router, for instance — has no addressing information to work with and copies every chunk to every endpoint it serves, telemetry radio included. `FILE_TRANSFER_PROTOCOL` is addressed, and both PX4 and ArduPilot send the reply back to the requesting sysid/compid, so the transfer is unicast to logloader and the other endpoints stay quiet. No router or autopilot configuration change is needed.
+
+The log *list* still comes from `LOG_ENTRY` (msg 118), which is also untargeted, but it is a handful of bytes per log rather than megabytes, and it is the only source of the timestamp the databases are keyed on.
+
+At startup logloader probes for the vehicle's log directory (`/fs/microsd/log` on PX4, `/APM/LOGS` on ArduPilot) and uses the result to pick the log file extension. Each downloaded file is size-checked against what `LOG_ENTRY` reported before it is moved into the logs directory; anything that does not line up is discarded and re-fetched over `LOG_DATA` instead.
+
+Requirements:
+- The autopilot's MAVLink instance must have FTP enabled (`mavlink start -x` on PX4).
+- If FTP is unavailable logloader falls back to `LOG_DATA` automatically. On ArduPilot without FTP, set `log_extension = ".BIN"` so downloaded files are named correctly.
+
+### ArduPilot notes
+ArduPilot needs FTP: MAVSDK's `LogFiles` plugin cannot enumerate its logs. The plugin assumes PX4's zero-based log ids — it discards any `LOG_ENTRY` whose id is not below `num_logs` and reads the collected entries back out at indices `0..num_logs-1` — while ArduPilot numbers list entries from one, so its final entry always trips that check and `get_entries()` returns `NoLogfiles`. When the FTP probe reports ArduPilot, logloader runs the `LOG_REQUEST_LIST` exchange itself (`LogEntryLister`) without assuming where the numbering starts.
+
+Mapping a list entry onto a file is also stack-specific. ArduPilot reports a *list entry number*, not the log number in the file name, and log numbers wrap at `LOG_MAX_FILES`. logloader reads `LASTLOG.TXT` over FTP and reproduces ArduPilot's own oldest-first ordering to resolve the two.
+
+`.BIN` files are not accepted by review.px4.io, so leave `upload_enabled = false` or point `remote_server` somewhere that understands dataflash logs.
+
+Set `download_protocol` in **config.toml** to `"mavlink"` to restore the old behavior, or `"ftp"` to never fall back.
+
+### Configuration
+| Key | Default | Description |
+| --- | --- | --- |
+| `connection_url` | `udp://:14551` | MAVSDK connection string |
+| `local_server` | `http://127.0.0.1:5006` | Local upload target |
+| `remote_server` | `https://review.px4.io` | Remote upload target |
+| `email` | `""` | Email attached to remote uploads |
+| `remote_api_key` | `""` | Per-account API key for authenticated Flight Review (`Authorization: Bearer` + `X-API-Key`). Empty = upload without API key headers (open servers). Generate under /account |
+| `upload_enabled` | `false` | Upload to the remote server |
+| `public_logs` | `false` | Mark remote uploads public |
+| `download_protocol` | `"auto"` | `"auto"`, `"ftp"` or `"mavlink"` |
+| `remote_log_directory` | `""` | Override the vehicle log directory |
+| `log_extension` | `""` | Override the downloaded file extension |
+| `ftp_use_burst` | `true` | Use FTP burst reads |
+
 ### Build
 Install dependencies
 ```

config.toml

@@ -4,3 +4,31 @@ remote_server = "https://review.px4.io"
 email = ""
 upload_enabled = false
 public_logs = false
+
+# Per-account API key for authenticated Flight Review (ARK).
+# Generate at https://review.arkelectron.com/account (shown once; format fr_...).
+# When set: sent as Authorization: Bearer and X-API-Key on POST /upload.
+# When empty: remote uploads are still attempted without API key headers
+# (works for open servers; ARK Flight Review will reject with 403).
+remote_api_key = ""
+
+# How to pull the log data off the vehicle.
+#   "auto"    - MAVLink FTP when the vehicle answers, LOG_DATA otherwise (default)
+#   "ftp"     - MAVLink FTP only, never fall back to LOG_DATA
+#   "mavlink" - the classic LOG_DATA protocol only
+#
+# FTP is preferred because FILE_TRANSFER_PROTOCOL is addressed to this system and
+# component, so mavlink-router sends the transfer to this endpoint only. LOG_DATA has no
+# target fields, which leaves a router no choice but to copy every chunk to every
+# endpoint it serves, telemetry radio included.
+download_protocol = "auto"
+
+# Remote log directory. Empty probes "/fs/microsd/log" (PX4) then "/APM/LOGS" (ArduPilot).
+remote_log_directory = ""
+
+# Extension for downloaded logs. Empty derives it from the flight stack, which needs FTP.
+# Set it to ".BIN" for ArduPilot when FTP is unavailable.
+log_extension = ""
+
+# FTP burst reads. Much faster, disable it if the flight stack's burst support misbehaves.
+ftp_use_burst = true

docs/hardware-test-pr28.md

@@ -0,0 +1,71 @@
+## Hardware test results (Pi + PX4 / ARK Pi6X)
+
+Tested branch `claude/logdata-mavlink-targeting-uj8d0y` including `7a23c8e` (`fix(upload): stop spamming when flight-review is offline`) on a live companion:
+
+| Item | Value |
+|------|--------|
+| Host | Raspberry Pi (ark-os), aarch64 |
+| Autopilot | PX4 via `mavlink-router` UART @ 2 Mbaud |
+| Link | `udp://:14551` (logloader endpoint) |
+| Binary | Built from this branch with MAVSDK 3.17.1, `DEBUG_BUILD=ON` |
+
+### Connect / index
+
+```text
+Connected.
+MAVLink FTP available, PX4 logs in /fs/microsd/log (68 files)
+Received 68 log entries in ~0.26s
+```
+
+After FC reboot this was reliable. Earlier mid-session hard-kills of logloader left the FC with `FTP list … File Does Not Exist` / `0 log entries` until reboot.
+
+### FTP download (success)
+
+| Log | Protocol | Result |
+|-----|----------|--------|
+| `LOG0058` (~40 MB) | MAVLink FTP (burst) | **Finished in 94s** — on-disk file is valid ULog (`ULog\x01…`) |
+| Path | `/fs/microsd/log/2025-02-05/20_39_10.ulg` | Mapped and transferred |
+
+### Path mapping miss → auto fallback (success)
+
+For a small entry (`LOG0020`, LOG_ENTRY size **259471**):
+
+```text
+Downloading …/LOG0020_….ulg from /fs/microsd/log/2024-06-18/16_06_57.ulg
+Size mismatch …: got 276495 bytes, expected 259471
+FTP download failed
+Downloading …/LOG0020_….ulg          # LOG_DATA fallback (download_protocol=auto)
+Finished in 0.55 seconds
+```
+
+Final file size **259471**, valid ULog magic. So **auto** correctly falls back when FTP path resolution picks the wrong remote file.
+
+### Large-log FTP timeouts
+
+Newest/largest queue head (~100 MB, `LOG0067`) often hits:
+
+```text
+FTP download of …/00_02_04.ulg failed: Timeout
+```
+
+Burst mode makes progress (multi‑Mbps) but MAVSDK can still time out on very large files in short runs. Non-burst was ~800 kbps. Worth a follow-up (timeouts / resume / queue order), not a blocker for basic FTP path.
+
+### Upload spam fix (`7a23c8e`)
+
+With **flight-review not running** (`:5006` down) and multiple pending local uploads:
+
+| Metric | Count |
+|--------|------:|
+| `Upload server … unreachable; skipping uploads for 60s` | **1** |
+| `TEMPORARILY FAILED` (old spam) | **0** |
+| Per-log `Connection to … failed` | **0** |
+
+### Not tested here
+
+- ArduPilot / `.BIN` / `LogEntryLister` path  
+- Live upload to a running local flight-review (only “server down” path)  
+- Full multi-hour campaign of all 68 logs over FTP only  
+
+### Verdict
+
+**Works on PX4 hardware after a healthy FC boot:** connect, FTP detect/index, successful multi‑MB FTP download, auto→LOG_DATA fallback on size mismatch, and no upload spam when flight-review is offline.