From 397d3b3afdfad6505387bb74223e18c7d35c74eb Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 26 Jul 2026 03:30:10 +0000 Subject: [PATCH 1/4] Download logs over MAVLink FTP, add ArduPilot support LOG_DATA (msg 120) carries no target_system/target_component, so a router between logloader and the autopilot has no addressing information and copies every chunk to every endpoint it serves -- the telemetry radio included. On a node running mavlink-router a log download saturates links that have no interest in it. FILE_TRANSFER_PROTOCOL (msg 110) is addressed, and both PX4 and ArduPilot send the reply back to the requesting sysid/compid, so the bulk transfer is unicast to logloader. Switch the download path to MAVLink FTP. Nothing changes in mavlink-router or in the autopilot. The log list still comes from LOG_ENTRY: it is a handful of bytes per log rather than megabytes, and it is the only source of the timestamp the databases are keyed on, so existing uuids and sqlite state carry over untouched. FtpLogFetcher probes /fs/microsd/log then /APM/LOGS to find the log directory and the flight stack, indexes it, and maps each LOG_ENTRY onto a remote path. An exact size match settles it when MAVSDK reports sizes; otherwise PX4 entries resolve through the mtime LOG_ENTRY reports and ArduPilot entries through the list ordering. Every download is size-checked before it is moved out of staging and anything that does not line up falls back to LOG_DATA. ArduPilot needed more than a new transport. MAVSDK's LogFiles plugin assumes PX4's zero-based log ids -- it discards any LOG_ENTRY whose id is not below num_logs -- while ArduPilot numbers list entries from one, so get_entries() always returns NoLogfiles. LogEntryLister runs the same exchange over MavlinkPassthrough without that assumption. Resolving a list entry to a file then means reproducing ArduPilot's oldest-first ordering, which wraps at LOG_MAX_FILES, so LASTLOG.TXT is read over FTP to find the wrap point. Downloaded files pick up the .BIN extension. download_protocol in config.toml selects "auto" (FTP with LOG_DATA fallback, the default), "ftp" or "mavlink". Also stop the download loop from spinning on a log it cannot fetch: it retried immediately and only the slowness of a LOG_DATA timeout kept it from becoming a hot loop. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AsZc1HUPVUwpMCU8QxNR8p --- CMakeLists.txt | 2 + README.md | 38 +++ config.toml | 21 ++ src/FtpLogFetcher.cpp | 565 ++++++++++++++++++++++++++++++++++++++++ src/FtpLogFetcher.hpp | 101 +++++++ src/LogEntryLister.cpp | 181 +++++++++++++ src/LogEntryLister.hpp | 56 ++++ src/LogLoader.cpp | 187 +++++++++++-- src/LogLoader.hpp | 17 +- src/ServerInterface.cpp | 31 ++- src/ServerInterface.hpp | 9 + src/main.cpp | 6 +- 12 files changed, 1194 insertions(+), 20 deletions(-) create mode 100644 src/FtpLogFetcher.cpp create mode 100644 src/FtpLogFetcher.hpp create mode 100644 src/LogEntryLister.cpp create mode 100644 src/LogEntryLister.hpp diff --git a/CMakeLists.txt b/CMakeLists.txt index 6529639..cee8444 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -33,6 +33,8 @@ include_directories(${SQLite3_INCLUDE_DIRS}) add_executable(${PROJECT_NAME} src/main.cpp src/ServerInterface.cpp + src/FtpLogFetcher.cpp + src/LogEntryLister.cpp src/LogLoader.cpp) target_link_libraries(${PROJECT_NAME} diff --git a/README.md b/README.md index 8993abb..4c4ba66 100644 --- a/README.md +++ b/README.md @@ -4,9 +4,47 @@ Downloads PX4 log files (.ulg) and uploads them to a local server and optionally The **config.toml** file is used to configure the program settings. +Works with PX4 (`.ulg`) and ArduPilot (`.BIN`). + ### Behavior Downloading and uploading will only occur while the vehicle is not armed. Downloading and uploading operations are performed in separate threads. An sqlite database per server is used to track log file download/upload status. +### Log transport +The bulk transfer uses **MAVLink FTP** (`FILE_TRANSFER_PROTOCOL`, msg 110), not the classic log protocol (`LOG_DATA`, msg 120). + +`LOG_DATA` has no `target_system`/`target_component` fields, so a router in between — mavlink-router, for instance — has no addressing information to work with and copies every chunk to every endpoint it serves, telemetry radio included. `FILE_TRANSFER_PROTOCOL` is addressed, and both PX4 and ArduPilot send the reply back to the requesting sysid/compid, so the transfer is unicast to logloader and the other endpoints stay quiet. No router or autopilot configuration change is needed. + +The log *list* still comes from `LOG_ENTRY` (msg 118), which is also untargeted, but it is a handful of bytes per log rather than megabytes, and it is the only source of the timestamp the databases are keyed on. + +At startup logloader probes for the vehicle's log directory (`/fs/microsd/log` on PX4, `/APM/LOGS` on ArduPilot) and uses the result to pick the log file extension. Each downloaded file is size-checked against what `LOG_ENTRY` reported before it is moved into the logs directory; anything that does not line up is discarded and re-fetched over `LOG_DATA` instead. + +Requirements: +- The autopilot's MAVLink instance must have FTP enabled (`mavlink start -x` on PX4). +- If FTP is unavailable logloader falls back to `LOG_DATA` automatically. On ArduPilot without FTP, set `log_extension = ".BIN"` so downloaded files are named correctly. + +### ArduPilot notes +ArduPilot needs FTP: MAVSDK's `LogFiles` plugin cannot enumerate its logs. The plugin assumes PX4's zero-based log ids — it discards any `LOG_ENTRY` whose id is not below `num_logs` and reads the collected entries back out at indices `0..num_logs-1` — while ArduPilot numbers list entries from one, so its final entry always trips that check and `get_entries()` returns `NoLogfiles`. When the FTP probe reports ArduPilot, logloader runs the `LOG_REQUEST_LIST` exchange itself (`LogEntryLister`) without assuming where the numbering starts. + +Mapping a list entry onto a file is also stack-specific. ArduPilot reports a *list entry number*, not the log number in the file name, and log numbers wrap at `LOG_MAX_FILES`. logloader reads `LASTLOG.TXT` over FTP and reproduces ArduPilot's own oldest-first ordering to resolve the two. + +`.BIN` files are not accepted by review.px4.io, so leave `upload_enabled = false` or point `remote_server` somewhere that understands dataflash logs. + +Set `download_protocol` in **config.toml** to `"mavlink"` to restore the old behavior, or `"ftp"` to never fall back. + +### Configuration +| Key | Default | Description | +| --- | --- | --- | +| `connection_url` | `udp://:14551` | MAVSDK connection string | +| `local_server` | `http://127.0.0.1:5006` | Local upload target | +| `remote_server` | `https://review.px4.io` | Remote upload target | +| `email` | `""` | Email attached to remote uploads | +| `upload_enabled` | `false` | Upload to the remote server | +| `public_logs` | `false` | Mark remote uploads public | +| `download_protocol` | `"auto"` | `"auto"`, `"ftp"` or `"mavlink"` | +| `remote_log_directory` | `""` | Override the vehicle log directory | +| `log_extension` | `""` | Override the downloaded file extension | +| `ftp_use_burst` | `true` | Use FTP burst reads | + ### Build Install dependencies ``` diff --git a/config.toml b/config.toml index ff208c2..b73af5d 100644 --- a/config.toml +++ b/config.toml @@ -4,3 +4,24 @@ remote_server = "https://review.px4.io" email = "" upload_enabled = false public_logs = false + +# How to pull the log data off the vehicle. +# "auto" - MAVLink FTP when the vehicle answers, LOG_DATA otherwise (default) +# "ftp" - MAVLink FTP only, never fall back to LOG_DATA +# "mavlink" - the classic LOG_DATA protocol only +# +# FTP is preferred because FILE_TRANSFER_PROTOCOL is addressed to this system and +# component, so mavlink-router sends the transfer to this endpoint only. LOG_DATA has no +# target fields, which leaves a router no choice but to copy every chunk to every +# endpoint it serves, telemetry radio included. +download_protocol = "auto" + +# Remote log directory. Empty probes "/fs/microsd/log" (PX4) then "/APM/LOGS" (ArduPilot). +remote_log_directory = "" + +# Extension for downloaded logs. Empty derives it from the flight stack, which needs FTP. +# Set it to ".BIN" for ArduPilot when FTP is unavailable. +log_extension = "" + +# FTP burst reads. Much faster, disable it if the flight stack's burst support misbehaves. +ftp_use_burst = true diff --git a/src/FtpLogFetcher.cpp b/src/FtpLogFetcher.cpp new file mode 100644 index 0000000..9f9baf5 --- /dev/null +++ b/src/FtpLogFetcher.cpp @@ -0,0 +1,565 @@ +#include "FtpLogFetcher.hpp" +#include "Log.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace fs = std::filesystem; + +namespace +{ + +struct Candidate { + const char* directory; + FtpLogFetcher::Flavor flavor; + const char* extension; +}; + +// PX4 keeps ulog files under PX4_STORAGEDIR "/log", ArduPilot keeps dataflash logs under +// HAL_BOARD_STORAGE_DIRECTORY "/LOGS". Both are reachable over MAVLink FTP. +constexpr Candidate kCandidates[] = { + {"/fs/microsd/log", FtpLogFetcher::Flavor::PX4, ".ulg"}, + {"/APM/LOGS", FtpLogFetcher::Flavor::ArduPilot, ".BIN"}, +}; + +// PX4 nests logs one level deep (log//