Dual-protocol integration testing and V1.009 review fixes

.github/workflows/ci.yml

@@ -59,10 +59,14 @@ jobs:
         run: |
           pip install -e ".[dev]"
 
-      - name: Run Behave tests
+      - name: Run Behave tests (V1.009+ protocol)
         run: |
           behave --no-capture --format progress
 
+      - name: Run Behave tests (legacy protocol)
+        run: |
+          behave --no-capture --format progress -D legacy_api=true
+
   build:
     # builds package on all events (PRs, main, and tags) to catch build errors early
     needs: [unit-tests, integration-tests]

.github/workflows/docker-build.yml

@@ -58,10 +58,14 @@ jobs:
           cache-from: type=gha
           cache-to: type=gha,mode=max
 
-      - name: Run Docker integration tests
+      - name: Run Docker integration tests (V1.009+ protocol)
         run: |
           behave -D implementation=docker -D image_name=${{ env.REGISTRY_IMAGE }}:test --no-capture --format progress
 
+      - name: Run Docker integration tests (legacy protocol)
+        run: |
+          behave -D implementation=docker -D image_name=${{ env.REGISTRY_IMAGE }}:test --no-capture --format progress -D legacy_api=true
+
   build:
     needs: [test]
     runs-on: ubuntu-latest

.github/workflows/sonarcloud.yml

@@ -42,10 +42,14 @@ jobs:
         run: |
           COVERAGE_FILE=.coverage.pytest pytest test/blackvuesync_test.py --cov=. --cov-report= -v
 
-      - name: Run integration tests with coverage
+      - name: Run integration tests with coverage (V1.009+ protocol)
         run: |
           behave --no-capture --format progress -D collect_coverage=true
 
+      - name: Run integration tests with coverage (legacy protocol)
+        run: |
+          behave --no-capture --format progress -D collect_coverage=true -D legacy_api=true
+
       - name: Combine coverage data
         run: |
           coverage combine --keep

CHANGELOG.md

@@ -2,7 +2,7 @@
 
 ## 2.2.0
 
-* Support BlackVue firmware V1.009+ cameras.
+* Support BlackVue firmware V1.009+ cameras. (#87)
 * Replace undocumented `--filter` with `--include` and `--exclude` options for filtering recordings by type and direction. Codes are comma-separated, direction is optional. (#61)
 * Add `--retry-failed-after` option to retry failed downloads after a configurable delay. (#58)
 * Add `--skip-metadata` option to skip downloading metadata files (thumbnails, accelerometer, GPS). (#14)

README.md

@@ -19,7 +19,7 @@ A typical setup would be a periodic cron job or a Docker container running on a
 ## Features
 
 > [!IMPORTANT]
-> As of version 2.2.0a7, supports the new API in BlackVue firmware V1.009+.
+> As of version 2.2.0, supports the new API in BlackVue firmware V1.009+.
 
 * **Portable runtimes:**
   * A [single, self-contained Python script](https://github.com/acolomba/blackvuesync/blob/master/blackvuesync.py) with no third-party dependencies. It can be copied and run anywhere, either [manually](#manual-usage) or [periodically](#unattended-usage).
@@ -130,7 +130,7 @@ Other options:
 * `--max-used-disk`: Downloads stop once the specified used disk percentage threshold is reached. Defaults to `90` (i.e. 90%.)
 * `--timeout`: Sets a timeout for establishing a connection to the dashcam, in seconds. Defaults to `10.0` seconds.
 * `--retry-failed-after`: Sets the minimum elapsed time before retrying a failed download. Accepted units are `s` for seconds, `h` for hours, `d` for days and `w` for weeks. If no unit is indicated, days are assumed. Defaults to `1d`.
-* `--skip-metadata`: Skips downloading metadata file types. Takes a string of characters: `t` for thumbnail (`.thm`), `3` for accelerometer (`.3gf`), `g` for GPS (`.gps`). For example, `--skip-metadata t3g` skips all metadata files, downloading only the `.mp4` video recordings.
+* `--skip-metadata`: Skips downloading metadata file types. Takes a string of characters: `t` for thumbnail (`.thm`), `3` for accelerometer (`.3gf`)[^1], `g` for GPS (`.gps`). For example, `--skip-metadata t3g` skips all metadata files, downloading only the `.mp4` video recordings.
 * `--include`: Downloads only recordings matching the given codes. Each code is a recording type letter optionally followed by a camera direction letter, comma-separated. For example, `--include P,NF` downloads all Parking recordings and Normal Front recordings. See the table below for valid codes.
 * `--exclude`: Excludes recordings matching the given codes, same format as `--include`. Takes priority over `--include`. For example, `--include N,E --exclude NR` downloads all Normal and Event recordings except Normal Rear.
 * `--quiet`: Quiets down output messages, except for unexpected errors. Takes precedence over `--verbose`.
@@ -308,7 +308,7 @@ Other parameters:
 * `TIMEOUT`: If set to a float value, sets the timeout in seconds for connecting to the dashcam. (Default: `10.0` seconds.)
 * `RETRY_FAILED_AFTER`: If set, sets the minimum elapsed time before retrying a failed download. Accepted units are `s` for seconds, `h` for hours, `d` for days and `w` for weeks. If no unit is indicated, days are assumed. (Default: `1d`.)
 * `VERBOSE`: If set to a number greater than zero, increases logging verbosity. (Default: `0`.)
-* `SKIP_METADATA`: If set, skips downloading the indicated metadata file types. Takes a string of characters: `t` for thumbnail (`.thm`), `3` for accelerometer (`.3gf`), `g` for GPS (`.gps`). For example, `t3g` skips all metadata files. (Default: empty.)
+* `SKIP_METADATA`: If set, skips downloading the indicated metadata file types. Takes a string of characters: `t` for thumbnail (`.thm`), `3` for accelerometer (`.3gf`)[^1], `g` for GPS (`.gps`). For example, `t3g` skips all metadata files. (Default: empty.)
 * `INCLUDE`: If set, downloads only recordings matching the given codes. Each code is a recording type letter optionally followed by a camera direction letter, comma-separated. For example, `P,NF` downloads all Parking recordings and Normal Front recordings. (Default: empty, meaning all recordings are downloaded.)
 * `EXCLUDE`: If set, excludes recordings matching the given codes, same format as `INCLUDE`. Takes priority over `INCLUDE`. For example, setting `INCLUDE=N` and `EXCLUDE=NR` downloads all Normal recordings except Normal Rear. (Default: empty.)
 * `QUIET`: If set to any value, quiets down logs: only unexpected errors will be logged. (Default: empty.)
@@ -327,3 +327,7 @@ Other parameters:
 This project is licensed under the MIT License - see the [COPYING](COPYING) file for details
 
 Copyright 2018-2026 [Alessandro Colomba](https://github.com/acolomba)
+
+## Footnotes
+
+[^1]: In firmware V1.009+, accelerometer data is embedded in the video recordings and no separate `.3gf` file exists, so the `3` flag has no effect.

behave.ini

@@ -13,5 +13,8 @@ log_level_http = WARN
 log_level_mock_dashcam = INFO
 # collects code coverage data (direct mode only)
 collect_coverage = false
+# runs the suite against the legacy camera protocol instead of V1.009+;
+# scenarios tagged @legacy only run when this is true
+legacy_api = false
 # execution implementation: direct (subprocess) or docker (container)
 implementation = direct

blackvuesync.py

@@ -25,7 +25,6 @@
 
 import argparse
 import base64
-import binascii
 import contextlib
 import datetime
 import errno
@@ -757,6 +756,21 @@ def to_recording(filename: str, grouping: str) -> Recording | None:
     )
 
 
+def to_dashcam_recordings(
+    dashcam_filenames: list[str], grouping: str
+) -> list[Recording]:
+    """converts filenames listed by the dashcam to recordings; raises for unrecognized filenames"""
+    dashcam_recordings = []
+    for dashcam_filename in dashcam_filenames:
+        if (recording := to_recording(dashcam_filename, grouping)) is None:
+            raise RuntimeError(
+                f"Unexpected filename in the recording list from the dashcam : {dashcam_filename}"
+            )
+        dashcam_recordings.append(recording)
+
+    return dashcam_recordings
+
+
 # pattern of a recording filename as returned in each line from from the dashcam index page
 file_line_re = re.compile(r"n:/Record/(?P<filename>.*\.mp4),s:1000000\r\n")
 
@@ -781,16 +795,23 @@ def _build_request(url: str) -> urllib.request.Request:
 
 
 def is_legacy_camera(base_url: str) -> bool:
-    """returns True when the /accessible probe does not answer 200; pre-V1.009
-    cameras lack the endpoint"""
+    """returns True when the /accessible probe answers 404, since pre-V1.009 cameras lack the
+    endpoint; any other error fails the run like a recording-list failure"""
     url = urllib.parse.urljoin(base_url, "accessible")
     try:
         with urllib.request.urlopen(_build_request(url)) as response:
-            legacy = bool(response.getcode() != 200)
-            probe_result = f"status code : {response.getcode()}"
+            response_status_code = response.getcode()
+            if response_status_code != 200:
+                raise RuntimeError(
+                    f"Error response from : {base_url} ; status code : {response_status_code}"
+                )
+            legacy = False
+            probe_result = f"status code : {response_status_code}"
     except urllib.error.HTTPError as e:
-        # any http error means the camera answered but does not support the endpoint;
-        # connection-level failures propagate to the shared unavailable/disconnected handling
+        # a 404 means the camera answered but lacks the endpoint, i.e. legacy firmware;
+        # other http errors and connection-level failures propagate to the shared handling
+        if e.code != 404:
+            raise
         legacy = True
         probe_result = str(e)
     logger.debug(
@@ -1155,7 +1176,8 @@ def download_metadata_file(
     group_name: str | None,
     metrics: SyncMetrics | None = None,
 ) -> tuple[bool, int | None]:
-    """downloads thumbnail or gps data via the /vodMetadata endpoint (V1.009+)"""
+    """downloads thumbnail or gps data via the /vodMetadata endpoint (V1.009+); returns whether data was
+    transferred (download speed is not measured)"""
     # pylint: disable=too-many-arguments,too-many-positional-arguments,too-many-branches,too-many-return-statements,too-many-locals,too-many-statements
     if group_name:
         ensure_destination(os.path.join(destination, group_name))
@@ -1249,7 +1271,8 @@ def download_metadata_file(
             )
             return False, None
 
-        content = base64.b64decode(encoded, validate=True)
+        # strips whitespace so line-wrapped payloads decode; validation still rejects corrupt base64
+        content = base64.b64decode("".join(encoded.split()), validate=True)
         with open(temp_filepath, "wb") as f:
             f.write(content)
         os.rename(temp_filepath, destination_filepath)
@@ -1297,9 +1320,9 @@ def download_metadata_file(
         raise UserWarning(
             f"Timeout communicating with dashcam at address : {base_url}; error : {e}"
         ) from e
-    except (json.JSONDecodeError, binascii.Error) as e:
-        # malformed JSON or base64; the camera answered, so marks failed like
-        # the other unusable responses
+    except ValueError as e:
+        # malformed json, encoding or base64; the camera answered, so marks
+        # failed like the other unusable responses
         _handle_download_failure(
             metadata_filename,
             e,
@@ -1319,8 +1342,9 @@ def download_recording(
     metrics: SyncMetrics | None = None,
     legacy: bool = False,
 ) -> None:
-    """downloads the set of recordings, including gps data, for the given filename from the dashcam to the destination
-    directory"""
+    """downloads the video and metadata files for the given recording from the dashcam to the destination directory;
+    on V1.009+ cameras, retrieves thumbnail and gps data via /vodMetadata and skips the unavailable accelerometer
+    data"""
     # pylint: disable=too-many-branches,too-many-locals
     # first checks that we have enough room left
     disk_usage = shutil.disk_usage(destination)
@@ -1723,9 +1747,7 @@ def sync(  # pylint: disable=too-many-arguments,too-many-positional-arguments
 
     base_url = f"http://{address}"
     legacy, dashcam_filenames = get_dashcam_filenames(base_url)
-    dashcam_recordings = [
-        r for x in dashcam_filenames if (r := to_recording(x, grouping)) is not None
-    ]
+    dashcam_recordings = to_dashcam_recordings(dashcam_filenames, grouping)
     if metrics:
         metrics.dashcam_recordings_seen = len(dashcam_recordings)