Skip to content

DEVELOPMENT.md

Latest commit

 

History

History
183 lines (127 loc) · 17.7 KB

DEVELOPMENT.md

File metadata and controls

183 lines (127 loc) · 17.7 KB
Raw

BitMatch Development Guide

The working to-do list is docs/TODO.md.

Read docs/THESIS.md for what BitMatch promises and the current plan, and ARCHITECTURE.md for how the code is laid out. AGENTS.md has the platform rules every change follows. This guide covers building, testing, debugging and releasing.

Building

  • Release validation uses local Xcode 27.1 (27A9275), Swift 6.4. Record the exact candidate and results in docs/validation/RELEASE_<version>.md. Open BitMatch.xcodeproj.
  • GitHub Actions CI is intentionally disabled; validation runs locally. The older Xcode 26.3 jobs failed before tests, and an abandoned Xcode 26.6 trial did not establish compatibility. Neither changes the app's macOS 15.5 minimum.
  • Schemes:
    • BitMatch: the macOS app (macOS 15.5 or newer).
    • BitMatch-iPad: the iPhone and iPad app (iOS/iPadOS 18.5 or newer). Despite the name it targets both device families.
  • Both apps compile everything under Shared/ and depend on the local package Packages/BitMatchEngine. Everything builds in Swift 6 language mode; keep it warning-free. bash test.sh engine-test runs the engine's own tests in seconds.
  • For an iPhone or iPad device build, set your development team in Signing & Capabilities.

Where changes go

  • The engine and its rules live in Packages/BitMatchEngine: copy, verify, safety, readiness (TransferReadiness), what may be chosen (DestinationSelectionPolicy, BackupTargetPolicy), the verdict (ResultOutcome, TransferCompletion), evidence and the journal. The one state owner, SharedAppCoordinator, and the presentation models live in Shared/Core/.
  • Screens live in Shared/Views/. Each draws a pure presentation value (*Presentation in Shared/Core/Models/) and decides nothing itself. Put a new rule in the presentation or model, with a test, not in a view.
  • Platform code stays in BitMatch/ (Mac) or BitMatch-iPad/ and Platforms/iOS/ (iPhone and iPad), and should be limited to what only that platform can do: pickers, drag and drop, panels and share sheets, Disk Arbitration and volume monitoring, SFTP (Mac only). Shared screens take these as slots or closures (SetupLocationsPlatform, MasterReportPlatform, the CoordinatorSetupScreen slots).
  • Files in Shared/ must compile on both platforms. Wrap AppKit or UIKit use in #if os(...).

Transfer safety boundary

Safety checks belong in the shared engine, not only in views. Readiness on Setup explains a problem early, but TransferPipeline and SafetyValidator must still refuse an unsafe source or backup when called directly, and BackupTargetPolicy must guard every path that adds a backup.

Required behavior (see Packages/BitMatchEngine/Sources/BitMatchEngine/File/SafetyValidator.swift, DestinationWriter.swift, BackupTargetPolicy.swift, and Packages/BitMatchEngine/Sources/BitMatchEngine/TransferPipeline.swift):

  • never write to the source; never overwrite an existing destination file
  • never replace an existing destination name; the hard-link path publishes a synced temporary file, while the exFAT/FAT path durably marks the attempt incomplete before exclusively creating the final name
  • keep interrupted or failed output incomplete; only independent readback and the final source, destination and root checks can advance it to verified status
  • refuse output roots that collide with the source, each other, symlinks, files or nested folders
  • include hidden files and empty folders; apply only the recorded AppleDouble exclusion policy and skip symlink entries
  • use uncached checksums for live verification and evidence
  • write result status only through ResultOutcome; a Quick copy is never "verified"

Execution path

Platform views present the selection and readiness. Start (the button and ⌘R) calls SharedAppCoordinator.startCurrentMode(), which builds a CopyVerifyConfig for CopyVerifyExecutor. The executor owns timing, error tracking, result coalescing, the sleep assertion, ASC MHL and report handoff, and the verdict. It calls TransferPipeline, which runs the real preflight and transfer. Do not add a platform-only transfer path, and do not rely on UI validation as the safety boundary.

Live progress and per-file rows reach views through LiveProgressFeed and LiveResultsFeed, not through the coordinator's objectWillChange. A view that draws live progress or rows observes the feed directly; do not republish per-tick values on SharedAppCoordinator.

Testing

Run the named jobs from the repository root:

bash test.sh engine-test     # shared engine tests
bash test.sh mac-test        # BitMatchTests on macOS
bash test.sh mac-build       # macOS Debug build
bash test.sh ipad-build      # iOS simulator Debug build (the shared-code gate)
IOS_SIMULATOR_DESTINATION='platform=iOS Simulator,name=iPad (A16)' bash test.sh ipad-test
bash test.sh release-builds  # macOS and iOS simulator Release builds

ipad-test refuses to guess a simulator; set IOS_SIMULATOR_DESTINATION to one installed on your machine. Set DERIVED_DATA_ROOT to choose a build folder; the default temporary location is described below.

Ordinary mac-test, ipad-test and engine-test jobs clear inherited BITMATCH_* and TEST_RUNNER_BITMATCH_* probe controls, so an old shell setting cannot start an opt-in hardware or stress workload. They retain workflow snapshot capture and the PDF, preview, matrix and snapshot output directories. Use the dedicated probe runners or their documented direct commands for deliberate stress runs; the ordinary jobs do not count those skipped probes as validation.

The Mac and iOS test jobs request serial execution with -parallel-testing-enabled NO; the engine uses --no-parallel. This reduces contention between independent tests that share the main actor or use real temporary files. Tests still exercise their own concurrent copy, verification and cancellation work. Keep assertion deadlines unchanged, and record failed runs separately from passing reruns.

App build/test jobs use a stable per-checkout build folder under /private/tmp/bitmatch-local-tests-<uid>/ by default. This keeps the Mac test host and Sparkle outside Desktop, where an Xcode 27.0 loader stall was observed. DERIVED_DATA_ROOT=/your/chosen/folder bash test.sh mac-test still overrides the location; SwiftPM engine jobs keep their existing package build folder. Retain logs and test results before removing a closed build cache. This changes no signing, sandbox or privacy settings, and actual external-drive access can still require authorization. A loader stall is not a test pass; retain it separately from a successful rerun.

GitHub Actions CI was disabled on October 20, 2026. The workflow file is retained, but automatic jobs are not part of the project’s validation or release process. Run the required checks locally before submitting changes; record any host or toolchain failures separately from passing tests. Do not repair or re-enable GitHub CI unless Mike asks.

The offline Python regression suite runs with python3 -m unittest discover -s Scripts -p 'test_*.py'. Choose an interpreter with descriptor-relative filesystem APIs for the cleanup tests; the local Python 3.12 run passes, while a Python 3.9 build here lacks dir_fd support. Keep that guard rather than weakening ownership-checked cleanup to accommodate an unsupported interpreter.

Change workflow

  1. Read the shared path before changing copy, verification, preflight, state, readiness or report behavior.
  2. Add or update a focused test in BitMatchTests (or BitMatch-iPadTests for iOS-only behavior). Tests that guard a rule carry a Plant: comment: a one-line production change that must make them fail. Try it, see the test fail, revert.
  3. Run bash test.sh mac-test.
  4. Run bash test.sh ipad-build after every change to shared code or a presentation model.
  5. For UI changes, check an iPhone width (under 600 pt), an iPad split view (600–959 pt) and a Mac window from its 580 pt minimum to wide. Say which checks ran in a simulator and which on a device.
  6. Run bash test.sh release-builds before release work.
  7. Automated tests are not proof of physical-media safety. Use the scripts below and docs/HARDWARE_TESTING.md, with throwaway data, when a change can affect transfer reliability.

Useful test helpers

  • BitMatchTests/TestHelpers/FakeFileSystemService.swift: the one fake FileSystemService.
  • DisposableTransferFixture and TestFixtures: real temporary folders.
  • CameraCardLayouts: folder trees for camera cards.
  • SharedCoordinatorTestSupport: builds a SharedAppCoordinator for tests.
  • WorkflowSnapshotTests (both test targets) render the real screens with seeded data. They are skipped unless BITMATCH_CAPTURE_WORKFLOW_SNAPSHOTS=1 is set.

Coverage

xcodebuild test -scheme BitMatch -enableCodeCoverage YES -resultBundlePath coverage.xcresult
xcrun xccov view --report coverage.xcresult
xcrun xccov view --report --json coverage.xcresult

Reliability diagnostics and hardware testing

  • Scripts/run_apfs_fault_tests.sh creates and removes a marked, disposable APFS image, then checks the case in which one destination becomes inaccessible while another succeeds.
  • Scripts/run_soak_tests.sh runs repeatable transfer stress. Set BITMATCH_SOAK_SEED and BITMATCH_SOAK_ITERATIONS to reproduce a failure; the defaults are 20260711 and 25. It prints validated JSON before removing its marked temporary folder.
  • Scripts/ascmhl/validate_reference.sh checks generated ASC MHL against the official reference tooling.
  • python Scripts/ascmhl/validate_corpus.py /path/to/pinned/ascmitc/mhl checks the permanent external-tool corpus on Linux or macOS without Swift. Use Python 3.11+ with the pinned reference checkout installed; see the fixture README. This does not run the Swift reader or validate newly generated BitMatch output.
  • The large-file app-host memory probe has an explicit marked-fixture wrapper. Run its 64 MiB smoke workflow before the separate 119.6 GB workload; a smoke pass does not validate the larger run.
  • For cable removal, sleep, low-power hubs, exFAT and other physical-media cases, follow docs/HARDWARE_TESTING.md and record results in docs/HARDWARE_COMPATIBILITY.md. The automated APFS and exFAT disk-image tests do not prove that real hardware faults are safe.

Debug tools (Mac)

Debug builds of the Mac app have a Developer menu (BitMatch/App/BitMatchApp.swift, #if DEBUG):

  • Open Interface Lab (⌥⌘L): launches a second app instance with --interface-lab, a no-I/O visual harness with its own sample screens (BitMatch/Views/InterfaceLab/).
  • Enable/Disable Dev Mode (⌥⌘D). The items below need Dev Mode on.
  • Fill Test Data (⌥⌘T): fills in a sample source, backups and detected camera.
  • Stress Test (Small / Medium / Large): runs a real copy between generated folders in the temp directory.
  • Verbose Dev Logs.
  • Clear All Data: resets the last operation's results and state.

DevModeManager (BitMatch/Core/Services/DevModeManager.swift) implements them. None of this exists in Release builds. iPhone and iPad have no debug menu.

Debugging notes

  • Folder picking on iOS does nothing. IOSFileSystemService retains its document-picker delegate (currentDelegate) until the picker finishes; check that it is still retained and that the picker is presented from a valid view controller. Look for "SWIFT TASK CONTINUATION MISUSE" in the console.
  • File access denied on iOS. Every scan, size check and copy must hold security-scoped access, released with defer. See IOSFileSystemService and SharedAppCoordinator.executeOperation.
  • A backup is refused on a device but not in the simulator. Simulator paths sit under the Mac's /Users; device Files locations sit under /private/var/mobile. SafetyValidator.isProtectedSystemPath treats that as user storage on iOS only (BitMatch-iPadTests/IOSStoragePathTests.swift).
  • A view does not update. State that drives UI should be @Published on SharedAppCoordinator or a model it forwards, except live progress and rows, which views observe on LiveProgressFeed, LiveResultsFeed or ProgressPresentationModel directly.
  • Concurrency. Do heavy file-system enumeration off the main actor and marshal results back. Do not iterate NSDirectoryEnumerator with for in inside async code; use while let url = enumerator.nextObject() as? URL. Do not call actor-isolated methods from nonisolated initializers.
  • Logging. Use SharedLogger. AppLogger (Mac) forwards to it.

macOS release

Prerequisites: a Developer ID Application certificate for the release team, a notarytool profile named bitmatch-notary (override with NOTARY_PROFILE), and a Sparkle EdDSA key in the login Keychain.

Release in this order:

  1. Raise MARKETING_VERSION and CURRENT_PROJECT_VERSION in the project, and add the version to CHANGELOG.md. Sparkle decides "newer" by the build number, so an unchanged build number means nobody gets the update.
  2. Scripts/release_mac.sh <version>: signed, notarized DMG in dist/. It re-signs Sparkle's helpers, which notarization requires.
  3. Smoke-test the signed app in dist/BitMatch-<version>.xcarchive on real or sample drives.
  4. Tag v<version> and publish the GitHub release as Latest with the DMG and its .sha256.
  5. Scripts/bump_cask.sh <version> for Homebrew.
  6. Scripts/publish_appcast.sh <version>, so installed copies see the update.
xcrun notarytool store-credentials bitmatch-notary \
  --apple-id "APPLE_ID_EMAIL" \
  --team-id "AUJW7AGG26" \
  --password "APP_SPECIFIC_PASSWORD"

Scripts/release_mac.sh 0.2.11

The script builds, signs, notarizes, staples and checksums dist/BitMatch-<version>.dmg with a matching .sha256. SKIP_NOTARIZE=1 Scripts/release_mac.sh <version> checks signing and DMG creation without submitting to Apple. iPhone and iPad are build-from-source for now.

Where the Sparkle signing key lives

Every update must be signed with the same private key, or installed copies refuse it. It lives in exactly two places, and neither is this repo:

  1. The release Mac's login Keychain: a generic password with account bitmatch (service https://sparkle-project.org). Scripts/publish_appcast.sh signs with it from there; nothing needs to be found by hand.
  2. The backup: an entry named "BitMatch Sparkle private key" in Apple's Passwords app. It syncs end-to-end encrypted to the maintainer's other Apple devices through iCloud Keychain.

The matching public key is SUPublicEDKey in Config/BitMatch-Info.plist. Never change it; doing so cuts off updates to every existing install.

To restore on a new Mac, copy the backup into a temporary file, run generate_keys --account bitmatch -f <file> from Sparkle's resolved package artifacts, then delete the file. To make a fresh backup export, run generate_keys --account bitmatch -x <file>. BitMatch uses its own Keychain account, bitmatch, separate from any other app's Sparkle key. Never commit an exported key.

Before tagging, run the signed app from dist/BitMatch-<version>.xcarchive against real or disk-image volumes: allow drive access, copy a card through verification, ASC MHL and reports, inspect its saved result, and relaunch. Check cancellation and any changed controls. A two-backup safe-to-erase walkthrough needs two independently identified destinations; two folders on one SSD do not qualify. Record unavailable hardware, skipped steps and earlier-revision results explicitly. Debug builds and tests do not exercise the sandboxed release paths.

After publishing the GitHub release, run Scripts/bump_cask.sh <version> for Homebrew, then publish the signed appcast entry last:

Scripts/publish_appcast.sh 0.2.11

The script reads the archived app's build, short version and minimum macOS version, signs the matching DMG with Sparkle's sign_update and the bitmatch key in the login Keychain, updates appcast.xml in BitmatchApp/bitmatchapp.github.io, then commits and pushes the newest entry. Missing or incompatible minimum-version metadata stops the script before signing or network access. It uses the resolved Sparkle package's tool when available and downloads the matching Sparkle release tools otherwise. It never exports the private key.

Run python3 Scripts/test_appcast_minimum.py for offline metadata and script checks. These use disposable fixtures and stub every signing, GitHub, Git and Xcode command; they do not publish an update or change the app's deployment target.

Scripts/bump_cask.sh <version> checks the published DMG against the local checksum, updates Casks/bitmatch.rb in BitmatchApp/homebrew-tap, pushes it, and runs brew audit. brew upgrade bitmatch sees the release from then on.

Conventions

  • Keep shared vs platform boundaries: no AppKit or UIKit in shared files outside #if os(...).
  • Main-actor correctness for anything that touches UI; progress callbacks reach the coordinator on the main actor.
  • New copy or verify behavior has tests for overwrite, source change, hidden files, symlinks and reporting.
  • Use BitMatchError or FileOperationError for user-facing errors, with a message that says what to do.
  • No naked TODOs in committed code: write "Future enhancement:" with intent, or file an issue.
  • Cancel does not delete copied files. Cancel-time cleanup stays disabled until the pipeline has more field validation.
  • Do not replace screenshots with mocks, stale simulator images, or views that show personal paths or media.