diff --git a/CHANGELOG.md b/CHANGELOG.md index 5a060f6..9245caf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,16 @@ ## Mobile (not released yet) +- Android test signal: Signal → Start test signal now works on Android, with + the same picture, options (1 or 2 events/s, title, subtitle, image), AVS4 QR + (test ID, title, sender calibration), stable sender ID and "Stopped" screen + as on the iPhone. Frames are timed with Choreographer, the sound with one + AudioTrack placed to the sample by its output timestamps + (`docs/phone-generator.md`). Not tested on a phone yet. +- Calibrator wizard on Android: Settings → With calibrator now shows on Android + too. The camera step uses the new camera view method `beginCalibrator` + (lowest ISO where the camera allows it), the screen step the generator's + calibration mode with the microphone listening. Not tested with a board yet. - Analysers subtract the sender calibration: when the AVS4 QR carries one, the AV offset shown and saved is the measured value minus the sender's own screen-vs-speaker offset. Measure and Results show the raw value and the diff --git a/IMPLEMENTATION_PLAN.md b/IMPLEMENTATION_PLAN.md index 32bb522..758525a 100644 --- a/IMPLEMENTATION_PLAN.md +++ b/IMPLEMENTATION_PLAN.md @@ -1288,8 +1288,9 @@ sender calibration) is one more entry. Name, subtitle and image are not inputs. when used, so a live rate change gives the new id at once, and config export keeps the empty value instead of freezing an old id. `GET /api/v1/generator` shows `test_id`. The analyzer's own fallback label stays the NMOS seed. -- Android has no generator yet (`supportsGenerator` is false); it should call the - same core function when it gets one. +- Android (2026-11-08): prefix `phone-`, a random install ID in the app's + preferences (new after a reinstall), version ` ()`, + settings `rate=<1|2>`, through the same core function. The core test holds the shared test vector. Generator name in the QR: not done. The live QR of the phone and the container @@ -1369,9 +1370,8 @@ Maintainer design (2026-11-08): Settings → With calibrator, a three-step wizar - Screen calibration is a mode of the generators, not a React view: iPhone `generateStyled(..., {mode: 'calibration'})` (same display link, layers, audio engine and player node), container `AVSYNC_PATTERN=calibration` - (live key, Signal API `pattern`, web form "Picture"). Android has no - generator (`supportsGenerator` false): skipped; its analyser could call - `begin_calibrator` through JNI later (not wired). + (live key, Signal API `pattern`, web form "Picture"). Android: added + later the same way (see "Android generator and calibrator take"). - Results: camera → analyser calibration record (method `calibrator-pulse`, same validity as the clap); screen → `senderCalibration` in the session state (method `transponder`, 90 days, same OS version, sent only on the same @@ -1400,9 +1400,38 @@ Deviations and choices: (`raw_av_offset_ms` / `rawOffsetMs`) and the calibration next to it. The phone's analyser calibration stays in the app (`liveView`), so nothing is counted twice. -- The wizard and the calibrator take are iOS only (`beginCalibrator` is not in - the Android camera view yet); the Settings entry is hidden on Android. +- The wizard and the calibrator take were iOS only at first; Android got them + with its generator (next section). - Debug → Clear calibration clears the sender calibration too (decided; with a confirm that says both are cleared). - The iPhone pattern size assumes about 6 pt per mm (current iPhones); the outline is a placing guide, the pattern line itself is exact. + +## Android generator and calibrator take (2026-11-08) + +Maintainer rule: all generators get the same features. Android gets the phone +generator (test signal and calibration pattern) and the calibrator take, behind +the same JS API as iOS (`supportsGenerator`, `generate`, `generateStyled`, +`pickGeneratorImage`, `generatorImageInfo`, camera view `beginCalibrator`), so +the app has no platform branch for them (only the Title placeholder). + +- Shared core and renderer: `PhoneGeneratorPlan`, `generator_test_id`, the AVS4 + QR, `render_picture`/`render_sync_foreground` and the calibration functions + compile into the Android library as on iOS (`generator_jni.cpp`). +- Text: Android draws the renderer's text itself (DejaVu Sans from the iOS + resource folder, `GeneratorText.kt` through JNI), as iOS uses CoreText. + Deviation from the server: not Blend2D; glyph shapes can differ by a pixel. +- Frame time: Choreographer's expected presentation time on Android 13+; + before that the frame start plus two display periods (an estimate). The + screen asks for a 60 Hz display mode; on a display that stays at 90/120 Hz + each frame still shows the 60 Hz plan's frame for its time. +- Sound: one streaming AudioTrack, packets placed by `AudioTrack.getTimestamp` + (frame ↔ System.nanoTime) instead of AAudio/Oboe: exact sample placement from + API 24, no native audio code or new dependency. The timestamp includes the + output latency, so nothing is subtracted. An underrun stops the signal. +- Calibrator camera take: lowest ISO when the camera allows manual exposure, + else auto exposure locked at the lowest exposure compensation; high-speed + mode only AE lock. +- The calibration pattern is drawn in real millimetres (display xdpi). +- Not verified on a device here (no emulator with KVM): timing, the pattern, + the microphone during playback, the photo picker. Device test list in the PR. diff --git a/docs/calibration.md b/docs/calibration.md index 1577c7e..7abe920 100644 --- a/docs/calibration.md +++ b/docs/calibration.md @@ -10,7 +10,7 @@ The rig is an LED and a piezo on one microcontroller pin. Parts and firmware are The calibrator is a small board with an LED, a buzzer and two light sensors ([signal contract](calibrator.md)). The phone powers it over a USB-C cable. -In the app: **Settings → With calibrator** (iPhone for now; the clap +In the app: **Settings → With calibrator** (iPhone and Android; the clap calibration stays as it is). Three steps: 1. **Plug in** the calibrator. diff --git a/docs/calibrator.md b/docs/calibrator.md index 0a4f5bb..1b8d1a4 100644 --- a/docs/calibrator.md +++ b/docs/calibrator.md @@ -1,6 +1,6 @@ # AV Sync calibrator: signal contract -**v1 proposal. The app side is built (iOS app and container generator, see [App side](#app-side-built)), but only tested on synthetic signals. The board is not built or tested yet.** +**v1 proposal. The app side is built (iOS and Android app and container generator, see [App side](#app-side-built)), but only tested on synthetic signals. The board is not built or tested yet.** The calibrator is a small board (66 × 22 mm) that the phone powers over a USB-C to USB-C cable. The cable is not included: the user brings their own. @@ -173,7 +173,7 @@ flips, no timed delay). Then it ignores the sensors for **200 ms** (lock-out). ## App side (built) Code: `core/include/avsync/calibrator.hpp` (all analysis, shared by every app), -`PhoneAnalyzer::begin_calibrator` (camera take), the iPhone generator's +`PhoneAnalyzer::begin_calibrator` (camera take), the iPhone and Android generators' calibration mode and the container's `AVSYNC_PATTERN=calibration`. User guide: [calibration.md](calibration.md#with-the-calibrator). @@ -197,7 +197,9 @@ flash lands on different rows, and click − flash grows with the row by exactly the readout time. Flashes and clicks are paired by the most common difference (±12 ms around it). Offset = median of the pairs after dropping outliers (beyond 3 σ, at least 0.5 ms); uncertainty = standard error of the median. -For the camera take the iPhone locks the exposure at the lowest ISO. +For the camera take the iPhone locks the exposure at the lowest ISO; Android +too when the camera allows manual exposure, else it locks auto exposure at the +lowest exposure compensation (and in the high-speed mode, which allows neither). **Display mode.** Every click is paired with the next beep less than 450 ms later. Offset = beep − click − D, the median per swap direction (every other @@ -207,8 +209,8 @@ times, so the same analysis works for a TV fed by the container generator. **Pattern.** Every generator uses the same core functions: period k lasts 500 ms on its frame grid (for 59.94 fps the swap is the first frame at or after k × 500 ms), left half white in even periods, the beep D after each -swap. The iPhone draws the halves in portrait near the bottom microphone, -about 62 mm tall, with a red outline of the board (sensor end down, notch +swap. The phones draw the halves in portrait near the bottom microphone, +about 62 mm tall (Android in real millimetres from the display density), with a red outline of the board (sensor end down, notch tick). The container fills the whole picture and draws a grey tick at both ends of the pattern line. diff --git a/docs/mobile-native.md b/docs/mobile-native.md index 5bb1454..b98ecb6 100644 --- a/docs/mobile-native.md +++ b/docs/mobile-native.md @@ -17,14 +17,15 @@ the core, so both platforms give the same numbers. | `synchronizeClock(server)` | Promise of `{source:'ntp', server, referenceUnixMs, referenceMonotonicMs, uncertaintyMs, roundTripMs, offsetMs, clockId, timebase}` (SNTP, 8 samples, smallest round trip) | | `setScreenAwake(on)` | keeps the display on | | `generate(rate, taiUtcSeconds, options)` | Promise; shows the fullscreen AVS3 source until closed, resolves `{skippedDisplayFrames}` | -| `generateStyled(rate, taiUtcSeconds, options)` | the same with string options; `mode: 'calibration'` shows the calibration pattern and resolves with `calibration` or `calibrationError`; `senderCalibrationMethod/OffsetMs/UncertaintyMs/Day/Route` put the sender calibration in the QR (iOS; [phone-generator.md](phone-generator.md#calibration-mode)) | +| `generateStyled(rate, taiUtcSeconds, options)` | the same with string options; `mode: 'calibration'` shows the calibration pattern and resolves with `calibration` or `calibrationError`; `senderCalibrationMethod/OffsetMs/UncertaintyMs/Day/Route` put the sender calibration in the QR ([phone-generator.md](phone-generator.md#calibration-mode)) | +| `pickGeneratorImage()` / `generatorImageInfo(id)` | the generator image: Promise of `{id, uri, width, height}` or null (cancelled); info for a saved id | ## Camera view `Avsync` Props: `active` (bool), `captureFps` (0 = automatic best rate), `taiUtcOffset` (seconds, default 37). Methods: `beginCalibration()`, `resumeAnalysis()`, `focusAt(x, y)` (tap to focus; x and y are 0–1 across the view, see below), -`beginCalibrator(mode)` (iOS only so far; `camera` or `display`, see +`beginCalibrator(mode)` (`camera` or `display`, see [calibrator.md](calibrator.md#app-side-built)). Event `onReport` with one of: diff --git a/docs/phone-generator.md b/docs/phone-generator.md index 1f47449..e5447ae 100644 --- a/docs/phone-generator.md +++ b/docs/phone-generator.md @@ -1,6 +1,6 @@ -# iPhone test-signal source +# Phone test-signal source (iPhone and Android) -Use the current native iOS/TestFlight build, open **Generate**, choose **1 or 2 chirps/s**, +Use the current native iOS/TestFlight build or the Android APK, open **Signal**, choose **1 or 2 chirps/s**, set **Title**, **Subtitle** and optionally **Choose image**, then **Start fullscreen signal**. Hold the phone sideways. The raster rotates inside the fullscreen screen; Measure/Results keep their portrait layout. Film the entire test picture, including all four corner fiducials, with an @@ -40,13 +40,36 @@ ends. Normal server and FILE packets retain their original bytes/semantics. ## Native scheduling and lifecycle -A native CADisplayLink supplies the next presentation target. The first source +iPhone: a native CADisplayLink supplies the next presentation target. The first source event is planned 1.5 s ahead on that cadence. Subsequent source frames/events follow a 60-Hz logical host-clock timeline. AVAudioPlayerNode buffers are queued 2.5 s ahead on the same host clock, subtracting the node's reported downstream output presentation latency. This is an OS timing estimate, not a microphone or photodiode measurement. All timing and synthesis run outside JavaScript. +Android does the same with its own clocks (`AvsyncGenerator.kt`): + +- **Picture.** A fullscreen screen over the app. Choreographer gives each + frame's expected presentation time (Android 13 and newer; before that the + frame start plus two display periods, an estimate). The screen asks for a + 60 Hz display mode. Each frame shows what the 60 Hz plan says for that time. +- **Sound.** One AudioTrack stream (48 kHz, mono) that a thread keeps writing + in 5 ms blocks, silence between the packets. `AudioTrack.getTimestamp` says + which sample leaves the output at which time, on the same clock as + Choreographer (`System.nanoTime`). A packet planned for time T starts at + that sample + (T − timestamp time) × 48 000, so it is placed to the sample, + with the reported output latency already inside. The position is worked out + again with the newest timestamp until the packet starts, so drift between + the audio clock and the system clock does not add up. Why AudioTrack and not + AAudio/Oboe: the timestamp gives the same exact mapping from API 24 on, + without native audio code or a new library; low latency does not matter + because every packet is planned 2.5 s ahead. A dropout (underrun) stops the + signal, because everything already written would then play late. +- **Same rules otherwise.** The same 1.5 s start, 2.5 s queue, status line, + "Stopped" screen, brightness and screen wake, and the same stop reasons + (app in the background, audio focus lost, output changed, clock jump, missed + deadline, 23 hours). + The screen reports skipped logical display frames and the reported audio lead. It prevents automatic screen lock and raises brightness while active, restoring both when stopped. Stop/Close cancel queued sound and clear the whole timing picture; a large "Stopped" shows instead. @@ -63,7 +86,10 @@ with the calibration pattern instead of the test picture layers, audio engine and player node; the beep (1.5 kHz, 4 ms) is scheduled 100 ms after each swap on the host clock, minus the reported output latency, like the chirps. The audio session is play-and-record on the built-in speaker -so the microphone can listen. The result comes back when the screen closes: +so the microphone can listen. Android: the same frame clock and AudioTrack as +the test signal (the beep is placed on the output's timestamps), the +microphone records with AudioRecord meanwhile, and the outline is drawn in +real millimetres from the display's density. The result comes back when the screen closes: `calibration` (`offsetMs`, `uncertaintyMs`, `pairs`, `directionMs`, `audioRoute`, `outputLatencyMs`) or `calibrationError`. @@ -72,8 +98,8 @@ so the microphone can listen. The result comes back when the screen closes: The source ID (the QR test id; the audio packet carries its CRC-32) stays the same when you stop and start the generator again. It is `phone-` plus 12 hex digits of a hash of a random install ID, the app version with its build -number (for example `2.0.8 (12)`) and the timing settings (now only the event -rate). So you get a new ID after a reinstall, a new app version or build (also a +number (for example `2.0.8 (12)`; Android: version name and version code) and +the timing settings (now only the event rate). So you get a new ID after a reinstall, a new app version or build (also a new TestFlight build) or another event rate. Name, subtitle and image do not change it. The install ID is random, not a hardware ID, and only lives in the app's settings. The container generator uses the same rule with its NMOS seed in @@ -124,6 +150,12 @@ Expo Go does not contain this native module. Rebuild/install the iOS development client after updating this branch (`npm install`, `npx pod-install`, then `npx expo run:ios` or the existing Xcode workspace/device workflow). +Android, from `mobile-app/android` after `npx expo prebuild -p android`: +`./gradlew :avsync:testDebugUnitTest` (audio placement and screen layout) and, +with a device or emulator, `./gradlew :avsync:connectedDebugAndroidTest` +(`PhoneGeneratorTest`: the picture with Android text and the AVS4 QR read by +ML Kit, the audio packet, the stable ID test vector, the calibration pattern). + ```sh cmake --build build --target avsync-core-tests avsync-picture-tests ctest --test-dir build --output-on-failure @@ -167,12 +199,15 @@ QR, source labels/logo and the early/zero/late sync scale. The square is red while early, white at the event and green while late; its zero crossing shares the planned chirp/flash onset. iOS animates it with native display-timestamped layers over a static plate, with the same coordinates/colors as the renderer. -The bundled font is DejaVu Sans on both targets (CoreText on iOS, Blend2D on -the server); source-event labels describe scheduled phone output. +The bundled font is DejaVu Sans on every target (CoreText on iOS, Android's +own text drawing on Android, Blend2D on the server); source-event labels +describe scheduled phone output. Android draws the same layers in one view. Name/subtitle and the selected image persist locally. Photos selection uses -the system picker, which shares only the image you select. The app stores a -normalized copy up to 512 px in private Application Support; Remove image +the system picker (Android: the photo picker, or the document picker before +Android 13), which shares only the image you select. The app stores a +normalized copy up to 512 px in private Application Support (Android: the +app's private files); Remove image removes that copy, preserving the original in Photos. Images are not uploaded with EAS builds or sent to a server. Labels/image are display identity and do not change protected timecode/source IDs or calibration. diff --git a/mobile-app/App.js b/mobile-app/App.js index bc28e74..20c962f 100644 --- a/mobile-app/App.js +++ b/mobile-app/App.js @@ -115,7 +115,7 @@ export default function App() { // With calibrator: step 1 is a camera take (this phone's camera vs microphone), step 2 the // generator's calibration pattern (this phone's screen vs speaker, sent in the AVS4 QR). - const supportsCalibrator = Platform.OS === 'ios' && Boolean(NativeCamera) && Boolean(native?.supportsGenerator?.()); + const supportsCalibrator = Boolean(NativeCamera) && Boolean(native?.supportsGenerator?.()); const startCalibrator = () => { setSheet(null); setTab('measure'); setNotice(null); setFlow(null); setCal({ step: 'plug' }); }; const closeCalibrator = message => { setCal(null); restart(); setNotice(message ?? null); }; const calibratorStart = async base => { diff --git a/mobile-app/README.md b/mobile-app/README.md index 3e34515..ff26a65 100644 --- a/mobile-app/README.md +++ b/mobile-app/README.md @@ -71,7 +71,7 @@ clap once. Then pick the first frame where the hands touch; the sound is marked orange. Save stores sound time minus that frame's time; it applies to AV offset only, for this OS version and camera mode, for 90 days. -Settings → With calibrator (iPhone): a wizard for the calibrator board. Step 2 +Settings → With calibrator: a wizard for the calibrator board. Step 2 measures camera vs microphone with the board's flashes and clicks and saves it like the clap calibration. Step 3 measures this phone's screen vs speaker with the generator's calibration pattern and saves it as the sender calibration, @@ -83,7 +83,9 @@ next to it. Debug → Clear calibration clears both (after a confirm). Details: ## Signal Signal → Start test signal shows the fullscreen AVS4 source with its sound -(2 events/s; 1/s under Options, with title, subtitle and image). +(2 events/s; 1/s under Options, with title, subtitle and image). iPhone and +Android have the same signal and options +([phone generator](../docs/phone-generator.md)). ## Settings and Debug @@ -107,7 +109,11 @@ per seq, share diagnostics, and clear results or calibration. `modules/avsync/android` builds the core into `libavsync-jni` (CMake, C++20, arm64-v8a and x86_64 only: the core uses `__int128`) and implements the [native contract](../docs/mobile-native.md) with Camera2, AudioRecord and ML Kit -(bundled QR model). Capture uses the normal session (ImageReader) up to the +(bundled QR model). The generator (`AvsyncGenerator.kt`) draws the shared +picture with Choreographer frame times and plays the sound through one +AudioTrack placed by its timestamps +([phone generator](../docs/phone-generator.md#native-scheduling-and-lifecycle)). +Capture uses the normal session (ImageReader) up to the best rate it lists; when 120/240 fps exist only in the constrained high-speed mode, `HighSpeedCapture.kt` reads the frames on the GPU instead, with a fallback to the normal session ([capture paths](../docs/mobile-native.md#android-capture-paths)). @@ -128,9 +134,11 @@ cd android ``` Unit tests cover the camera mode choice (normal and high-speed), the -high-speed texture crop, the fixed-exposure ISO, SNTP and the -calibration WAV/waveform. `src/androidTest` checks the JNI bridge (live -snapshot/poll and a clap calibration take). +high-speed texture crop, the fixed-exposure ISO, SNTP, the +calibration WAV/waveform, the generator's audio placement and its screen +layout. `src/androidTest` checks the JNI bridge (live snapshot/poll and a +clap calibration take) and the generator (picture with the AVS4 QR, audio, +stable ID, calibration pattern). Not yet verified: the app on the emulator with a camera preview, the instrumented JNI test, and any real device (frame rate, exposure lock, the diff --git a/mobile-app/components/Generate.js b/mobile-app/components/Generate.js index 80a5716..564ab16 100644 --- a/mobile-app/components/Generate.js +++ b/mobile-app/components/Generate.js @@ -1,5 +1,5 @@ import { useState } from 'react'; -import { Image, Pressable, ScrollView, Text, TextInput } from 'react-native'; +import { Image, Platform, Pressable, ScrollView, Text, TextInput } from 'react-native'; import { Button, Card, Choice, colors, small, text } from './ui'; const input = { ...text, backgroundColor: colors.bg, padding: 12, borderRadius: 8, marginTop: 6 }; @@ -23,7 +23,7 @@ export default function Generate({ session, onChange, image, busy, picking, onPi onChange('generatorRate', value)} disabled={busy} /> Use 1 / s when the measuring camera runs at 30 fps. Title - onChange('generatorName', value)} style={input} /> + onChange('generatorName', value)} style={input} /> Subtitle onChange('generatorSubtitle', value)} style={input} /> {image && } diff --git a/mobile-app/modules/avsync/android/CMakeLists.txt b/mobile-app/modules/avsync/android/CMakeLists.txt index 04752aa..168dd3c 100644 --- a/mobile-app/modules/avsync/android/CMakeLists.txt +++ b/mobile-app/modules/avsync/android/CMakeLists.txt @@ -1,15 +1,17 @@ cmake_minimum_required(VERSION 3.22) project(avsync-android CXX) -# The shared phone analyzer, compiled from the repository's core like the iOS Core_*.cpp wrappers. +# The shared phone analyzer and generator, compiled from the repository's core like the iOS Core_*.cpp +# wrappers, plus the shared picture renderer with Android text (src/main/cpp/text_android.cpp). set(REPO ${CMAKE_CURRENT_SOURCE_DIR}/../../../..) -set(CORE_SOURCES chirp detect g711 grade locate marker optical_code phone phone_analyzer plan qr sync_scale timecode two_way calibrator) +set(CORE_SOURCES chirp detect g711 grade locate marker optical_code phone phone_analyzer phone_generator plan qr sync_scale timecode two_way calibrator) list(TRANSFORM CORE_SOURCES PREPEND ${REPO}/core/src/) list(TRANSFORM CORE_SOURCES APPEND .cpp) -add_library(avsync-jni SHARED src/main/cpp/avsync_jni.cpp ${CORE_SOURCES} ${REPO}/third_party/nayuki/qrcodegen.cpp) +add_library(avsync-jni SHARED src/main/cpp/avsync_jni.cpp src/main/cpp/generator_jni.cpp src/main/cpp/text_android.cpp + ${CORE_SOURCES} ${REPO}/generator/picture_render.cpp ${REPO}/third_party/nayuki/qrcodegen.cpp) set_target_properties(avsync-jni PROPERTIES CXX_STANDARD 20 CXX_STANDARD_REQUIRED ON) -target_include_directories(avsync-jni PRIVATE ${REPO}/core/include ${REPO}/third_party/pocketfft ${REPO}/third_party/nayuki) +target_include_directories(avsync-jni PRIVATE ${REPO}/core/include ${REPO}/generator ${REPO}/third_party/pocketfft ${REPO}/third_party/nayuki) target_compile_options(avsync-jni PRIVATE -O2 -fexceptions) target_link_options(avsync-jni PRIVATE -Wl,-z,max-page-size=16384) target_link_libraries(avsync-jni PRIVATE log) diff --git a/mobile-app/modules/avsync/android/build.gradle b/mobile-app/modules/avsync/android/build.gradle index b69ea6c..3dc77a7 100644 --- a/mobile-app/modules/avsync/android/build.gradle +++ b/mobile-app/modules/avsync/android/build.gradle @@ -19,6 +19,8 @@ android { ndk { abiFilters 'arm64-v8a', 'x86_64' } } externalNativeBuild { cmake { path 'CMakeLists.txt'; version '3.22.1' } } + // The generator's font and icon: the same files as the iOS resource bundle (DejaVuSans.ttf, icon.png, LICENSE). + sourceSets { main { assets.srcDirs += '../ios/Resources' } } // React Native ships its own libc++_shared.so; the instrumented test APK would get two. packaging { jniLibs { pickFirsts += ['**/libc++_shared.so'] } } } diff --git a/mobile-app/modules/avsync/android/src/androidTest/java/io/leeo86/mxl/avsync/PhoneGeneratorTest.kt b/mobile-app/modules/avsync/android/src/androidTest/java/io/leeo86/mxl/avsync/PhoneGeneratorTest.kt new file mode 100644 index 0000000..979972a --- /dev/null +++ b/mobile-app/modules/avsync/android/src/androidTest/java/io/leeo86/mxl/avsync/PhoneGeneratorTest.kt @@ -0,0 +1,60 @@ +package io.leeo86.mxl.avsync +import android.graphics.Color +import androidx.test.ext.junit.runners.AndroidJUnit4 +import androidx.test.platform.app.InstrumentationRegistry +import com.google.android.gms.tasks.Tasks +import com.google.mlkit.vision.barcode.BarcodeScanning +import com.google.mlkit.vision.common.InputImage +import org.json.JSONObject +import org.junit.Assert.* +import org.junit.Test +import org.junit.runner.RunWith + +// The generator bridge on a device or emulator: the shared renderer with Android text, the AVS4 QR, +// the audio packet, the stable ID and the calibration pattern. Timing on a real screen and speaker +// is a device test (docs/phone-generator.md). +@RunWith(AndroidJUnit4::class) +class PhoneGeneratorTest { + private val context = InstrumentationRegistry.getInstrumentation().targetContext + + @Test fun stableIdMatchesTheCoreVector() { + // docs/phone-generator.md: the shared test vector of generator_test_id. + assertEquals("phone-985274814721", PhoneGenerator.testId("phone-", "00000000-0000-4000-8000-000000000000", "2.0.8", arrayOf("rate=1"))) + val id = AvsyncGenerator.stableSourceID(context, 2) + assertEquals(id, AvsyncGenerator.stableSourceID(context, 2)) + assertNotEquals(id, AvsyncGenerator.stableSourceID(context, 1)) + } + + @Test fun pictureCarriesTheAvs4Qr() { + GeneratorText.load(context) + assertTrue(GeneratorText.width("AVsync", 16f) > 20) + PhoneGenerator("phone-985274814721", 2, 1_800_000_000_000_000_000L, 100.0, "Android").use { g -> + g.setSource("Studio A", "Camera 3", null) + g.setSenderCalibration("transponder", -38.0, 0.5, 2100) + val image = g.image(3) + assertEquals(1280, image.width); assertEquals(720, image.height) + val codes = Tasks.await(BarcodeScanning.getClient().process(InputImage.fromBitmap(image, 0))) + val text = codes.mapNotNull { it.rawValue }.first { it.startsWith("AVS4:") } + assertTrue(text.isNotEmpty()) + // Event timing: event 3 at 2/s is 1 s after the epoch. + assertEquals(101.0, g.hostTime(3), 1e-9) + assertTrue(g.audio(3).size > 1000) + // Frame 60 (1 s after the epoch) belongs to event 3, position 0. + val frame = g.frame(101.0 + 1e-6) + assertEquals(3L, frame.sequence); assertEquals(0, frame.position) + val foreground = g.syncForeground() + assertTrue((0 until 720 step 4).any { y -> (0 until 1280 step 4).any { x -> foreground.getPixel(x, y) == Color.WHITE } }) + } + } + + @Test fun calibrationPatternAndAnalysis() { + // Period k lasts 500 ms: at 60 fps the swaps are every 30 frames, left half white in even periods. + assertTrue(PhoneGenerator.calibrationLeftWhite(0, 60)); assertFalse(PhoneGenerator.calibrationLeftWhite(30, 60)) + assertEquals(30L, PhoneGenerator.calibrationSwapFrame(1, 60)) + val beep = PhoneGenerator.calibrationBeep(48_000) + assertEquals(192, beep.size) // 4 ms + // Silence: no pairs, a clear error. + val json = JSONObject(PhoneGenerator.displayCalibration(FloatArray(48_000), 48_000, 48_000, 8)) + assertFalse(json.getBoolean("ok")); assertEquals(0, json.getInt("pairs")) + } +} diff --git a/mobile-app/modules/avsync/android/src/main/cpp/avsync_jni.cpp b/mobile-app/modules/avsync/android/src/main/cpp/avsync_jni.cpp index 1ed4b0d..4d68782 100644 --- a/mobile-app/modules/avsync/android/src/main/cpp/avsync_jni.cpp +++ b/mobile-app/modules/avsync/android/src/main/cpp/avsync_jni.cpp @@ -61,6 +61,15 @@ JNIEXPORT jstring JNICALL AVSYNC(calibrationError)(JNIEnv* env, jclass, jlong h) return error ? env->NewStringUTF(error->c_str()) : nullptr; } +// Calibrator board take (docs/calibrator.md): camera (flash + click) or display (an external screen, microphone only). +JNIEXPORT void JNICALL AVSYNC(beginCalibrator)(JNIEnv*, jclass, jlong h, jboolean display) +{ + self(h)->begin_calibrator(display ? PhoneAnalyzer::CalibratorMode::Display : PhoneAnalyzer::CalibratorMode::Camera); +} +JNIEXPORT jboolean JNICALL AVSYNC(calibratorActive)(JNIEnv*, jclass, jlong h) { return self(h)->calibrator_active(); } +// A progress report about once a second, then the result or an error (JSON), else null. +JNIEXPORT jstring JNICALL AVSYNC(pollCalibrator)(JNIEnv* env, jclass, jlong h) { return text(env, self(h)->poll_calibrator()); } + // Returns a PhoneAnalyzer.CalibrationTake, or null when no take is ready. JNIEXPORT jobject JNICALL AVSYNC(takeCalibration)(JNIEnv* env, jclass, jlong h) { diff --git a/mobile-app/modules/avsync/android/src/main/cpp/generator_jni.cpp b/mobile-app/modules/avsync/android/src/main/cpp/generator_jni.cpp new file mode 100644 index 0000000..23ef9b1 --- /dev/null +++ b/mobile-app/modules/avsync/android/src/main/cpp/generator_jni.cpp @@ -0,0 +1,240 @@ +// JNI bridge to avsync::PhoneGeneratorPlan, the shared picture renderer and the calibration pattern, +// like ios/AVSyncGenerator.mm. One handle per Kotlin PhoneGenerator. image() runs on a worker while +// frame() runs on the main thread: both only read the plan, and image() renders from a copy of the info. +#include "avsync/calibrator.hpp" +#include "avsync/phone_generator.hpp" +#include "picture.hpp" +#include "text_android.hpp" + +#include + +#include +#include +#include +#include +#include + +namespace { +struct Generator { + std::unique_ptr plan; + avsync::PictureInfo info; + avsync::SourceStill source_image; + avsync::SourceStill app_icon; + avsync::SenderCalibration calibration; +}; +Generator* self(jlong handle) { return reinterpret_cast(handle); } + +std::string utf8(JNIEnv* env, jstring value) +{ + if (!value) return {}; + const char* chars = env->GetStringUTFChars(value, nullptr); + std::string out(chars ? chars : ""); + env->ReleaseStringUTFChars(value, chars); + return out; +} + +// 10-bit BT.709 limited range to 8-bit RGB, the same as the iOS bridge. +void rgb(uint16_t y, uint16_t cb, uint16_t cr, uint8_t* out) +{ + const double l = (double(y) - 64) / 876, u = (double(cb) - 512) / 896, v = (double(cr) - 512) / 896; + out[0] = uint8_t(std::lround(std::clamp(l + 1.5748 * v, 0.0, 1.0) * 255)); + out[1] = uint8_t(std::lround(std::clamp(l - .187324 * u - .468124 * v, 0.0, 1.0) * 255)); + out[2] = uint8_t(std::lround(std::clamp(l + 1.8556 * u, 0.0, 1.0) * 255)); +} + +// ARGB ints for Bitmap.createBitmap. white_only: the sync foreground mask (white or transparent). +jintArray pixels(JNIEnv* env, const avsync::Yuv422p10& frame, bool white_only = false) +{ + const jsize n = jsize(frame.width) * frame.height; + std::vector argb(size_t(n), 0); + for (int y = 0; y < frame.height; ++y) + for (int x = 0; x < frame.width; ++x) { + const size_t i = size_t(y) * frame.width + x, ci = size_t(y) * (frame.width / 2) + x / 2; + if (white_only) { argb[i] = frame.y[i] == avsync::kYWhite ? jint(0xFFFFFFFF) : 0; continue; } + uint8_t c[3]; + rgb(frame.y[i], frame.cb[ci], frame.cr[ci], c); + argb[i] = jint(0xFF000000u | uint32_t(c[0]) << 16 | uint32_t(c[1]) << 8 | c[2]); + } + jintArray out = env->NewIntArray(n); + env->SetIntArrayRegion(out, 0, n, argb.data()); + return out; +} + +// A still from ARGB ints (straight alpha, already scaled to at most 512 px by the caller). +void still(JNIEnv* env, jintArray argb, jint width, jint height, avsync::SourceStill& out) +{ + out = {}; + if (!argb || width <= 0 || height <= 0 || env->GetArrayLength(argb) < width * height) return; + std::vector in(size_t(width) * height); + env->GetIntArrayRegion(argb, 0, width * height, in.data()); + out.width = width; out.height = height; out.rgba.resize(in.size() * 4); + for (size_t i = 0; i < in.size(); ++i) { + const auto p = uint32_t(in[i]); + out.rgba[i * 4] = uint8_t(p >> 16); out.rgba[i * 4 + 1] = uint8_t(p >> 8); out.rgba[i * 4 + 2] = uint8_t(p); out.rgba[i * 4 + 3] = uint8_t(p >> 24); + } +} +} // namespace + +#define GENERATOR(name) Java_io_leeo86_mxl_avsync_PhoneGenerator_##name + +extern "C" { + +JNIEXPORT jstring JNICALL GENERATOR(testId)(JNIEnv* env, jclass, jstring prefix, jstring install, jstring version, jobjectArray settings) +{ + std::vector values; + for (jsize i = 0, n = settings ? env->GetArrayLength(settings) : 0; i < n; ++i) { + auto item = jstring(env->GetObjectArrayElement(settings, i)); + values.push_back(utf8(env, item)); + env->DeleteLocalRef(item); + } + return env->NewStringUTF(avsync::generator_test_id(utf8(env, prefix), utf8(env, install), utf8(env, version), values).c_str()); +} + +// Returns 0 when the plan is invalid (the core throws). +JNIEXPORT jlong JNICALL GENERATOR(create)(JNIEnv* env, jclass, jstring testId, jint rate, jlong epochTaiNs, jdouble epochHost, jstring source) +{ + try { + auto g = std::make_unique(); + g->plan = std::make_unique(utf8(env, testId), int(rate), int64_t(epochTaiNs), double(epochHost)); + g->info.title = "AVsync"; g->info.source = utf8(env, source); + g->info.format_line = "720p60 / 1200 baud / " + std::to_string(rate) + " Hz"; + g->info.notes = "Estimated phone clock/output"; g->info.offset_line = "Set AV offset 0 ms"; + g->info.period_ms = 1000.0 / rate; g->info.show_sync_indicator = false; + return reinterpret_cast(g.release()); + } catch (const std::exception&) { return 0; } +} +JNIEXPORT void JNICALL GENERATOR(destroy)(JNIEnv*, jclass, jlong h) { delete self(h); } + +JNIEXPORT void JNICALL GENERATOR(setSource)(JNIEnv* env, jclass, jlong h, jstring name, jstring subtitle, jintArray argb, jint width, jint height) +{ + auto* g = self(h); + const auto shown = utf8(env, name); + if (!shown.empty()) g->info.source = shown; + g->info.subtitle = utf8(env, subtitle); + still(env, argb, width, height, g->source_image); + g->info.source_image = g->source_image.width > 0 ? &g->source_image : nullptr; +} +JNIEXPORT void JNICALL GENERATOR(setIcon)(JNIEnv* env, jclass, jlong h, jintArray argb, jint width, jint height) +{ + auto* g = self(h); + still(env, argb, width, height, g->app_icon); + g->info.app_icon = g->app_icon.width > 0 ? &g->app_icon : nullptr; +} + +// AVS4 sender calibration (this phone's screen vs speaker); method none unless valid. +JNIEXPORT void JNICALL GENERATOR(setSenderCalibration)(JNIEnv* env, jclass, jlong h, jstring method, jdouble offset, jdouble uncertainty, jint day) +{ + auto& c = self(h)->calibration; + c = {}; + const auto m = utf8(env, method); + // The QR encoder throws on values it cannot carry: keep method none then. + if ((m != "clap" && m != "transponder" && m != "pulse-train") || !std::isfinite(offset) || std::abs(offset) > 3276 || !std::isfinite(uncertainty) || + uncertainty < 0 || uncertainty > 127 || day < 0 || day > 0xFFFF) + return; + c.method = m; c.offset_ms = offset; c.uncertainty_ms = uncertainty; c.day = int(day); +} + +// [markerX, markerY, markerSide] in the 1280 × 720 raster. +JNIEXPORT jintArray JNICALL GENERATOR(geometry)(JNIEnv* env, jclass, jlong) +{ + const auto g = avsync::qr_flash_geometry(1280, 720); + const jint values[3] = {jint(g.marker_x), jint(g.marker_y), jint(g.marker_side)}; + jintArray out = env->NewIntArray(3); + env->SetIntArrayRegion(out, 0, 3, values); + return out; +} + +JNIEXPORT jintArray JNICALL GENERATOR(syncForeground)(JNIEnv* env, jclass, jlong h) +{ + auto* g = self(h); + avsync::TextEnv text(env); + avsync::Yuv422p10 frame; + frame.resize(1280, 720); + avsync::render_sync_foreground(frame, g->plan->period_frames(), g->info.period_ms, true); + return pixels(env, frame, true); +} + +// The frame shown at this host time: [sequence, position, aWhite, named, syncX, syncY, syncSize, syncR, syncG, syncB]. +JNIEXPORT jdoubleArray JNICALL GENERATOR(frame)(JNIEnv* env, jclass, jlong h, jdouble target) +{ + auto* g = self(h); + const auto frame = g->plan->frame_at(target); + const int period = g->plan->period_frames(); + const uint64_t seq = 1 + frame / period; + const int pos = int(frame % period); + const auto dot = avsync::sync_indicator(1280, 720, pos, period, g->info.period_ms, true); + uint8_t c[3]; + rgb(dot.y10, dot.cb10, dot.cr10, c); + const jdouble values[10] = {double(seq), double(pos), g->plan->a_white(frame) ? 1.0 : 0.0, double(g->plan->named_seq(frame)), + double(dot.x), double(dot.y), double(dot.side), double(c[0]), double(c[1]), double(c[2])}; + jdoubleArray out = env->NewDoubleArray(10); + env->SetDoubleArrayRegion(out, 0, 10, values); + return out; +} + +JNIEXPORT jdouble JNICALL GENERATOR(hostTime)(JNIEnv*, jclass, jlong h, jlong seq) +{ + try { return self(h)->plan->host_seconds(uint64_t(seq)); } catch (const std::exception&) { return NAN; } +} +JNIEXPORT jfloatArray JNICALL GENERATOR(audio)(JNIEnv* env, jclass, jlong h, jlong seq) +{ + std::vector samples; + try { samples = self(h)->plan->audio(uint64_t(seq)); } catch (const std::exception&) {} + jfloatArray out = env->NewFloatArray(jsize(samples.size())); + env->SetFloatArrayRegion(out, 0, jsize(samples.size()), samples.data()); + return out; +} +// The test picture for this event: AVS4 QR with the shown name and the sender calibration. +JNIEXPORT jintArray JNICALL GENERATOR(image)(JNIEnv* env, jclass, jlong h, jlong seq) +{ + auto* g = self(h); + avsync::TextEnv text(env); + try { + auto qr = g->plan->qr(uint64_t(seq)); + qr.name = g->info.source; qr.calibration = g->calibration; + auto info = g->info; + info.qr_json = avsync::qr_payload_text(qr); + info.counter = "Event " + std::to_string(seq); info.time_line = "Scheduled source output"; + avsync::Yuv422p10 frame; + frame.resize(1280, 720); + avsync::render_picture(frame, info, uint64_t(seq), 0, g->plan->period_frames(), false); + return pixels(env, frame); + } catch (const std::exception&) { return nullptr; } +} + +// ---- Calibration pattern (docs/calibrator.md display mode; core/include/avsync/calibrator.hpp) ---- + +JNIEXPORT jboolean JNICALL GENERATOR(calibrationLeftWhite)(JNIEnv*, jclass, jlong frame, jint fps) { return avsync::calibration_left_white(uint64_t(frame), fps, 1); } +JNIEXPORT jlong JNICALL GENERATOR(calibrationSwapFrame)(JNIEnv*, jclass, jlong swap, jint fps) { return jlong(avsync::calibration_swap_frame(uint64_t(swap), fps, 1)); } +JNIEXPORT jfloatArray JNICALL GENERATOR(calibrationBeep)(JNIEnv* env, jclass, jint rate) +{ + const auto beep = avsync::calibration_beep(rate); + jfloatArray out = env->NewFloatArray(jsize(beep.size())); + env->SetFloatArrayRegion(out, 0, jsize(beep.size()), beep.data()); + return out; +} +// Clicks and beeps in mono float samples: pairs, clicks and, with enough pairs, the result (JSON text). +JNIEXPORT jstring JNICALL GENERATOR(displayCalibration)(JNIEnv* env, jclass, jfloatArray samples, jint count, jint rate, jint minimumPairs) +{ + std::vector mono(size_t(std::max(0, std::min(count, samples ? env->GetArrayLength(samples) : 0)))); + if (!mono.empty()) env->GetFloatArrayRegion(samples, 0, jsize(mono.size()), mono.data()); + const auto sounds = avsync::find_sound_onsets(mono.data(), mono.size(), rate, 0); + avsync::DisplayCalibrationOptions options; + options.min_pairs = std::max(1, int(minimumPairs)); + const auto r = avsync::display_calibration(sounds, options); + long clicks = 0; + for (const auto& sound : sounds) clicks += sound.click(); + char text[512]; + if (!r.ok) { + std::string error; + for (char c : r.error) if (c != '"' && c != '\\') error += c; + std::snprintf(text, sizeof text, R"({"ok":false,"error":"%s","clicks":%ld,"sounds":%zu,"pairs":0})", error.c_str(), clicks, sounds.size()); + } else { + std::snprintf(text, sizeof text, + R"({"ok":true,"offsetMs":%.4f,"uncertaintyMs":%.4f,"spreadMs":%.4f,"pairs":%d,"rejected":%d,"directionMs":%.4f,"clicks":%ld,"sounds":%zu})", + r.offset_ms, r.uncertainty_ms, r.spread_ms, r.pairs, r.rejected, r.direction_ms, clicks, sounds.size()); + } + return env->NewStringUTF(text); +} + +} // extern "C" diff --git a/mobile-app/modules/avsync/android/src/main/cpp/text_android.cpp b/mobile-app/modules/avsync/android/src/main/cpp/text_android.cpp new file mode 100644 index 0000000..44db861 --- /dev/null +++ b/mobile-app/modules/avsync/android/src/main/cpp/text_android.cpp @@ -0,0 +1,87 @@ +// Text for the shared picture renderer on Android: Android's own text drawing (GeneratorText.kt) with +// the bundled DejaVu Sans, as iOS uses CoreText (ios/Mobile_text.mm) and the server Blend2D +// (core/src/text.cpp). Font size 8 × scale pixels on every target. +#include "text_android.hpp" + +#include "avsync/avsync.hpp" + +#include +#include +#include + +namespace avsync { +namespace { +thread_local JNIEnv* current = nullptr; +jclass text_class = nullptr; +jmethodID width_id = nullptr, height_id = nullptr, draw_id = nullptr; + +JNIEnv* env() +{ + if (!current) return nullptr; + static std::once_flag once; + std::call_once(once, [] { + jclass local = current->FindClass("io/leeo86/mxl/avsync/GeneratorText"); + if (!local) { current->ExceptionClear(); return; } + text_class = jclass(current->NewGlobalRef(local)); + current->DeleteLocalRef(local); + width_id = current->GetStaticMethodID(text_class, "width", "(Ljava/lang/String;F)I"); + height_id = current->GetStaticMethodID(text_class, "height", "(F)I"); + draw_id = current->GetStaticMethodID(text_class, "draw", "(Ljava/lang/String;FII)[B"); + if (current->ExceptionCheck()) { current->ExceptionClear(); text_class = nullptr; } + }); + return text_class ? current : nullptr; +} +float font_px(int scale) { return float(8 * std::max(1, scale)); } +jstring java_text(JNIEnv* e, std::string_view text) { return e->NewStringUTF(std::string(text).c_str()); } +} // namespace + +TextEnv::TextEnv(JNIEnv* env) : previous_(current) { current = env; } +TextEnv::~TextEnv() { current = previous_; } + +int text_px_width(std::string_view text, int scale) +{ + JNIEnv* e = env(); + if (text.empty() || !e) return 0; + jstring s = java_text(e, text); + const int width = e->CallStaticIntMethod(text_class, width_id, s, font_px(scale)); + e->DeleteLocalRef(s); + if (e->ExceptionCheck()) { e->ExceptionClear(); return 0; } + return std::max(0, width); +} + +int text_px_height(int scale) +{ + JNIEnv* e = env(); + if (!e) return int(font_px(scale)); + const int height = e->CallStaticIntMethod(text_class, height_id, font_px(scale)); + if (e->ExceptionCheck()) { e->ExceptionClear(); return int(font_px(scale)); } + return std::max(1, height); +} + +void draw_text(LumaView view, int x, int y, std::string_view text, int scale, uint16_t value) +{ + JNIEnv* e = env(); + if (text.empty() || !view.data || !e) return; + const int w = std::max(1, text_px_width(text, scale)), h = text_px_height(scale); + jstring s = java_text(e, text); + auto alpha = jbyteArray(e->CallStaticObjectMethod(text_class, draw_id, s, font_px(scale), w, h)); + e->DeleteLocalRef(s); + if (e->ExceptionCheck()) { e->ExceptionClear(); return; } + if (!alpha) return; + // Rows may be padded (Bitmap row bytes): the stride is the array length over the height. + const jsize length = e->GetArrayLength(alpha); + const int stride = length / h; + if (stride >= w) { + jbyte* coverage = e->GetByteArrayElements(alpha, nullptr); + for (int row = 0; row < h; ++row) + for (int col = 0; col < w; ++col) { + if (y + row < 0 || y + row >= view.height || x + col < 0 || x + col >= view.width) continue; + const unsigned c = uint8_t(coverage[size_t(row) * stride + col]); + auto& pixel = view.data[size_t(y + row) * view.stride + x + col]; + pixel = uint16_t((pixel * (255 - c) + value * c + 127) / 255); + } + e->ReleaseByteArrayElements(alpha, coverage, JNI_ABORT); + } + e->DeleteLocalRef(alpha); +} +} // namespace avsync diff --git a/mobile-app/modules/avsync/android/src/main/cpp/text_android.hpp b/mobile-app/modules/avsync/android/src/main/cpp/text_android.hpp new file mode 100644 index 0000000..bf0f176 --- /dev/null +++ b/mobile-app/modules/avsync/android/src/main/cpp/text_android.hpp @@ -0,0 +1,17 @@ +#pragma once +#include + +namespace avsync { +// The shared renderer's text on Android goes through JNI to GeneratorText.kt (text_android.cpp). +// Every JNI call that renders text sets the calling thread's JNIEnv with this guard first. +class TextEnv { +public: + explicit TextEnv(JNIEnv* env); + ~TextEnv(); + TextEnv(const TextEnv&) = delete; + TextEnv& operator=(const TextEnv&) = delete; + +private: + JNIEnv* previous_; +}; +} // namespace avsync diff --git a/mobile-app/modules/avsync/android/src/main/java/io/leeo86/mxl/avsync/AvsyncCameraView.kt b/mobile-app/modules/avsync/android/src/main/java/io/leeo86/mxl/avsync/AvsyncCameraView.kt index 49fb715..f608d19 100644 --- a/mobile-app/modules/avsync/android/src/main/java/io/leeo86/mxl/avsync/AvsyncCameraView.kt +++ b/mobile-app/modules/avsync/android/src/main/java/io/leeo86/mxl/avsync/AvsyncCameraView.kt @@ -87,6 +87,9 @@ class AvsyncCameraView(context: Context, appContext: AppContext) : ExpoView(cont private var lastQrScan = 0L private var lastHousekeeping = 0L private var calibrationReported = false + // Calibrator camera take: starts at the next lock (endFocus), which uses the lowest ISO (lowGain). + private var pendingCalibrator: String? = null + private var lowGain = false private var lastFrameMs = -1.0 private var maxFrameGapMs = 0.0 private var failedFrames = 0 @@ -125,9 +128,20 @@ class AvsyncCameraView(context: Context, appContext: AppContext) : ExpoView(cont } } fun beginCalibration() { outline.clear(); capture.post { generation++; focusing = false; calibrationReported = false; analyzer.beginCalibration() } } - // Restart: drop the history and run the auto step again, metered at the centre. - fun resumeAnalysis() { outline.clear(); capture.post { generation++; focusing = false; calibrationReported = false; analyzer.reset(); refocus(null, null) } } - fun focusAt(x: Double, y: Double) { capture.post { refocus(x, y) } } + // Calibrator board take ("camera" or "display", docs/calibrator.md). For the camera the LED floods + // the lens: run the auto step again and lock at the lowest ISO; the take starts at the lock. + fun beginCalibrator(mode: String) { + outline.clear() + capture.post { + generation++; focusing = false; calibrationReported = false; analyzer.reset() + if (mode != "camera") { pendingCalibrator = null; lowGain = false; analyzer.beginCalibrator(mode); return@post } + pendingCalibrator = mode; lowGain = true + refocus(null, null) + } + } + // Restart: drop the history and run the auto step again, metered at the centre (also cancels a calibrator take). + fun resumeAnalysis() { outline.clear(); capture.post { generation++; focusing = false; calibrationReported = false; pendingCalibrator = null; lowGain = false; analyzer.reset(); refocus(null, null) } } + fun focusAt(x: Double, y: Double) { capture.post { if (pendingCalibrator == null && !analyzer.calibratorActive()) refocus(x, y) } } override fun onAttachedToWindow() { super.onAttachedToWindow() @@ -314,7 +328,8 @@ class AvsyncCameraView(context: Context, appContext: AppContext) : ExpoView(cont val exposureRange = c.get(CameraCharacteristics.SENSOR_INFO_EXPOSURE_TIME_RANGE) val isoRange = c.get(CameraCharacteristics.SENSOR_INFO_SENSITIVITY_RANGE) if (CameraMetadata.REQUEST_AVAILABLE_CAPABILITIES_MANUAL_SENSOR in capabilities && exposure != null && iso != null && exposureRange != null && isoRange != null) { - val (fixed, gain) = CaptureProfile.fixedExposure(exposure, iso, frameNs, exposureRange.lower..exposureRange.upper, isoRange.lower..isoRange.upper) + val (fixed, auto) = CaptureProfile.fixedExposure(exposure, iso, frameNs, exposureRange.lower..exposureRange.upper, isoRange.lower..isoRange.upper) + val gain = if (lowGain) isoRange.lower else auto request.set(CaptureRequest.CONTROL_AE_MODE, CaptureRequest.CONTROL_AE_MODE_OFF) request.set(CaptureRequest.SENSOR_EXPOSURE_TIME, fixed) request.set(CaptureRequest.SENSOR_SENSITIVITY, gain) @@ -366,6 +381,8 @@ class AvsyncCameraView(context: Context, appContext: AppContext) : ExpoView(cont if ((c.get(CameraCharacteristics.CONTROL_MAX_REGIONS_AE) ?: 0) > 0) request.set(CaptureRequest.CONTROL_AE_REGIONS, region) request.set(CaptureRequest.CONTROL_AE_LOCK, false) request.set(CaptureRequest.CONTROL_AWB_LOCK, false) + // Cameras without manual exposure: the calibrator take meters as dark as allowed instead of the lowest ISO. + request.set(CaptureRequest.CONTROL_AE_EXPOSURE_COMPENSATION, if (lowGain) c.get(CameraCharacteristics.CONTROL_AE_COMPENSATION_RANGE)?.lower ?: 0 else 0) runCatching { if (s is CameraConstrainedHighSpeedCaptureSession) { // AF stays continuous-video in this mode: one cancel trigger releases the locked focus. @@ -386,6 +403,8 @@ class AvsyncCameraView(context: Context, appContext: AppContext) : ExpoView(cont private fun endFocus() { if (!focusing) return focusing = false; generation++; calibrationReported = false; analyzer.reset() + pendingCalibrator?.let { analyzer.beginCalibrator(it) } + pendingCalibrator = null; lowGain = false } private val results = object : CameraCaptureSession.CaptureCallback() { @@ -469,6 +488,13 @@ class AvsyncCameraView(context: Context, appContext: AppContext) : ExpoView(cont analyzer.setTaiUtcSeconds(taiUtcSeconds) val corners = analyzer.picture() main.post { outline.show(corners) } + // A calibrator take reports about once a second (as iOS: otherwise the live poll goes on). + analyzer.pollCalibrator()?.let { text -> + val report = toMap(JSONObject(text)) + if (report["kind"] != "calibrator-progress") analyzer.reset() // done: live analysis resumes + emit(report) + return + } if (analyzer.calibrating()) { if (calibrationReported) return analyzer.takeCalibration()?.let { take -> diff --git a/mobile-app/modules/avsync/android/src/main/java/io/leeo86/mxl/avsync/AvsyncGenerator.kt b/mobile-app/modules/avsync/android/src/main/java/io/leeo86/mxl/avsync/AvsyncGenerator.kt new file mode 100644 index 0000000..fdb5f62 --- /dev/null +++ b/mobile-app/modules/avsync/android/src/main/java/io/leeo86/mxl/avsync/AvsyncGenerator.kt @@ -0,0 +1,515 @@ +package io.leeo86.mxl.avsync + +import android.Manifest +import android.app.Activity +import android.app.Dialog +import android.content.Context +import android.content.pm.PackageManager +import android.graphics.Bitmap +import android.graphics.BitmapFactory +import android.graphics.Canvas +import android.graphics.Color +import android.graphics.Paint +import android.media.AudioManager +import android.os.Build +import android.os.SystemClock +import android.util.TypedValue +import android.view.Choreographer +import android.view.Gravity +import android.view.View +import android.view.ViewGroup.LayoutParams.MATCH_PARENT +import android.view.ViewGroup.LayoutParams.WRAP_CONTENT +import android.view.WindowManager +import android.widget.Button +import android.widget.FrameLayout +import android.widget.LinearLayout +import android.widget.TextView +import androidx.annotation.RequiresApi +import org.json.JSONObject +import java.util.UUID +import java.util.concurrent.Executors +import kotlin.math.abs +import kotlin.math.roundToLong + +// The fullscreen test signal (docs/phone-generator.md), like ios/AvsyncGeneratorController.swift. +// Output scheduling stays native; JS only opens this screen and gets the result when it closes. +// Frames: Choreographer gives each frame's expected presentation time (System.nanoTime, the clock of +// the audio timestamps too); the 60 Hz plan says what that frame shows. Sound: GeneratorAudio. +// With `calibrate` it shows the calibration pattern instead (docs/calibrator.md, display mode): the +// same frame clock, audio output and scheduling, plus the microphone. +class AvsyncGenerator(private val activity: Activity, private val options: Options, private val onClose: (Map) -> Unit) : + Dialog(activity, android.R.style.Theme_Black_NoTitleBar_Fullscreen) { + + class Options(val rate: Int, val taiUtc: Double, val name: String, val subtitle: String, val image: Bitmap?, + val clockReference: Map, val keepScreenAwake: Boolean, val senderCalibration: Map, val calibrate: Boolean) + + private val rate = options.rate + private val calibrate = options.calibrate + private val raster = Raster(context) + private val pattern = CalibrationPattern(context) + private val status = TextView(context) + private val stoppedLabel = TextView(context) // big "Stopped" over the picture while no signal runs + private val toggle = Button(context) + private val close = Button(context) + private val audio = GeneratorAudio { message -> stop(message) } + private val audioManager = context.getSystemService(Context.AUDIO_SERVICE) as AudioManager + private val choreographer = Choreographer.getInstance() + private val imageWorker = Executors.newSingleThreadExecutor { Thread(it, "avsync.generator.images") } + private var ticking = false + private var session = 0 // one frame-callback chain per start; a callback of an older start ends its chain + private var finishing = false + private var plan: PhoneGenerator? = null + private var epochHost = -1.0 // host seconds (System.nanoTime) of frame 0; -1 while no session runs + private var startedNs = 0L + private var vsyncNs = 16_666_667L + private var nextAudio = 1L + private var shown = 0L + private val images = HashMap() + private var generation = 0 + private var outputLatency = 0.0 + private var lateFrames = 0 + private var lastFrame: Long? = null + private var sourceID = "" + private var lastStatus = 0.0 + private var utcMinusHost = 0.0 + private var usingReference = false + private var senderCalibrationNote = "" + // Calibration pattern: the microphone take. + private var recorder: AudioCapture? = null + private val recordLock = Any() + private var recorded = FloatArray(0) // recordLock + private var recordedCount = 0 // recordLock + private var nextBeep = 1L + private var analysing = false + private var lastAnalysis = 0.0 + private val analysisWorker = Executors.newSingleThreadExecutor { Thread(it, "avsync.generator.analysis") } + private var calibrationResult: Map? = null + private var calibrationError: String? = null + + init { + GeneratorText.load(context) + setCancelable(true) + setOnCancelListener { closeScreen() } + setOnDismissListener { if (!finishing) closeScreen() } + val root = FrameLayout(context).apply { setBackgroundColor(Color.BLACK) } + val content = LinearLayout(context).apply { orientation = LinearLayout.VERTICAL } + status.setTextColor(Color.LTGRAY); status.setTextSize(TypedValue.COMPLEX_UNIT_SP, 12f); status.gravity = Gravity.CENTER; status.setLines(3) + status.contentDescription = "generatorStatus" + stoppedLabel.text = "Stopped"; stoppedLabel.setTextColor(Color.WHITE); stoppedLabel.setTextSize(TypedValue.COMPLEX_UNIT_SP, 34f) + stoppedLabel.paint.isFakeBoldText = true; stoppedLabel.gravity = Gravity.CENTER; stoppedLabel.visibility = View.GONE + toggle.text = if (calibrate) "Stop" else "Stop signal"; toggle.setOnClickListener { if (ticking) stop("Signal stopped. Tap Start for a fresh source session.") else start() } + close.text = "Close"; close.setOnClickListener { closeScreen() } + val buttons = LinearLayout(context).apply { + orientation = LinearLayout.HORIZONTAL + addView(toggle, LinearLayout.LayoutParams(0, WRAP_CONTENT, 1f)); addView(close, LinearLayout.LayoutParams(0, WRAP_CONTENT, 1f)) + } + val pad = dp(12) + if (calibrate) { + content.addView(buttons, LinearLayout.LayoutParams(MATCH_PARENT, WRAP_CONTENT).apply { setMargins(pad, pad, pad, 0) }) + content.addView(status, LinearLayout.LayoutParams(MATCH_PARENT, WRAP_CONTENT).apply { setMargins(pad, 0, pad, 0) }) + content.addView(pattern, LinearLayout.LayoutParams(MATCH_PARENT, 0, 1f)) + } else { + // Keep capture screens portrait. Only this test raster rotates; holding the phone sideways + // gives a large landscape target without changing camera UI. + content.addView(raster, LinearLayout.LayoutParams(MATCH_PARENT, 0, 1f).apply { setMargins(pad, pad, pad, 0) }) + content.addView(status, LinearLayout.LayoutParams(MATCH_PARENT, WRAP_CONTENT).apply { setMargins(pad, 0, pad, 0) }) + content.addView(buttons, LinearLayout.LayoutParams(MATCH_PARENT, WRAP_CONTENT).apply { setMargins(pad, 0, pad, pad) }) + } + root.addView(content, FrameLayout.LayoutParams(MATCH_PARENT, MATCH_PARENT)) + root.addView(stoppedLabel, FrameLayout.LayoutParams(MATCH_PARENT, WRAP_CONTENT, Gravity.CENTER)) + setContentView(root) + window?.let { w -> + w.setLayout(MATCH_PARENT, MATCH_PARENT) + @Suppress("DEPRECATION") + w.decorView.systemUiVisibility = View.SYSTEM_UI_FLAG_FULLSCREEN or View.SYSTEM_UI_FLAG_HIDE_NAVIGATION or View.SYSTEM_UI_FLAG_IMMERSIVE_STICKY or + View.SYSTEM_UI_FLAG_LAYOUT_STABLE or View.SYSTEM_UI_FLAG_LAYOUT_FULLSCREEN or View.SYSTEM_UI_FLAG_LAYOUT_HIDE_NAVIGATION + // A 60 Hz display mode at the current resolution when the display has one (the plan's frame grid). + @Suppress("DEPRECATION") val display = activity.windowManager.defaultDisplay + val current = display.mode + display.supportedModes.filter { it.physicalWidth == current.physicalWidth && it.physicalHeight == current.physicalHeight } + .minByOrNull { abs(it.refreshRate - 60f) }?.let { mode -> w.attributes = w.attributes.apply { preferredDisplayModeId = mode.modeId } } + } + setOnShowListener { start() } + } + + private fun dp(value: Int) = TypedValue.applyDimension(TypedValue.COMPLEX_UNIT_DIP, value.toFloat(), context.resources.displayMetrics).toInt() + private fun nowSeconds() = System.nanoTime() / 1e9 + + // ---- Start, stop, close ---- + + @Suppress("DEPRECATION") // requestAudioFocus(listener, stream, duration): the only form on API 24–25. + private fun start() { + if (ticking || finishing) return + stoppedLabel.visibility = View.GONE + try { + audioManager.requestAudioFocus(focus, AudioManager.STREAM_MUSIC, AudioManager.AUDIOFOCUS_GAIN_TRANSIENT) + if (calibrate) startRecording() + audio.start { if (plan != null || epochHost >= 0) stop("Audio output changed. Tap Start to re-plan timing.") } + plan = null; epochHost = -1.0; startedNs = System.nanoTime() + @Suppress("DEPRECATION") val display = activity.windowManager.defaultDisplay + vsyncNs = (1e9 / display.refreshRate.coerceIn(24f, 240f)).toLong() + window?.let { w -> + w.attributes = w.attributes.apply { screenBrightness = 1f } + if (options.keepScreenAwake) w.addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON) + } + ticking = true; session++; postTick(session) + toggle.text = if (calibrate) "Stop" else "Stop signal" + status.text = if (calibrate) "Lay the calibrator on the red outline, sensors over the line. Keep the volume up." + else "Turn phone sideways · audible test signal starts shortly\n1200 baud · $rate/s · estimated source clock / output timing" + } catch (e: Exception) { stop("Cannot start: ${e.message}") } + } + + @Suppress("DEPRECATION") + private val focus = AudioManager.OnAudioFocusChangeListener { change -> + if (change == AudioManager.AUDIOFOCUS_LOSS || change == AudioManager.AUDIOFOCUS_LOSS_TRANSIENT) stop("Audio interrupted. Tap Start to restart.") + } + + @Suppress("DEPRECATION") + fun stop(message: String) { + ticking = false; generation++ + recorder?.stop(); recorder = null + audio.stop() + plan?.let { old -> imageWorker.execute { old.close() } } // after the renders already queued + plan = null; epochHost = -1.0; images.clear() + // Clear the whole raster (scale and marker too), so no stale test picture stays on screen. + raster.clear() + stoppedLabel.visibility = View.VISIBLE + window?.let { w -> + w.attributes = w.attributes.apply { screenBrightness = WindowManager.LayoutParams.BRIGHTNESS_OVERRIDE_NONE } + w.clearFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON) + } + audioManager.abandonAudioFocus(focus) + status.text = message; toggle.text = if (calibrate) "Start" else "Start signal" + } + + // The app went to the background (module lifecycle). + fun pause() { if (ticking) stop("Paused: app is inactive. Tap Start to create a fresh source session.") } + fun shutdown() = closeScreen() + + private fun closeScreen() { + if (finishing) return + finishing = true + val result = mutableMapOf("source" to sourceID, "eventRate" to rate, "skippedDisplayFrames" to lateFrames, + "audioPresentationLatencyMs" to outputLatency * 1000, "estimated" to true, "clockReference" to options.clockReference, "clockApplied" to usingReference) + calibrationResult?.let { result["calibration"] = it } + calibrationError?.let { result["calibrationError"] = it } + stop("Closed") + imageWorker.shutdown(); analysisWorker.shutdown() + dismiss() + onClose(result) + } + + // ---- Frame clock ---- + + // Before Android 13 there is no expected presentation time: a frame started at vsync N is shown about two vsyncs later. + private fun postTick(chain: Int) { + if (Build.VERSION.SDK_INT >= 33) Vsync33.post(choreographer) { tick(it, chain) } + else choreographer.postFrameCallback { frameNs -> tick(frameNs + 2 * vsyncNs, chain) } + } + + @RequiresApi(33) + private object Vsync33 { + fun post(choreographer: Choreographer, tick: (Long) -> Unit) = + choreographer.postVsyncCallback { data -> tick(data.preferredFrameTimeline.expectedPresentationTimeNanos) } + } + + // targetNs: when the frame drawn now is expected on screen (System.nanoTime). + private fun tick(targetNs: Long, chain: Int) { + if (!ticking || chain != session) return + postTick(chain) + val target = targetNs / 1e9 + if (calibrate) return calibrationTick(target) + if (plan == null) { + // The plan starts once the output reports its timing (iOS: outputPresentationLatency). + val lead = audio.latencyNs + if (lead !in 0 until 1_000_000_000L) { + if (System.nanoTime() - startedNs > 3_000_000_000L) stop("Audio output latency is unavailable.") + return + } + try { beginPlan(target) } catch (e: Exception) { return stop(e.message ?: "Cannot start the signal.") } + } + val plan = plan ?: return + val now = nowSeconds() + if (now - epochHost >= 23 * 3600) return stop("Session complete. Tap Start for a new source.") + try { fillAhead(now) } catch (e: Exception) { return stop(e.message ?: "Scheduling deadline missed. Restart the signal.") } + if (target < epochHost) return + val frame = plan.frame(target) + val current = (frame.sequence - 1) * (60 / rate) + frame.position + lastFrame?.let { previous -> if (current > previous + 1) lateFrames += (current - previous - 1).toInt() } + lastFrame = current + // The QR names the next event from half a period before its burst. + if (frame.named != shown) { + val image = images[frame.named] ?: return stop("Picture scheduling deadline missed. Restart the signal.") + shown = frame.named; raster.plate = image + } + raster.show(frame) + if (now - lastStatus > 0.5) { + lastStatus = now + if (referenceExpired()) return stop("Clock reference expired. Close and synchronize again.") + if (abs(System.currentTimeMillis() / 1000.0 - now - utcMinusHost) > 0.1) return stop("Phone clock changed. Tap Start to create a fresh clock mapping.") + outputLatency = audio.latencyNs / 1e9 + status.text = "TEST SIGNAL ACTIVE · 1200 baud · $rate/s · frame skips $lateFrames\nEstimated output timing · ${if (usingReference) "NTP UTC" else "device UTC"} · " + + "audio lead ${Math.round(outputLatency * 1000)} ms$senderCalibrationNote" + } + } + + private fun beginPlan(target: Double) { + sourceID = stableSourceID(context, rate) + // UTC and host time are sampled as a bracket. Keep one mapping for the whole session; stop if the + // wall clock later jumps instead of rewriting it. + val before = System.nanoTime(); val utc = System.currentTimeMillis() / 1000.0; val after = System.nanoTime() + val bootMinusHost = (SystemClock.elapsedRealtimeNanos() - System.nanoTime()) / 1e9 + epochHost = target + 1.5 + utcMinusHost = utc - (before + after) / 2e9 + var referenceUtcMinusHost = utcMinusHost + usingReference = false + // The clock reference is on PhoneClock (elapsedRealtime); the host clock here is System.nanoTime. + val r = options.clockReference + val unixMs = r["clockUnixMs"]?.toDoubleOrNull(); val bootMs = r["clockMonotonicMs"]?.toDoubleOrNull() + val nowBootMs = before / 1e6 + bootMinusHost * 1000 + if (r["clockId"] == PhoneClock.identifier && unixMs != null && bootMs != null && unixMs in 1e12..4e12 && bootMs.isFinite() && + nowBootMs - bootMs >= 0 && nowBootMs - bootMs < 300_000) { + referenceUtcMinusHost = (unixMs - bootMs) / 1000 + bootMinusHost + usingReference = true + } + val epochTai = ((referenceUtcMinusHost + options.taiUtc + epochHost) * 1e9).roundToLong() + val next = PhoneGenerator(sourceID, rate, epochTai, epochHost, options.name.ifEmpty { "Android" }) + plan = next + next.setSource(options.name, options.subtitle, options.image) + appIcon(context)?.let { next.setIcon(it) } + applySenderCalibration(next) + raster.foreground = next.syncForeground() + raster.geometry = next.geometry() + outputLatency = audio.latencyNs / 1e9 + nextAudio = 1; shown = 0; lateFrames = 0; lastFrame = null; lastStatus = 0.0; images.clear() + generation++ + val first = next.image(1); images[1] = first + raster.plate = first + fillAhead(nowSeconds()) + } + + // Sound packets and pictures 2.5 s ahead. A packet must still be ahead of the output's latency. + private fun fillAhead(now: Double) { + val plan = plan ?: return + while (plan.hostTime(nextAudio) < now + 2.5) { + val host = plan.hostTime(nextAudio) + if (host <= now + audio.latencyNs / 1e9 + 0.03) throw IllegalStateException("Scheduling deadline missed. Restart the signal.") + audio.schedule((host * 1e9).roundToLong(), plan.audio(nextAudio)) + if (images[nextAudio] == null) { + val sequence = nextAudio; val ticket = generation + imageWorker.execute { + val image = runCatching { plan.image(sequence) }.getOrNull() ?: return@execute + raster.post { if (generation == ticket && this.plan === plan) images[sequence] = image } + } + } + nextAudio++ + } + images.keys.removeAll { it + 1 < shown } + } + + private fun referenceExpired(): Boolean { + if (!usingReference) return false + val r = options.clockReference + val anchor = r["clockMonotonicMs"]?.toDoubleOrNull() ?: return false + val radius = r["clockUncertaintyMs"]?.toDoubleOrNull() ?: return false + val limit = r["clockLimitMs"]?.toDoubleOrNull() ?: return false + val age = PhoneClock.nowMs() - anchor + return age > 300_000 || radius + age * 50 / 1e6 > limit + } + + // ---- Sender calibration and calibration pattern ---- + + // The saved screen-vs-speaker calibration goes into the AVS4 QR, but only on the audio output it was measured on. + private fun applySenderCalibration(plan: PhoneGenerator) { + senderCalibrationNote = "" + val c = options.senderCalibration + val method = c["senderCalibrationMethod"] ?: return + val offset = c["senderCalibrationOffsetMs"]?.toDoubleOrNull() ?: return + val uncertainty = c["senderCalibrationUncertaintyMs"]?.toDoubleOrNull() ?: return + val day = c["senderCalibrationDay"]?.toIntOrNull() ?: return + val route = c["senderCalibrationRoute"] + if (route != null && route != audio.route) { senderCalibrationNote = " · sender calibration not sent (other audio output)"; return } + plan.setSenderCalibration(method, offset, uncertainty, day) + senderCalibrationNote = String.format(java.util.Locale.ROOT, " · sender calibration %+.1f ms", offset) + } + + private fun startRecording() { + if (context.checkSelfPermission(Manifest.permission.RECORD_AUDIO) != PackageManager.PERMISSION_GRANTED) + throw IllegalStateException("The microphone is not available. Allow microphone access in Settings.") + synchronized(recordLock) { recorded = FloatArray(MIC_RATE * (CALIBRATION_LIMIT_S + 5)); recordedCount = 0 } + val microphone = AudioCapture(audioManager) { samples, n, _ -> + synchronized(recordLock) { + val take = minOf(n, recorded.size - recordedCount) + System.arraycopy(samples, 0, recorded, recordedCount, take); recordedCount += take + } + } + microphone.start(); recorder = microphone + } + + private fun swapHost(k: Long) = epochHost + PhoneGenerator.calibrationSwapFrame(k, 60) / 60.0 + + private fun calibrationTick(target: Double) { + val now = nowSeconds() + if (epochHost < 0) { + if (audio.latencyNs !in 0 until 1_000_000_000L) { + if (System.nanoTime() - startedNs > 3_000_000_000L) stop("Audio output latency is unavailable.") + return + } + epochHost = target + 1.0 + nextBeep = 1; lateFrames = 0; lastFrame = null; lastAnalysis = now; calibrationResult = null; calibrationError = null + outputLatency = audio.latencyNs / 1e9 + } + // The beep: D = 100 ms after each swap on the same host clock (GeneratorAudio places it on the output's timestamps). + while (swapHost(nextBeep) + 0.1 < now + 2.5) { + val host = swapHost(nextBeep) + 0.1 + if (host <= now + audio.latencyNs / 1e9 + 0.03) return stop("Scheduling deadline missed. Tap Start to try again.") + audio.schedule((host * 1e9).roundToLong(), BEEP) + nextBeep++ + } + val position = (target - epochHost) * 60 + val frame = maxOf(0L, Math.round(position)) + if (position > -0.5) { + lastFrame?.let { previous -> if (frame > previous + 1) lateFrames += (frame - previous - 1).toInt() } + lastFrame = frame + } + pattern.leftWhite = position < -0.5 || PhoneGenerator.calibrationLeftWhite(frame, 60) + if (analysing || now - lastAnalysis < 1 || now <= epochHost + 1.5) return + lastAnalysis = now; analysing = true + val running = now - epochHost; val skips = lateFrames; val latency = audio.latencyNs / 1e9; val route = audio.route; val ticket = generation + analysisWorker.execute { + val (samples, count) = synchronized(recordLock) { recorded.copyOf(recordedCount) to recordedCount } + val progress = JSONObject(PhoneGenerator.displayCalibration(samples, count, MIC_RATE, 1)) + val pairs = progress.optInt("pairs"); val clicks = progress.optInt("clicks") + val done = pairs >= WANTED_PAIRS || running >= CALIBRATION_LIMIT_S + val final = if (done) JSONObject(PhoneGenerator.displayCalibration(samples, count, MIC_RATE, 8)) else progress + pattern.post { + analysing = false + if (!ticking || generation != ticket) return@post + if (!done) { + status.text = if (clicks == 0 && running > 3) "No click yet. Lay the calibrator on the red outline, sensors over the line, and check the cable." + else "Measuring · $pairs of $WANTED_PAIRS swaps · frame skips $skips" + return@post + } + if (final.optBoolean("ok")) { + calibrationResult = AvsyncCameraView.toMap(final).filterValues { it != null }.mapValues { it.value!! } + + mapOf("method" to "calibrator-display", "audioRoute" to route, "outputLatencyMs" to latency * 1000, "skippedDisplayFrames" to skips, + "sampleRate" to MIC_RATE.toDouble()) + closeScreen() + } else { + calibrationError = final.optString("error").ifEmpty { null } + stop(calibrationError ?: "Calibration failed. Tap Start to try again.") + } + } + } + } + + // ---- Views ---- + + // The 1280 × 720 test raster: the event's picture, the AVS3 fields A and B, the sync square and + // the scale's white foreground, scaled (rotated in portrait) into the view, nearest-neighbour. + private class Raster(context: Context) : View(context) { + var plate: Bitmap? = null + set(value) { field = value; invalidate() } + var foreground: Bitmap? = null + var geometry: IntArray? = null // markerX, markerY, markerSide + private var frame: PhoneGenerator.Frame? = null + private val paint = Paint().apply { isFilterBitmap = false; isAntiAlias = false } + fun show(value: PhoneGenerator.Frame) { frame = value; invalidate() } + fun clear() { plate = null; foreground = null; frame = null; invalidate() } + override fun onDraw(canvas: Canvas) { + val fit = GeneratorLayout.fitRaster(width.toFloat(), height.toFloat()) + canvas.save() + canvas.translate(width / 2f, height / 2f) + if (fit.rotate) canvas.rotate(90f) + canvas.scale(fit.scale, fit.scale) + canvas.translate(-PhoneGenerator.WIDTH / 2f, -PhoneGenerator.HEIGHT / 2f) + plate?.let { canvas.drawBitmap(it, 0f, 0f, paint) } + geometry?.let { g -> + // AVS3 optical code: field A (left half of the panel) and B (right half), always opposite; both black when stopped. + val half = (g[2] / 2).toFloat(); val x = g[0].toFloat(); val y = g[1].toFloat(); val side = g[2].toFloat() + val aWhite = frame?.aWhite == true + paint.color = if (frame != null && aWhite) Color.WHITE else Color.BLACK + canvas.drawRect(x, y, x + half, y + side, paint) + paint.color = if (frame != null && !aWhite) Color.WHITE else Color.BLACK + canvas.drawRect(x + half, y, x + side, y + side, paint) + } + frame?.let { f -> + paint.color = f.syncColor + canvas.drawRect(f.syncX, f.syncY, f.syncX + f.syncSize, f.syncY + f.syncSize, paint) + paint.color = Color.BLACK + val line = 2f; val cx = f.syncX + (f.syncSize - line) / 2 + canvas.drawRect(cx, f.syncY, cx + line, f.syncY + f.syncSize, paint) + } + foreground?.let { canvas.drawBitmap(it, 0f, 0f, paint) } + canvas.restore() + } + } + + // Portrait, halves left and right (the pattern line crosses the display rows), near the microphone + // end at the bottom. The outline is the 66 × 22 mm board, sensor end down, with a tick at the notch. + private class CalibrationPattern(context: Context) : View(context) { + var leftWhite = true + set(value) { if (field != value) { field = value; invalidate() } } + private val fill = Paint() + private val outline = Paint(Paint.ANTI_ALIAS_FLAG).apply { style = Paint.Style.STROKE; color = Color.rgb(255, 59, 48); strokeWidth = 2 * context.resources.displayMetrics.density } + private val mm = GeneratorLayout.pixelsPerMm(context.resources.displayMetrics.xdpi, context.resources.displayMetrics.densityDpi.toFloat()) + override fun onDraw(canvas: Canvas) { + val l = GeneratorLayout.calibration(width.toFloat(), height.toFloat(), mm) + fill.color = if (leftWhite) Color.WHITE else Color.BLACK + canvas.drawRect(0f, l.top, width / 2f, height.toFloat(), fill) + fill.color = if (leftWhite) Color.BLACK else Color.WHITE + canvas.drawRect(width / 2f, l.top, width.toFloat(), height.toFloat(), fill) + canvas.drawRoundRect(l.boardLeft, l.boardTop, l.boardRight, l.boardBottom, 1.5f * mm, 1.5f * mm, outline) + canvas.drawLine(width / 2f - 1.5f * mm, l.notchTick, width / 2f + 1.5f * mm, l.notchTick, outline) + } + } + + companion object { + private const val WANTED_PAIRS = 20 + private const val CALIBRATION_LIMIT_S = 30 + private const val MIC_RATE = 48_000 // AudioCapture records at 48 kHz + private val BEEP by lazy { PhoneGenerator.calibrationBeep(GeneratorAudio.RATE) } + + // Same install, app version (with build number) and timing settings give the same ID, so a restart keeps it. + // The install ID is random, kept in the app's preferences and gone after a reinstall. + // Name, subtitle and image are cosmetic and stay out. Add a new timing setting to `settings`. + fun stableSourceID(context: Context, rate: Int): String { + val preferences = context.getSharedPreferences("avsync", Context.MODE_PRIVATE) + val install = preferences.getString("generator.installId", null) ?: UUID.randomUUID().toString().also { + preferences.edit().putString("generator.installId", it).apply() + } + return PhoneGenerator.testId("phone-", install, appVersion(context), arrayOf("rate=$rate")) + } + + fun appVersion(context: Context): String { + val info = context.packageManager.getPackageInfo(context.packageName, 0) + @Suppress("DEPRECATION") val code = if (Build.VERSION.SDK_INT >= 28) info.longVersionCode else info.versionCode.toLong() + return "${info.versionName ?: ""} ($code)" + } + + private fun appIcon(context: Context): Bitmap? = runCatching { context.assets.open("icon.png").use { BitmapFactory.decodeStream(it) } }.getOrNull() + } +} + +// Pure layout of the generator screens (JVM tests). +object GeneratorLayout { + class Fit(val rotate: Boolean, val scale: Float) + // The 1280 × 720 raster in a view: rotated by 90° when the view is portrait, as large as fits. + fun fitRaster(width: Float, height: Float): Fit { + val portrait = height > width + val long = if (portrait) height else width; val short = if (portrait) width else height + return Fit(portrait, minOf(long / PhoneGenerator.WIDTH, short / PhoneGenerator.HEIGHT)) + } + + // xdpi is the physical density; a few phones report nonsense there, then the density bucket stands in. + fun pixelsPerMm(xdpi: Float, densityDpi: Float) = (if (xdpi in 100f..1000f) xdpi else densityDpi) / 25.4f + + data class Calibration(val top: Float, val boardLeft: Float, val boardTop: Float, val boardRight: Float, val boardBottom: Float, val notchTick: Float) + // The halves fill the bottom 62 mm (or the whole view when shorter); the board's sensors (5 mm from + // the notch end) sit at the middle of the halves. + fun calibration(width: Float, height: Float, mm: Float): Calibration { + val top = maxOf(0f, height - 62 * mm) + val notch = (top + height) / 2 + 5 * mm + return Calibration(top, width / 2 - 11 * mm, notch - 66 * mm, width / 2 + 11 * mm, notch, notch + 2 * mm) + } +} diff --git a/mobile-app/modules/avsync/android/src/main/java/io/leeo86/mxl/avsync/AvsyncModule.kt b/mobile-app/modules/avsync/android/src/main/java/io/leeo86/mxl/avsync/AvsyncModule.kt index 195106c..10ba4c0 100644 --- a/mobile-app/modules/avsync/android/src/main/java/io/leeo86/mxl/avsync/AvsyncModule.kt +++ b/mobile-app/modules/avsync/android/src/main/java/io/leeo86/mxl/avsync/AvsyncModule.kt @@ -1,7 +1,9 @@ package io.leeo86.mxl.avsync import android.Manifest +import android.app.Activity import android.content.Context +import android.content.Intent import android.content.pm.PackageManager import android.location.GnssMeasurementsEvent import android.location.Location @@ -12,7 +14,9 @@ import android.os.Bundle import android.os.Handler import android.os.Looper import android.os.SystemClock +import android.provider.MediaStore import expo.modules.kotlin.Promise +import expo.modules.kotlin.functions.Queues import expo.modules.kotlin.modules.Module import expo.modules.kotlin.modules.ModuleDefinition import android.view.WindowManager @@ -21,13 +25,49 @@ class AvsyncModule : Module() { private val handler = Handler(Looper.getMainLooper()) private var cancel: (() -> Unit)? = null private var ntpBusy = false + private var generator: AvsyncGenerator? = null + private var imagePick: Promise? = null + private val context get() = appContext.reactContext ?: throw IllegalStateException("The app is not ready.") override fun definition() = ModuleDefinition { Name("Avsync") Function("supportsLiveCapture") { true } - Function("supportsGenerator") { false } // generate() comes with the Android generator. Function("deviceVersion") { Build.VERSION.RELEASE } Function("loadState") { appContext.reactContext?.getSharedPreferences("avsync", Context.MODE_PRIVATE)?.getString("session.v1", null) } - Function("saveState") { value: String -> appContext.reactContext?.getSharedPreferences("avsync", Context.MODE_PRIVATE)?.edit()?.putString("session.v1", value)?.apply(); Unit } + Function("saveState") { value: String -> + val preferences = appContext.reactContext?.getSharedPreferences("avsync", Context.MODE_PRIVATE) + val old = imageID(preferences?.getString("session.v1", null)); val next = imageID(value) + preferences?.edit()?.putString("session.v1", value)?.apply() + if (old != null && old != next) GeneratorImageStore.remove(context, old) + Unit + } + + // The fullscreen test signal (AvsyncGenerator); the promise resolves when it closes. + Function("supportsGenerator") { true } + AsyncFunction("generate") { rate: Int, taiUtc: Double, promise: Promise -> presentGenerator(rate, taiUtc, emptyMap(), promise) }.runOnQueue(Queues.MAIN) + AsyncFunction("generateStyled") { rate: Int, taiUtc: Double, options: Map, promise: Promise -> + presentGenerator(rate, taiUtc, options, promise) + }.runOnQueue(Queues.MAIN) + Function("generatorImageInfo") { id: String -> GeneratorImageStore.info(context, id) } + // The system photo picker (Android 13+) or document picker: only the chosen image is shared, no storage permission. + AsyncFunction("pickGeneratorImage") { promise: Promise -> + val activity = appContext.currentActivity + if (imagePick != null || activity == null || generator != null) { promise.reject("IMAGE_PICKER_BUSY", "Close the current screen first.", null); return@AsyncFunction } + val intent = if (Build.VERSION.SDK_INT >= 33) Intent(MediaStore.ACTION_PICK_IMAGES).setType("image/*") + else Intent(Intent.ACTION_GET_CONTENT).setType("image/*").addCategory(Intent.CATEGORY_OPENABLE) + imagePick = promise + try { activity.startActivityForResult(intent, PICK_IMAGE) } + catch (e: Exception) { imagePick = null; promise.reject("IMAGE_IMPORT_FAILED", e.message ?: "No image picker on this phone.", e) } + }.runOnQueue(Queues.MAIN) + OnActivityResult { _, result -> + if (result.requestCode != PICK_IMAGE) return@OnActivityResult + val promise = imagePick ?: return@OnActivityResult + imagePick = null + val uri = result.data?.data + if (result.resultCode != Activity.RESULT_OK || uri == null) { promise.resolve(null); return@OnActivityResult } + Thread({ + runCatching { GeneratorImageStore.save(context, uri) }.fold({ promise.resolve(it) }, { promise.reject("IMAGE_IMPORT_FAILED", it.message ?: "Image import failed.", it) }) + }, "avsync.image").start() + } Function("clockNow") { PhoneClock.now() } AsyncFunction("synchronizeClock") { server: String, promise: Promise -> if (ntpBusy) { promise.reject("CLOCK_BUSY", "A clock check is already running", null); return@AsyncFunction } @@ -55,13 +95,39 @@ class AvsyncModule : Module() { Prop("captureFps") { view: AvsyncCameraView, fps: Double -> view.setRequestedFps(fps) } Prop("taiUtcOffset") { view: AvsyncCameraView, seconds: Double -> view.setTaiUtcOffset(seconds) } AsyncFunction("beginCalibration") { view: AvsyncCameraView -> view.beginCalibration() } + AsyncFunction("beginCalibrator") { view: AvsyncCameraView, mode: String -> view.beginCalibrator(mode) } AsyncFunction("resumeAnalysis") { view: AvsyncCameraView -> view.resumeAnalysis() } AsyncFunction("focusAt") { view: AvsyncCameraView, x: Double, y: Double -> view.focusAt(x, y) } } OnActivityEntersForeground { AvsyncCameraView.setForeground(true) } - OnActivityEntersBackground { AvsyncCameraView.setForeground(false); handler.post { cancel?.invoke() } } - OnDestroy { handler.post { cancel?.invoke() } } + OnActivityEntersBackground { AvsyncCameraView.setForeground(false); handler.post { cancel?.invoke(); generator?.pause() } } + OnDestroy { handler.post { cancel?.invoke(); generator?.shutdown() } } + } + + private fun imageID(json: String?): String? = runCatching { org.json.JSONObject(json ?: return null).optString("generatorImageId").ifEmpty { null } }.getOrNull() + + private fun presentGenerator(rate: Int, taiUtc: Double, options: Map, promise: Promise) { + val activity = appContext.currentActivity + if ((rate != 1 && rate != 2) || !taiUtc.isFinite() || taiUtc < 0 || taiUtc > 100 || activity == null || generator != null || imagePick != null) { + promise.reject("GENERATOR_UNAVAILABLE", "Close the current screen and use 1 or 2 events/s.", null); return + } + // Whitespace runs become one space; at most `limit` characters (code points), as the app sends them. + fun label(key: String, limit: Int): String { + val text = (options[key] ?: "").split(Regex("\\s+")).filter { it.isNotEmpty() }.joinToString(" ") + return text.substring(0, text.offsetByCodePoints(0, minOf(limit, text.codePointCount(0, text.length)))) + } + try { + val image = GeneratorImageStore.load(context, options["imageId"] ?: "") + // mode "calibration": the calibration pattern for the calibrator board instead of the test signal. + val screen = AvsyncGenerator(activity, AvsyncGenerator.Options(rate, taiUtc, label("name", 64), label("subtitle", 128), image, + options.filterKeys { it.startsWith("clock") }, options["keepAwake"] != "false", options.filterKeys { it.startsWith("senderCalibration") }, + options["mode"] == "calibration")) { result -> generator = null; promise.resolve(result) } + generator = screen + screen.show() + } catch (e: Exception) { generator = null; promise.reject("GENERATOR_IMAGE_UNAVAILABLE", e.message ?: "The generator could not start.", e) } } + + private companion object { const val PICK_IMAGE = 0x4156 } private fun startClock(promise: Promise) { val context = appContext.reactContext if (context == null || Build.VERSION.SDK_INT < 29) { promise.reject("GNSS_UNAVAILABLE", "GNSS clock mapping requires Android 10 or newer", null); return } diff --git a/mobile-app/modules/avsync/android/src/main/java/io/leeo86/mxl/avsync/GeneratorAudio.kt b/mobile-app/modules/avsync/android/src/main/java/io/leeo86/mxl/avsync/GeneratorAudio.kt new file mode 100644 index 0000000..d0801ba --- /dev/null +++ b/mobile-app/modules/avsync/android/src/main/java/io/leeo86/mxl/avsync/GeneratorAudio.kt @@ -0,0 +1,129 @@ +package io.leeo86.mxl.avsync + +import android.media.AudioAttributes +import android.media.AudioDeviceInfo +import android.media.AudioFormat +import android.media.AudioRouting +import android.media.AudioTimestamp +import android.media.AudioTrack +import android.os.Handler +import android.os.Looper +import android.os.Process +import java.util.concurrent.ConcurrentLinkedQueue + +// The generator's speaker output: one AudioTrack stream (48 kHz mono float) that a thread keeps +// writing in 5 ms blocks, silence between packets. AudioTrack.getTimestamp says which frame leaves the +// output at which System.nanoTime (the clock of Choreographer), so a packet planned for host time T +// starts at frame position + (T − time) × 48 000: sample-exact on the output's own clock, with the +// reported output latency already inside (iOS subtracts it from the schedule time instead). +// The start frame is worked out again for every block until the packet begins, so clock drift +// between the audio and the host clock never adds up. Errors go to onError on the main thread. +class GeneratorAudio(private val onError: (String) -> Unit) { + private class Packet(val hostNs: Long, val samples: FloatArray) { var start = -1L } + private val queue = ConcurrentLinkedQueue() + private val main = Handler(Looper.getMainLooper()) + private var track: AudioTrack? = null + private var thread: Thread? = null + @Volatile private var running = false + // Time from writing a frame until it plays (the audio lead); -1 until the output reports timing. + @Volatile var latencyNs = -1L; private set + // The output it plays on now, as in the sender calibration's audioRoute (for example builtin-speaker). + val route: String get() = track?.routedDevice?.let { routeName(it.type) } ?: "unknown" + val isRunning get() = running + + fun start(onRouteChange: () -> Unit) { + val format = AudioFormat.Builder().setEncoding(AudioFormat.ENCODING_PCM_FLOAT).setSampleRate(RATE).setChannelMask(AudioFormat.CHANNEL_OUT_MONO).build() + val minimum = AudioTrack.getMinBufferSize(RATE, AudioFormat.CHANNEL_OUT_MONO, AudioFormat.ENCODING_PCM_FLOAT) + // About 80 ms of buffer: timing comes from the timestamps, so a larger buffer costs nothing but safety against dropouts. + val output = AudioTrack.Builder() + .setAudioAttributes(AudioAttributes.Builder().setUsage(AudioAttributes.USAGE_MEDIA).setContentType(AudioAttributes.CONTENT_TYPE_SONIFICATION).build()) + .setAudioFormat(format).setTransferMode(AudioTrack.MODE_STREAM).setBufferSizeInBytes(maxOf(minimum, RATE * 4 * 80 / 1000)).build() + if (output.state != AudioTrack.STATE_INITIALIZED) { output.release(); throw IllegalStateException("The speaker is not available.") } + var device: Int? = null + output.addOnRoutingChangedListener(AudioRouting.OnRoutingChangedListener { routed -> + val now = routed.routedDevice ?: return@OnRoutingChangedListener + if (device != null && device != now.id) onRouteChange() + device = now.id + }, main) + queue.clear(); latencyNs = -1L + track = output; running = true + thread = Thread({ write(output) }, "avsync.generator.audio").apply { start() } + } + + // Plays samples from host time hostNs (System.nanoTime). Call well ahead (the main thread queues 2.5 s ahead). + fun schedule(hostNs: Long, samples: FloatArray) { queue.add(Packet(hostNs, samples)) } + + fun stop() { + running = false + thread?.join(500); thread = null + track?.let { runCatching { it.pause(); it.flush(); it.stop() }; it.release() } + track = null; queue.clear(); latencyNs = -1L + } + + private fun fail(message: String) { running = false; main.post { onError(message) } } + + private fun write(output: AudioTrack) { + Process.setThreadPriority(Process.THREAD_PRIORITY_URGENT_AUDIO) + val block = FloatArray(BLOCK) + val pending = ArrayList() + val stamp = AudioTimestamp() + var written = 0L + var mapFrame = -1L; var mapNs = 0L + var underruns = 0 + try { + output.play() + while (running) { + if (output.getTimestamp(stamp) && stamp.framePosition > 0) { + mapFrame = stamp.framePosition; mapNs = stamp.nanoTime + latencyNs = mapNs + framesToNs(written - mapFrame) - System.nanoTime() + } + while (true) pending.add(queue.poll() ?: break) + block.fill(0f) + val iterator = pending.iterator() + while (iterator.hasNext()) { + val p = iterator.next() + if (p.start < 0 || p.start >= written + 2 * BLOCK) { + // Not about to begin: place it with the newest timestamp; it waits while the output has none. + // Within two blocks of its start it stays put, so timestamp jitter cannot move it into the past. + if (mapFrame < 0) continue + p.start = startFrame(p.hostNs, mapFrame, mapNs) + if (p.start < written) return fail("Scheduling deadline missed. Restart the signal.") + } + if (!mix(block, written, p.samples, p.start)) iterator.remove() + } + // A dropout delays everything already written: the plan is wrong from here on. + val count = output.underrunCount + if (mapFrame >= 0 && count > underruns) return fail("Audio dropout. Tap Start to restart.") + underruns = count + val n = output.write(block, 0, BLOCK, AudioTrack.WRITE_BLOCKING) + if (n < 0) return fail("Audio output stopped. Tap Start to restart.") + written += n + } + } catch (e: Exception) { fail(e.message ?: "Audio output failed.") } + } + + companion object { + const val RATE = 48_000 + const val BLOCK = RATE / 200 // 5 ms + + fun framesToNs(frames: Long) = frames * 1_000_000_000L / RATE + // The output frame that plays at host time hostNs, from one timestamp (frame mapFrame plays at mapNs). + fun startFrame(hostNs: Long, mapFrame: Long, mapNs: Long) = mapFrame + Math.round((hostNs - mapNs) * RATE / 1e9) + // Adds the part of a packet (first frame start) that falls into the block at frame position; + // false once the packet ends before the next block. + fun mix(block: FloatArray, position: Long, samples: FloatArray, start: Long): Boolean { + val from = maxOf(position, start); val to = minOf(position + block.size, start + samples.size) + for (frame in from until to) block[(frame - position).toInt()] += samples[(frame - start).toInt()] + return start + samples.size > position + block.size + } + fun routeName(type: Int) = when (type) { + AudioDeviceInfo.TYPE_BUILTIN_SPEAKER -> "builtin-speaker" + AudioDeviceInfo.TYPE_BUILTIN_EARPIECE -> "builtin-earpiece" + AudioDeviceInfo.TYPE_WIRED_HEADPHONES, AudioDeviceInfo.TYPE_WIRED_HEADSET -> "wired" + AudioDeviceInfo.TYPE_USB_DEVICE, AudioDeviceInfo.TYPE_USB_HEADSET -> "usb" + AudioDeviceInfo.TYPE_BLUETOOTH_A2DP, AudioDeviceInfo.TYPE_BLUETOOTH_SCO -> "bluetooth" + AudioDeviceInfo.TYPE_HDMI -> "hdmi" + else -> "type-$type" + } + } +} diff --git a/mobile-app/modules/avsync/android/src/main/java/io/leeo86/mxl/avsync/GeneratorImageStore.kt b/mobile-app/modules/avsync/android/src/main/java/io/leeo86/mxl/avsync/GeneratorImageStore.kt new file mode 100644 index 0000000..3a9a1ab --- /dev/null +++ b/mobile-app/modules/avsync/android/src/main/java/io/leeo86/mxl/avsync/GeneratorImageStore.kt @@ -0,0 +1,60 @@ +package io.leeo86.mxl.avsync + +import android.content.Context +import android.graphics.Bitmap +import android.graphics.BitmapFactory +import android.graphics.ImageDecoder +import android.net.Uri +import android.os.Build +import java.io.File +import java.util.UUID + +// A small app-owned copy (PNG, at most 512 px) of the image chosen for the generator, like the iOS +// GeneratorImageStore. A UUID names it, so callers cannot hand the renderer an arbitrary path. +object GeneratorImageStore { + private const val MAX_SIDE = 512 + private val ID = Regex("^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$") + + private fun file(context: Context, id: String): File { + require(ID.matches(id)) { "Invalid generator image ID." } + return File(File(context.filesDir, "avsync-generator").apply { mkdirs() }, "$id.png") + } + + fun load(context: Context, id: String): Bitmap? { + if (id.isEmpty()) return null + return BitmapFactory.decodeFile(file(context, id).path) ?: throw IllegalStateException("Saved image is unavailable. Choose it again.") + } + + fun info(context: Context, id: String): Map? { + val image = runCatching { load(context, id) }.getOrNull() ?: return null + return mapOf("id" to id, "uri" to Uri.fromFile(file(context, id)).toString(), "width" to image.width.toDouble(), "height" to image.height.toDouble()) + } + + // Decodes the picked image (EXIF rotation applied on Android 9 and newer), scales it down and saves it. + fun save(context: Context, uri: Uri): Map { + val decoded = if (Build.VERSION.SDK_INT >= 28) { + ImageDecoder.decodeBitmap(ImageDecoder.createSource(context.contentResolver, uri)) { decoder, info, _ -> + val scale = minOf(1.0, MAX_SIDE.toDouble() / maxOf(info.size.width, info.size.height)) + decoder.setTargetSize(maxOf(1, Math.round(info.size.width * scale).toInt()), maxOf(1, Math.round(info.size.height * scale).toInt())) + decoder.allocator = ImageDecoder.ALLOCATOR_SOFTWARE + } + } else { + val bounds = BitmapFactory.Options().apply { inJustDecodeBounds = true } + context.contentResolver.openInputStream(uri)?.use { BitmapFactory.decodeStream(it, null, bounds) } + var sample = 1 + while (maxOf(bounds.outWidth, bounds.outHeight) / (sample * 2) >= MAX_SIDE) sample *= 2 + context.contentResolver.openInputStream(uri)?.use { BitmapFactory.decodeStream(it, null, BitmapFactory.Options().apply { inSampleSize = sample }) } + } ?: throw IllegalStateException("The selected image could not be imported.") + if (decoded.width <= 0 || decoded.height <= 0) throw IllegalStateException("The selected image is empty.") + val scale = minOf(1.0, MAX_SIDE.toDouble() / maxOf(decoded.width, decoded.height)) + val image = if (scale < 1) Bitmap.createScaledBitmap(decoded, maxOf(1, Math.round(decoded.width * scale).toInt()), maxOf(1, Math.round(decoded.height * scale).toInt()), true) else decoded + val id = UUID.randomUUID().toString() + val target = file(context, id) + val temporary = File(target.path + ".tmp") + temporary.outputStream().use { if (!image.compress(Bitmap.CompressFormat.PNG, 100, it)) throw IllegalStateException("The selected image could not be imported.") } + if (!temporary.renameTo(target)) throw IllegalStateException("The selected image could not be saved.") + return mapOf("id" to id, "uri" to Uri.fromFile(target).toString(), "width" to image.width.toDouble(), "height" to image.height.toDouble()) + } + + fun remove(context: Context, id: String) { runCatching { file(context, id).delete() } } +} diff --git a/mobile-app/modules/avsync/android/src/main/java/io/leeo86/mxl/avsync/GeneratorText.kt b/mobile-app/modules/avsync/android/src/main/java/io/leeo86/mxl/avsync/GeneratorText.kt new file mode 100644 index 0000000..7cce768 --- /dev/null +++ b/mobile-app/modules/avsync/android/src/main/java/io/leeo86/mxl/avsync/GeneratorText.kt @@ -0,0 +1,35 @@ +package io.leeo86.mxl.avsync + +import android.content.Context +import android.graphics.Bitmap +import android.graphics.Canvas +import android.graphics.Color +import android.graphics.Paint +import android.graphics.Typeface +import java.nio.ByteBuffer +import kotlin.math.ceil + +// Text for the shared picture renderer (called from text_android.cpp): the bundled DejaVu Sans +// drawn by Android, as iOS draws it with CoreText. Coverage only; the renderer blends it into luma. +object GeneratorText { + @Volatile private var typeface: Typeface = Typeface.SANS_SERIF + + fun load(context: Context) { + runCatching { typeface = Typeface.createFromAsset(context.assets, "DejaVuSans.ttf") } + } + + private fun paint(px: Float) = Paint(Paint.ANTI_ALIAS_FLAG).apply { typeface = this@GeneratorText.typeface; textSize = px; color = Color.WHITE } + + @JvmStatic fun width(text: String, px: Float): Int = Math.round(paint(px).measureText(text)) + @JvmStatic fun height(px: Float): Int = paint(px).fontMetrics.let { maxOf(1, ceil(it.descent - it.ascent).toInt()) } + // Alpha coverage, row by row (rows may be padded: the caller takes size / height as the stride). + @JvmStatic fun draw(text: String, px: Float, width: Int, height: Int): ByteArray { + val paint = paint(px) + val bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ALPHA_8) + Canvas(bitmap).drawText(text, 0f, -paint.fontMetrics.ascent, paint) + val out = ByteArray(bitmap.rowBytes * height) + bitmap.copyPixelsToBuffer(ByteBuffer.wrap(out)) + bitmap.recycle() + return out + } +} diff --git a/mobile-app/modules/avsync/android/src/main/java/io/leeo86/mxl/avsync/PhoneAnalyzer.kt b/mobile-app/modules/avsync/android/src/main/java/io/leeo86/mxl/avsync/PhoneAnalyzer.kt index 7ad8106..35d1650 100644 --- a/mobile-app/modules/avsync/android/src/main/java/io/leeo86/mxl/avsync/PhoneAnalyzer.kt +++ b/mobile-app/modules/avsync/android/src/main/java/io/leeo86/mxl/avsync/PhoneAnalyzer.kt @@ -28,6 +28,10 @@ class PhoneAnalyzer private constructor(private var handle: Long) : AutoCloseabl fun calibrating() = calibrating(live()) fun calibrationError(): String? = calibrationError(live()) fun takeCalibration(): CalibrationTake? = takeCalibration(live()) + // Calibrator board take: "camera" or "display" (docs/calibrator.md); pollCalibrator gives JSON reports. + fun beginCalibrator(mode: String) = beginCalibrator(live(), mode == "display") + fun calibratorActive() = calibratorActive(live()) + fun pollCalibrator(): String? = pollCalibrator(live()) fun picture(): DoubleArray? = picture(live()) override fun close() { if (handle != 0L) { destroy(handle); handle = 0 } } @@ -52,6 +56,9 @@ class PhoneAnalyzer private constructor(private var handle: Long) : AutoCloseabl @JvmStatic external fun calibrating(h: Long): Boolean @JvmStatic external fun calibrationError(h: Long): String? @JvmStatic external fun takeCalibration(h: Long): CalibrationTake? + @JvmStatic external fun beginCalibrator(h: Long, display: Boolean) + @JvmStatic external fun calibratorActive(h: Long): Boolean + @JvmStatic external fun pollCalibrator(h: Long): String? @JvmStatic external fun picture(h: Long): DoubleArray? } } diff --git a/mobile-app/modules/avsync/android/src/main/java/io/leeo86/mxl/avsync/PhoneGenerator.kt b/mobile-app/modules/avsync/android/src/main/java/io/leeo86/mxl/avsync/PhoneGenerator.kt new file mode 100644 index 0000000..42f8282 --- /dev/null +++ b/mobile-app/modules/avsync/android/src/main/java/io/leeo86/mxl/avsync/PhoneGenerator.kt @@ -0,0 +1,68 @@ +package io.leeo86.mxl.avsync + +import android.graphics.Bitmap + +// Kotlin handle on avsync::PhoneGeneratorPlan and the shared picture renderer (generator_jni.cpp), +// like ios/AVSyncGenerator.mm. image() may run on a worker while frame() runs on the main thread. +class PhoneGenerator(testId: String, rate: Int, epochTaiNs: Long, epochHostSeconds: Double, source: String) : AutoCloseable { + private var handle = create(testId, rate, epochTaiNs, epochHostSeconds, source).also { require(it != 0L) { "Invalid generator plan." } } + + // One display frame: what the layers show at this host time. + class Frame(val sequence: Long, val position: Int, val aWhite: Boolean, val named: Long, + val syncX: Float, val syncY: Float, val syncSize: Float, val syncColor: Int) + + fun setSource(name: String, subtitle: String, image: Bitmap?) { + val still = image?.let { argb(it) } + setSource(live(), name, subtitle, still, image?.width ?: 0, image?.height ?: 0) + } + fun setIcon(icon: Bitmap) = setIcon(live(), argb(icon), icon.width, icon.height) + fun setSenderCalibration(method: String, offsetMs: Double, uncertaintyMs: Double, day: Int) = setSenderCalibration(live(), method, offsetMs, uncertaintyMs, day) + // [markerX, markerY, markerSide] in the 1280 × 720 raster. + fun geometry(): IntArray = geometry(live()) + fun syncForeground(): Bitmap = bitmap(syncForeground(live())) + fun frame(targetHostSeconds: Double): Frame { + val v = frame(live(), targetHostSeconds) + return Frame(v[0].toLong(), v[1].toInt(), v[2] != 0.0, v[3].toLong(), v[4].toFloat(), v[5].toFloat(), v[6].toFloat(), + android.graphics.Color.rgb(v[7].toInt(), v[8].toInt(), v[9].toInt())) + } + fun hostTime(sequence: Long): Double = hostTime(live(), sequence) + fun audio(sequence: Long): FloatArray = audio(live(), sequence) + fun image(sequence: Long): Bitmap = bitmap(image(live(), sequence) ?: throw IllegalStateException("Session exceeds one day.")) + + override fun close() { if (handle != 0L) { destroy(handle); handle = 0 } } + private fun live(): Long { check(handle != 0L) { "PhoneGenerator closed" }; return handle } + + companion object { + const val WIDTH = 1280 + const val HEIGHT = 720 + init { System.loadLibrary("avsync-jni") } + + // A still at most 512 px on the long side, as ARGB ints (straight alpha), like the iOS bridge. + private fun argb(image: Bitmap): IntArray { + val scale = minOf(1.0, 512.0 / maxOf(image.width, image.height)) + val small = if (scale < 1) Bitmap.createScaledBitmap(image, maxOf(1, Math.round(image.width * scale).toInt()), maxOf(1, Math.round(image.height * scale).toInt()), true) else image + return IntArray(small.width * small.height).also { small.getPixels(it, 0, small.width, 0, 0, small.width, small.height) } + } + private fun bitmap(argb: IntArray) = Bitmap.createBitmap(argb, WIDTH, HEIGHT, Bitmap.Config.ARGB_8888) + + @JvmStatic external fun testId(prefix: String, installId: String, version: String, settings: Array): String + // Calibration pattern (docs/calibrator.md display mode), the same core functions as every generator. + @JvmStatic external fun calibrationLeftWhite(frame: Long, fps: Int): Boolean + @JvmStatic external fun calibrationSwapFrame(swap: Long, fps: Int): Long + @JvmStatic external fun calibrationBeep(sampleRate: Int): FloatArray + // Clicks and beeps in mono samples: JSON with ok, pairs, clicks and, when ok, offsetMs, uncertaintyMs, … + @JvmStatic external fun displayCalibration(samples: FloatArray, count: Int, sampleRate: Int, minimumPairs: Int): String + + @JvmStatic private external fun create(testId: String, rate: Int, epochTaiNs: Long, epochHost: Double, source: String): Long + @JvmStatic private external fun destroy(h: Long) + @JvmStatic private external fun setSource(h: Long, name: String, subtitle: String, argb: IntArray?, width: Int, height: Int) + @JvmStatic private external fun setIcon(h: Long, argb: IntArray, width: Int, height: Int) + @JvmStatic private external fun setSenderCalibration(h: Long, method: String, offset: Double, uncertainty: Double, day: Int) + @JvmStatic private external fun geometry(h: Long): IntArray + @JvmStatic private external fun syncForeground(h: Long): IntArray + @JvmStatic private external fun frame(h: Long, target: Double): DoubleArray + @JvmStatic private external fun hostTime(h: Long, seq: Long): Double + @JvmStatic private external fun audio(h: Long, seq: Long): FloatArray + @JvmStatic private external fun image(h: Long, seq: Long): IntArray? + } +} diff --git a/mobile-app/modules/avsync/android/src/test/java/io/leeo86/mxl/avsync/GeneratorTest.kt b/mobile-app/modules/avsync/android/src/test/java/io/leeo86/mxl/avsync/GeneratorTest.kt new file mode 100644 index 0000000..c207114 --- /dev/null +++ b/mobile-app/modules/avsync/android/src/test/java/io/leeo86/mxl/avsync/GeneratorTest.kt @@ -0,0 +1,48 @@ +package io.leeo86.mxl.avsync +import org.junit.Test +import org.junit.Assert.* +class GeneratorTest { + @Test fun packetStartsOnTheOutputTimestamp() { + // Frame 48 000 plays at 10 s: a packet planned for 10.5 s starts 24 000 frames later. + assertEquals(72_000L, GeneratorAudio.startFrame(10_500_000_000L, 48_000, 10_000_000_000L)) + // Rounded to the nearest frame (one frame = 20.83 µs). + assertEquals(48_001L, GeneratorAudio.startFrame(10_000_015_000L, 48_000, 10_000_000_000L)) + assertEquals(48_000L, GeneratorAudio.startFrame(10_000_010_000L, 48_000, 10_000_000_000L)) + assertEquals(5_000_000L, GeneratorAudio.framesToNs(240)) + } + @Test fun mixPlacesSamplesExactly() { + val packet = FloatArray(300) { it + 1f } + // The packet starts 100 frames into the block at position 1000 and runs into the next block. + val first = FloatArray(GeneratorAudio.BLOCK) + assertTrue(GeneratorAudio.mix(first, 1000, packet, 1100)) + assertEquals(0f, first[99]); assertEquals(1f, first[100]); assertEquals(140f, first[239]) + val second = FloatArray(GeneratorAudio.BLOCK) + assertFalse(GeneratorAudio.mix(second, 1240, packet, 1100)) + assertEquals(141f, second[0]); assertEquals(300f, second[159]); assertEquals(0f, second[160]) + // Not begun yet: nothing written, still pending. + val before = FloatArray(GeneratorAudio.BLOCK) + assertTrue(GeneratorAudio.mix(before, 0, packet, 5000)) + assertTrue(before.all { it == 0f }) + } + @Test fun rasterRotatesInPortrait() { + val portrait = GeneratorLayout.fitRaster(1000f, 2000f) + assertTrue(portrait.rotate); assertEquals(1000f / 720, portrait.scale, 1e-6f) + val landscape = GeneratorLayout.fitRaster(2560f, 1000f) + assertFalse(landscape.rotate); assertEquals(1000f / 720, landscape.scale, 1e-6f) + assertEquals(1f, GeneratorLayout.fitRaster(1280f, 720f).scale, 1e-6f) + } + @Test fun calibrationPatternInMillimetres() { + val mm = GeneratorLayout.pixelsPerMm(25.4f * 16, 480f) // 16 px per mm + assertEquals(16f, mm, 1e-4f) + assertEquals(480f / 25.4f, GeneratorLayout.pixelsPerMm(0f, 480f), 1e-4f) + val l = GeneratorLayout.calibration(1080f, 2400f, mm) + assertEquals(2400f - 62 * 16, l.top, 1e-3f) + // Board 22 × 66 mm centred on the pattern line; sensors (5 mm from the notch end) at the middle of the halves. + assertEquals(22f * 16, l.boardRight - l.boardLeft, 1e-3f); assertEquals(66f * 16, l.boardBottom - l.boardTop, 1e-3f) + assertEquals(540f, (l.boardLeft + l.boardRight) / 2, 1e-3f) + assertEquals((l.top + 2400f) / 2, l.boardBottom - 5 * 16, 1e-3f) + assertEquals(l.boardBottom + 2 * 16, l.notchTick, 1e-3f) + // A short view: the halves fill it. + assertEquals(0f, GeneratorLayout.calibration(1080f, 500f, mm).top, 0f) + } +}