Skip to content

Android generator and calibrator take - #71

Merged
LeeO86 merged 24 commits into
mainfrom
feat/android-generator
Nov 9, 2026
Merged

LeeO86 merged 24 commits into
mainfrom
feat/android-generator

Conversation

@LeeO86

@LeeO86 LeeO86 commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #70. Rule: all generators get the same features. Android had no generator and no calibrator take; now it has both, behind the same JS API as iOS.

What this does

  1. Android generator (Signal → Start test signal), same as the iPhone: fullscreen test picture (AVS3 fields, sync square and scale, AVS4 QR with test ID, Title and sender calibration), sound packets on the same plan, 1 or 2 events/s, Title, Subtitle, image, stable sender ID (phone- + core generator_test_id, install ID in the app preferences, version <versionName> (<versionCode>)), large "Stopped" after Stop, same stop reasons (background, audio focus lost, output changed, clock jump, missed deadline, 23 h), clock reference from the NTP check, screen brightness and wake.
  2. Calibration mode (generateStyled(..., {mode: 'calibration'})): halves swap every 500 ms on the 60 Hz grid (core calibration_left_white / calibration_swap_frame), 4 ms 1.5 kHz beep 100 ms after each swap, red board outline in real millimetres, microphone records meanwhile, core display_calibration once a second, result or error back when the screen closes.
  3. Calibrator take in the Android camera view: beginCalibrator('camera' | 'display') → PhoneAnalyzer::begin_calibrator, progress/result from poll_calibrator. The camera take re-runs the auto step and locks at the lowest ISO.
  4. App: no platform branch any more for the wizard (supportsCalibrator no longer checks iOS); the Title placeholder says "Android" on Android.

Design choices

  • Frame timing: Choreographer. Android 13+: postVsyncCallback → preferredFrameTimeline.expectedPresentationTimeNanos (like iOS targetTimestamp). Older: frame start + 2 display periods (estimate). The screen asks for a 60 Hz display mode. All on System.nanoTime.
  • Audio: one streaming AudioTrack (48 kHz mono float, ~80 ms buffer), a thread writes 5 ms blocks, silence between packets. AudioTrack.getTimestamp maps an output frame to System.nanoTime (the Choreographer clock), so a packet for host time T starts at frame pos + (T − time) × 48000: placed to the sample, output latency already inside (iOS subtracts outputPresentationLatency instead). The start frame is recomputed with the newest timestamp each block until two blocks before it starts, so audio-vs-system clock drift does not add up and timestamp jitter cannot push it into the past. Underrun → stop (everything queued would play late). Why not AAudio/Oboe: the timestamp gives the same exact mapping from API 24 without native audio code or a new dependency; low latency is not needed because packets are planned 2.5 s ahead.
  • Shared core: PhoneGeneratorPlan, AVS4 QR, render_picture / render_sync_foreground, calibration functions compile into libavsync-jni like the iOS Core_*.cpp (generator_jni.cpp mirrors AVSyncGenerator.mm). Text: JNI to Android's own text drawing with the bundled DejaVu Sans (GeneratorText.kt), as iOS uses CoreText; the font and icon come from the iOS resource folder (gradle assets.srcDirs).
  • Screen: a fullscreen Dialog over the app (no new Activity / manifest entry); one view draws plate, fields, sync square and foreground, rotated in portrait like iOS.
  • Image: Android photo picker (13+) or document picker, copy ≤ 512 px PNG in the app's files (GeneratorImageStore.kt), removed when replaced.

Tests

  • Core ctest (core + clock): pass (after merging the latest Calibrator wizard: camera and screen calibration with the calibrator board #70).
  • node --test mobile-app/src/*.test.mjs tests/web/*.test.cjs: 58 pass.
  • Android JVM unit tests (:avsync:testDebugUnitTest, local docker): 14/14 pass, incl. new GeneratorTest (sample placement from a timestamp, block mixing, raster fit, calibration layout in mm).
  • Host check of the JNI bridge (the same generator_jni.cpp + text_android.cpp built for x86_64, a JVM stand-in for GeneratorText): test-ID vector phone-985274814721, picture renders with all labels, title/subtitle, QR and fields; frame/position/named sequence right; synthetic click+beep train gives −37.93 ms for a planted −38 ms (24 pairs); silence gives the "No clicks heard" error.
  • src/androidTest/PhoneGeneratorTest (needs a device/emulator, not run here; no KVM in the lab): stable ID test vector, picture 1280×720 with AVS4 QR read by ML Kit, host times, audio packet, frame/position, foreground, calibration pattern and silent analysis.
  • Release APK (arm64-v8a) and the androidTest APK built locally in docker (steps of android-apk.yaml); APK contains libavsync-jni.so, assets/DejaVuSans.ttf, assets/icon.png. CI apk job on this PR: green.

Device test list (Android phone)

  1. Signal → Start test signal: picture fills the screen, rotates in portrait, sync square animates, fields burst at each event; status shows frame skips ≈ 0 and an audio lead (typ. 20–100 ms).
  2. Measure it with a second phone (iPhone or Android) or the container: QR read, testId phone-…, Title shown, AV offset stable over 2 min, no "deadline missed"/"dropout".
  3. Stop → black picture with "Stopped"; Start again → same sender ID. Change rate → new ID.
  4. Home button / screen off while running → stops with "Paused"; plug headphones → "Audio output changed".
  5. Options: Title, Subtitle, Add image (photo picker), Change, Remove; image appears in the picture.
  6. Settings → With calibrator shows on Android. Camera step with a board: exposure dark (lowest ISO), ring fills, result plausible; Save.
  7. Screen step: halves at the bottom, outline about 22 × 66 mm (measure with a ruler), beep audible, mic hears click + beep, result closes the screen; Save; then a normal test signal shows "sender calibration ±x ms" in the status and the analyser shows it.
  8. 90/120 Hz phone: does it switch to 60 Hz (skips stay ≈ 0)?
  9. Android 12 or older phone (no expected-presentation API): compare AV offset with an Android 13+ phone or iPhone to check the two-period estimate.

Open risks

  • Frame time before Android 13 is an estimate (+2 vsync); the real pipeline depth can differ per phone.
  • AudioTrack.getTimestamp accuracy depends on the HAL (some report coarse or late positions); measured AV offset will show it.
  • Some phones refuse 60 Hz and stay at 90/120 Hz: the 60 Hz plan still shows the right frame per vsync, but at 90 Hz frames alternate 1/2 vsyncs.
  • Recording while playing: AudioRecord with UNPROCESSED/VOICE_RECOGNITION should not cancel the beep; to check on devices with aggressive processing.
  • Calibrator camera take on phones without manual exposure: only exposure compensation, flash may saturate.

🤖 Generated with Claude Code

https://claude.ai/code/session_017ybuzgBaGxuB5UiaE8dRa4

LeeO86 and others added 23 commits November 8, 2026 15:30
Shared C++ for the AV Sync calibrator board (docs/calibrator.md, contract v1):
- find_sound_onsets: clicks and beeps in a recording (trailing 0.25 ms RMS,
  25 % of peak), told apart by 2.7 kHz vs 1.5 kHz energy.
- find_flash_onsets: LED flash start from row-band means, on the
  rolling-shutter row the analyzer uses; readout from the band height or,
  with full-frame exposure, fitted over the pulses in camera_calibration.
- camera_calibration: pair flashes with clicks, robust median, uncertainty.
- display_calibration: click then beep D = 100 ms later, offset = gap - D,
  both swap directions weighted equally.
- calibration pattern for all generators: halves swap every 500 ms, beep.
Unit tests on synthetic signals (noise, missing pulses, stray sound, rolling
shutter with known/unknown readout, slow screen, sensor mismatch).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ybuzgBaGxuB5UiaE8dRa4
PhoneAnalyzer::begin_calibrator keeps 32 row-band means per frame and the
microphone, and poll_calibrator reports progress about once a second, then
the result (20 pairs or 30 s) or an error. No live report or QR scan while
it runs. iOS: beginCalibrator(mode) on the camera view; the camera take
first re-locks exposure (frame interval) at the lowest ISO, because the LED
floods the lens.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ybuzgBaGxuB5UiaE8dRa4
…the QR

generateStyled(..., {mode: 'calibration'}) shows the calibration pattern in
the generator's own screen: the same display link, layers, audio engine and
player node as the test signal. Portrait halves left and right near the
bottom microphone, a red outline of the board, halves swap every 500 ms, a
1.5 kHz beep 100 ms after each swap (scheduled on the host clock minus the
reported output latency). The session is play-and-record on the built-in
speaker; the microphone take is analysed with the shared core
(display_calibration) and the result comes back when the screen closes.

The test signal sends a saved sender calibration in the AVS4 QR
(senderCalibration* options), only on the audio output it was measured on.
Core pattern helpers are now frame based (also 59.94 fps).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ybuzgBaGxuB5UiaE8dRa4
Same feature as the phone: the generator can send the calibration pattern
instead of the test signal, so a phone with the calibrator board can
measure a TV/monitor chain. Left/right halves swap every 500 ms on the
grain grid (also interlaced and 59.94), a 1.5 kHz beep 100 ms after each
swap at the chirp level, a grey tick at the ends of the pattern line.
Live key, also in the Signal API (pattern) and the web form (Picture).

tests/tools/calibration_flow + tests/integration/calibration_pattern.py
read both flows, check the timing and run a simulated calibrator through
display_calibration (1080p50, 1080i25, 1080p59.94: 0.03 ms).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ybuzgBaGxuB5UiaE8dRa4
Three steps with small animations: plug in, camera (the calibrator on the
rear camera; progress ring of 20 flashes; 'Camera vs mic' result saved as
this phone's analyser calibration, method calibrator-pulse, same record and
validity as the clap), screen (the generator's calibration pattern;
'Screen vs speaker' saved as this phone's sender calibration, method
transponder). The test signal then sends it in the AVS4 QR. A TV/monitor
showing the container's calibration pattern can be measured too (result
only, not saved). The clap calibration is unchanged. iOS only for now
(Android has no generator and no calibrator take yet).

Pure logic in src/calibrator.mjs with node tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ybuzgBaGxuB5UiaE8dRa4
docs/calibrator.md (copy of PR #69's contract) gets an 'App side' section
and the list of changes from the v1 proposal (none needs a firmware change).
calibration.md: user guide 'With the calibrator'. phone-generator.md:
calibration mode and the sender calibration in the QR. SPECIFICATION:
AVSYNC_PATTERN and the calibrator in 12.3. IMPLEMENTATION_PLAN: entry with
the deviations. CHANGELOG, mobile-app README.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ybuzgBaGxuB5UiaE8dRa4
… clap gap fill unchanged

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ybuzgBaGxuB5UiaE8dRa4
Done once in core (Pairer::set_sender_calibration) for the phone session,
the container analyzer and the file analyzer. Events keep raw_av_offset_ms
and sender_calibration_ms; the phone reading has rawOffsetMs and
senderCalibrationMs; /api/v1/measurement has raw_offset_ms and
sender_calibration_ms; the web UI Input card shows AV sync raw. Not
calibrated: no change. The phone fixture sends a sender calibration at 2 Hz.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ybuzgBaGxuB5UiaE8dRa4
…ickname

Measure and Results show the raw value and the sender calibration under a
corrected result; saved results keep them. The analyser calibration stays
subtracted in the app only, so nothing counts twice. Clear calibration asks
first and clears the analyser and the sender calibration. A sender's default
name is the newest Title from its QR; a set nickname wins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ybuzgBaGxuB5UiaE8dRa4
… nickname

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ybuzgBaGxuB5UiaE8dRa4
PhoneGeneratorPlan, the AVS4 QR, render_picture/render_sync_foreground and the
calibration pattern functions compile into libavsync-jni as on iOS
(generator_jni.cpp mirrors AVSyncGenerator.mm). The renderer's text goes through
JNI to Android's text drawing with the bundled DejaVu Sans (GeneratorText.kt),
as iOS uses CoreText; the font and icon come from the iOS resource folder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ybuzgBaGxuB5UiaE8dRa4
Same behaviour and options as the iPhone generator: test picture with the AVS3
fields, sync square and scale, AVS4 QR with test ID, title and sender calibration,
stable sender ID, 'Stopped' screen, calibration pattern with the microphone.
Frames: Choreographer's expected presentation time (API 33+, else frame start
plus two periods). Sound: one AudioTrack stream written in 5 ms blocks; packets
are placed to the sample by AudioTrack.getTimestamp on System.nanoTime, the
clock Choreographer uses. Module: supportsGenerator, generate, generateStyled,
pickGeneratorImage, generatorImageInfo, like iOS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ybuzgBaGxuB5UiaE8dRa4
beginCalibrator('camera' | 'display') as on iOS: the camera take re-runs the
auto step and locks at the lowest ISO (or the lowest exposure compensation when
the camera has no manual exposure), then PhoneAnalyzer::begin_calibrator starts;
display mode listens only. Progress and result reports come from
poll_calibrator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ybuzgBaGxuB5UiaE8dRa4
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ybuzgBaGxuB5UiaE8dRa4
…on, clear both)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ybuzgBaGxuB5UiaE8dRa4
Same file on both branches now, so the merge has no conflict here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ybuzgBaGxuB5UiaE8dRa4
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ybuzgBaGxuB5UiaE8dRa4
@LeeO86
LeeO86 changed the base branch from feat/calibrator-wizard to main November 9, 2026 10:23
@LeeO86
LeeO86 marked this pull request as ready for review November 9, 2026 10:23
@LeeO86
LeeO86 merged commit 7575c91 into main Nov 9, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant