Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
21b6826
Core: calibrator analysis and calibration pattern
LeeO86 Nov 8, 2026
6d752d9
Phone analyzer: calibrator take (camera and external display)
LeeO86 Nov 8, 2026
2df391f
iPhone generator: calibration pattern mode and sender calibration in …
LeeO86 Nov 8, 2026
9ddbe93
Container generator: AVSYNC_PATTERN=calibration
LeeO86 Nov 8, 2026
6f485ac
App: calibrator wizard (Settings > With calibrator)
LeeO86 Nov 8, 2026
e94b39f
Docs: calibrator wizard, calibration pattern, contract app side
LeeO86 Nov 8, 2026
520975f
Docs: native contract for the calibrator take and calibration mode
LeeO86 Nov 8, 2026
aea384b
Calibrator take: Cancel during the exposure lock no longer starts it;…
LeeO86 Nov 8, 2026
811926e
iOS: type displayCalibration as [String: Any] for Swift
LeeO86 Nov 8, 2026
0369bd9
iPhone generator: Stop clears the whole raster and shows "Stopped"
LeeO86 Nov 8, 2026
16b7a5a
Analysers subtract the sender calibration from the AVS4 QR
LeeO86 Nov 8, 2026
574d151
App: raw value and sender calibration, clear both, Title as default n…
LeeO86 Nov 8, 2026
0516c69
Docs: analysers subtract the sender calibration, clear both, Title as…
LeeO86 Nov 8, 2026
6fbcdba
Android: generator bridge to the shared core and picture renderer
LeeO86 Nov 8, 2026
c1126cf
Android: fullscreen generator, calibration mode and image picker
LeeO86 Nov 8, 2026
405fbc9
Android: calibrator take in the camera view
LeeO86 Nov 8, 2026
1607084
App: calibrator wizard on Android, generator title placeholder
LeeO86 Nov 8, 2026
4690637
Android: generator tests (audio placement, layout, bridge on a device)
LeeO86 Nov 8, 2026
5ec6532
Docs: Android generator and calibrator take
LeeO86 Nov 8, 2026
0fe96a5
Merge feat/calibrator-wizard (analysers subtract the sender calibrati…
LeeO86 Nov 8, 2026
104caab
docs/calibrator.md: BOOT button and 1 nF filter row from #69
LeeO86 Nov 8, 2026
651ac8b
Merge feat/calibrator-wizard (calibrator.md wording)
LeeO86 Nov 8, 2026
c7deebe
Android camera view: live poll goes on between calibrator reports, as…
LeeO86 Nov 8, 2026
ef0a52e
Merge main (#70 squashed) into feat/android-generator
LeeO86 Nov 9, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,16 @@

## Mobile (not released yet)

- Android test signal: Signal → Start test signal now works on Android, with
the same picture, options (1 or 2 events/s, title, subtitle, image), AVS4 QR
(test ID, title, sender calibration), stable sender ID and "Stopped" screen
as on the iPhone. Frames are timed with Choreographer, the sound with one
AudioTrack placed to the sample by its output timestamps
(`docs/phone-generator.md`). Not tested on a phone yet.
- Calibrator wizard on Android: Settings → With calibrator now shows on Android
too. The camera step uses the new camera view method `beginCalibrator`
(lowest ISO where the camera allows it), the screen step the generator's
calibration mode with the microphone listening. Not tested with a board yet.
- Analysers subtract the sender calibration: when the AVS4 QR carries one, the
AV offset shown and saved is the measured value minus the sender's own
screen-vs-speaker offset. Measure and Results show the raw value and the
Expand Down
43 changes: 36 additions & 7 deletions IMPLEMENTATION_PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -1288,8 +1288,9 @@ sender calibration) is one more entry. Name, subtitle and image are not inputs.
when used, so a live rate change gives the new id at once, and config export
keeps the empty value instead of freezing an old id. `GET /api/v1/generator`
shows `test_id`. The analyzer's own fallback label stays the NMOS seed.
- Android has no generator yet (`supportsGenerator` is false); it should call the
same core function when it gets one.
- Android (2026-11-08): prefix `phone-`, a random install ID in the app's
preferences (new after a reinstall), version `<versionName> (<versionCode>)`,
settings `rate=<1|2>`, through the same core function.
The core test holds the shared test vector.

Generator name in the QR: not done. The live QR of the phone and the container
Expand Down Expand Up @@ -1369,9 +1370,8 @@ Maintainer design (2026-11-08): Settings → With calibrator, a three-step wizar
- Screen calibration is a mode of the generators, not a React view: iPhone
`generateStyled(..., {mode: 'calibration'})` (same display link, layers,
audio engine and player node), container `AVSYNC_PATTERN=calibration`
(live key, Signal API `pattern`, web form "Picture"). Android has no
generator (`supportsGenerator` false): skipped; its analyser could call
`begin_calibrator` through JNI later (not wired).
(live key, Signal API `pattern`, web form "Picture"). Android: added
later the same way (see "Android generator and calibrator take").
- Results: camera → analyser calibration record (method `calibrator-pulse`,
same validity as the clap); screen → `senderCalibration` in the session
state (method `transponder`, 90 days, same OS version, sent only on the same
Expand Down Expand Up @@ -1400,9 +1400,38 @@ Deviations and choices:
(`raw_av_offset_ms` / `rawOffsetMs`) and the calibration next to it. The
phone's analyser calibration stays in the app (`liveView`), so nothing is
counted twice.
- The wizard and the calibrator take are iOS only (`beginCalibrator` is not in
the Android camera view yet); the Settings entry is hidden on Android.
- The wizard and the calibrator take were iOS only at first; Android got them
with its generator (next section).
- Debug → Clear calibration clears the sender calibration too (decided; with
a confirm that says both are cleared).
- The iPhone pattern size assumes about 6 pt per mm (current iPhones); the
outline is a placing guide, the pattern line itself is exact.

## Android generator and calibrator take (2026-11-08)

Maintainer rule: all generators get the same features. Android gets the phone
generator (test signal and calibration pattern) and the calibrator take, behind
the same JS API as iOS (`supportsGenerator`, `generate`, `generateStyled`,
`pickGeneratorImage`, `generatorImageInfo`, camera view `beginCalibrator`), so
the app has no platform branch for them (only the Title placeholder).

- Shared core and renderer: `PhoneGeneratorPlan`, `generator_test_id`, the AVS4
QR, `render_picture`/`render_sync_foreground` and the calibration functions
compile into the Android library as on iOS (`generator_jni.cpp`).
- Text: Android draws the renderer's text itself (DejaVu Sans from the iOS
resource folder, `GeneratorText.kt` through JNI), as iOS uses CoreText.
Deviation from the server: not Blend2D; glyph shapes can differ by a pixel.
- Frame time: Choreographer's expected presentation time on Android 13+;
before that the frame start plus two display periods (an estimate). The
screen asks for a 60 Hz display mode; on a display that stays at 90/120 Hz
each frame still shows the 60 Hz plan's frame for its time.
- Sound: one streaming AudioTrack, packets placed by `AudioTrack.getTimestamp`
(frame ↔ System.nanoTime) instead of AAudio/Oboe: exact sample placement from
API 24, no native audio code or new dependency. The timestamp includes the
output latency, so nothing is subtracted. An underrun stops the signal.
- Calibrator camera take: lowest ISO when the camera allows manual exposure,
else auto exposure locked at the lowest exposure compensation; high-speed
mode only AE lock.
- The calibration pattern is drawn in real millimetres (display xdpi).
- Not verified on a device here (no emulator with KVM): timing, the pattern,
the microphone during playback, the photo picker. Device test list in the PR.
2 changes: 1 addition & 1 deletion docs/calibration.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ The rig is an LED and a piezo on one microcontroller pin. Parts and firmware are

The calibrator is a small board with an LED, a buzzer and two light sensors
([signal contract](calibrator.md)). The phone powers it over a USB-C cable.
In the app: **Settings → With calibrator** (iPhone for now; the clap
In the app: **Settings → With calibrator** (iPhone and Android; the clap
calibration stays as it is). Three steps:

1. **Plug in** the calibrator.
Expand Down
12 changes: 7 additions & 5 deletions docs/calibrator.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# AV Sync calibrator: signal contract

**v1 proposal. The app side is built (iOS app and container generator, see [App side](#app-side-built)), but only tested on synthetic signals. The board is not built or tested yet.**
**v1 proposal. The app side is built (iOS and Android app and container generator, see [App side](#app-side-built)), but only tested on synthetic signals. The board is not built or tested yet.**

The calibrator is a small board (66 × 22 mm) that the phone powers over a USB-C to USB-C cable.
The cable is not included: the user brings their own.
Expand Down Expand Up @@ -173,7 +173,7 @@ flips, no timed delay). Then it ignores the sensors for **200 ms** (lock-out).
## App side (built)

Code: `core/include/avsync/calibrator.hpp` (all analysis, shared by every app),
`PhoneAnalyzer::begin_calibrator` (camera take), the iPhone generator's
`PhoneAnalyzer::begin_calibrator` (camera take), the iPhone and Android generators'
calibration mode and the container's `AVSYNC_PATTERN=calibration`. User guide:
[calibration.md](calibration.md#with-the-calibrator).

Expand All @@ -197,7 +197,9 @@ flash lands on different rows, and click − flash grows with the row by exactly
the readout time. Flashes and clicks are paired by the most common difference
(±12 ms around it). Offset = median of the pairs after dropping outliers
(beyond 3 σ, at least 0.5 ms); uncertainty = standard error of the median.
For the camera take the iPhone locks the exposure at the lowest ISO.
For the camera take the iPhone locks the exposure at the lowest ISO; Android
too when the camera allows manual exposure, else it locks auto exposure at the
lowest exposure compensation (and in the high-speed mode, which allows neither).

**Display mode.** Every click is paired with the next beep less than 450 ms
later. Offset = beep − click − D, the median per swap direction (every other
Expand All @@ -207,8 +209,8 @@ times, so the same analysis works for a TV fed by the container generator.
**Pattern.** Every generator uses the same core functions: period k lasts
500 ms on its frame grid (for 59.94 fps the swap is the first frame at or
after k × 500 ms), left half white in even periods, the beep D after each
swap. The iPhone draws the halves in portrait near the bottom microphone,
about 62 mm tall, with a red outline of the board (sensor end down, notch
swap. The phones draw the halves in portrait near the bottom microphone,
about 62 mm tall (Android in real millimetres from the display density), with a red outline of the board (sensor end down, notch
tick). The container fills the whole picture and draws a grey tick at both
ends of the pattern line.

Expand Down
5 changes: 3 additions & 2 deletions docs/mobile-native.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,15 @@ the core, so both platforms give the same numbers.
| `synchronizeClock(server)` | Promise of `{source:'ntp', server, referenceUnixMs, referenceMonotonicMs, uncertaintyMs, roundTripMs, offsetMs, clockId, timebase}` (SNTP, 8 samples, smallest round trip) |
| `setScreenAwake(on)` | keeps the display on |
| `generate(rate, taiUtcSeconds, options)` | Promise; shows the fullscreen AVS3 source until closed, resolves `{skippedDisplayFrames}` |
| `generateStyled(rate, taiUtcSeconds, options)` | the same with string options; `mode: 'calibration'` shows the calibration pattern and resolves with `calibration` or `calibrationError`; `senderCalibrationMethod/OffsetMs/UncertaintyMs/Day/Route` put the sender calibration in the QR (iOS; [phone-generator.md](phone-generator.md#calibration-mode)) |
| `generateStyled(rate, taiUtcSeconds, options)` | the same with string options; `mode: 'calibration'` shows the calibration pattern and resolves with `calibration` or `calibrationError`; `senderCalibrationMethod/OffsetMs/UncertaintyMs/Day/Route` put the sender calibration in the QR ([phone-generator.md](phone-generator.md#calibration-mode)) |
| `pickGeneratorImage()` / `generatorImageInfo(id)` | the generator image: Promise of `{id, uri, width, height}` or null (cancelled); info for a saved id |

## Camera view `Avsync`

Props: `active` (bool), `captureFps` (0 = automatic best rate), `taiUtcOffset`
(seconds, default 37). Methods: `beginCalibration()`, `resumeAnalysis()`,
`focusAt(x, y)` (tap to focus; x and y are 0–1 across the view, see below),
`beginCalibrator(mode)` (iOS only so far; `camera` or `display`, see
`beginCalibrator(mode)` (`camera` or `display`, see
[calibrator.md](calibrator.md#app-side-built)).
Event `onReport` with one of:

Expand Down
55 changes: 45 additions & 10 deletions docs/phone-generator.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# iPhone test-signal source
# Phone test-signal source (iPhone and Android)

Use the current native iOS/TestFlight build, open **Generate**, choose **1 or 2 chirps/s**,
Use the current native iOS/TestFlight build or the Android APK, open **Signal**, choose **1 or 2 chirps/s**,
set **Title**, **Subtitle** and optionally **Choose image**, then **Start fullscreen signal**. Hold the phone sideways. The raster rotates
inside the fullscreen screen; Measure/Results keep their portrait layout.
Film the entire test picture, including all four corner fiducials, with an
Expand Down Expand Up @@ -40,13 +40,36 @@ ends. Normal server and FILE packets retain their original bytes/semantics.

## Native scheduling and lifecycle

A native CADisplayLink supplies the next presentation target. The first source
iPhone: a native CADisplayLink supplies the next presentation target. The first source
event is planned 1.5 s ahead on that cadence. Subsequent source frames/events
follow a 60-Hz logical host-clock timeline. AVAudioPlayerNode buffers are queued
2.5 s ahead on the same host clock, subtracting the node's reported downstream
output presentation latency. This is an OS timing estimate, not a microphone or
photodiode measurement. All timing and synthesis run outside JavaScript.

Android does the same with its own clocks (`AvsyncGenerator.kt`):

- **Picture.** A fullscreen screen over the app. Choreographer gives each
frame's expected presentation time (Android 13 and newer; before that the
frame start plus two display periods, an estimate). The screen asks for a
60 Hz display mode. Each frame shows what the 60 Hz plan says for that time.
- **Sound.** One AudioTrack stream (48 kHz, mono) that a thread keeps writing
in 5 ms blocks, silence between the packets. `AudioTrack.getTimestamp` says
which sample leaves the output at which time, on the same clock as
Choreographer (`System.nanoTime`). A packet planned for time T starts at
that sample + (T − timestamp time) × 48 000, so it is placed to the sample,
with the reported output latency already inside. The position is worked out
again with the newest timestamp until the packet starts, so drift between
the audio clock and the system clock does not add up. Why AudioTrack and not
AAudio/Oboe: the timestamp gives the same exact mapping from API 24 on,
without native audio code or a new library; low latency does not matter
because every packet is planned 2.5 s ahead. A dropout (underrun) stops the
signal, because everything already written would then play late.
- **Same rules otherwise.** The same 1.5 s start, 2.5 s queue, status line,
"Stopped" screen, brightness and screen wake, and the same stop reasons
(app in the background, audio focus lost, output changed, clock jump, missed
deadline, 23 hours).

The screen reports skipped logical display frames and the reported audio lead.
It prevents automatic screen lock and raises brightness while active, restoring
both when stopped. Stop/Close cancel queued sound and clear the whole timing picture; a large "Stopped" shows instead.
Expand All @@ -63,7 +86,10 @@ with the calibration pattern instead of the test picture
layers, audio engine and player node; the beep (1.5 kHz, 4 ms) is scheduled
100 ms after each swap on the host clock, minus the reported output latency,
like the chirps. The audio session is play-and-record on the built-in speaker
so the microphone can listen. The result comes back when the screen closes:
so the microphone can listen. Android: the same frame clock and AudioTrack as
the test signal (the beep is placed on the output's timestamps), the
microphone records with AudioRecord meanwhile, and the outline is drawn in
real millimetres from the display's density. The result comes back when the screen closes:
`calibration` (`offsetMs`, `uncertaintyMs`, `pairs`, `directionMs`,
`audioRoute`, `outputLatencyMs`) or `calibrationError`.

Expand All @@ -72,8 +98,8 @@ so the microphone can listen. The result comes back when the screen closes:
The source ID (the QR test id; the audio packet carries its CRC-32) stays the
same when you stop and start the generator again. It is `phone-` plus 12 hex
digits of a hash of a random install ID, the app version with its build
number (for example `2.0.8 (12)`) and the timing settings (now only the event
rate). So you get a new ID after a reinstall, a new app version or build (also a
number (for example `2.0.8 (12)`; Android: version name and version code) and
the timing settings (now only the event rate). So you get a new ID after a reinstall, a new app version or build (also a
new TestFlight build) or another event rate. Name, subtitle and image do not change
it. The install ID is random, not a hardware ID, and only lives in the app's
settings. The container generator uses the same rule with its NMOS seed in
Expand Down Expand Up @@ -124,6 +150,12 @@ Expo Go does not contain this native module. Rebuild/install the iOS development
client after updating this branch (`npm install`, `npx pod-install`, then
`npx expo run:ios` or the existing Xcode workspace/device workflow).

Android, from `mobile-app/android` after `npx expo prebuild -p android`:
`./gradlew :avsync:testDebugUnitTest` (audio placement and screen layout) and,
with a device or emulator, `./gradlew :avsync:connectedDebugAndroidTest`
(`PhoneGeneratorTest`: the picture with Android text and the AVS4 QR read by
ML Kit, the audio packet, the stable ID test vector, the calibration pattern).

```sh
cmake --build build --target avsync-core-tests avsync-picture-tests
ctest --test-dir build --output-on-failure
Expand Down Expand Up @@ -167,12 +199,15 @@ QR, source labels/logo and the early/zero/late sync scale. The square is red
while early, white at the event and green while late; its zero crossing shares
the planned chirp/flash onset. iOS animates it with native display-timestamped
layers over a static plate, with the same coordinates/colors as the renderer.
The bundled font is DejaVu Sans on both targets (CoreText on iOS, Blend2D on
the server); source-event labels describe scheduled phone output.
The bundled font is DejaVu Sans on every target (CoreText on iOS, Android's
own text drawing on Android, Blend2D on the server); source-event labels
describe scheduled phone output. Android draws the same layers in one view.

Name/subtitle and the selected image persist locally. Photos selection uses
the system picker, which shares only the image you select. The app stores a
normalized copy up to 512 px in private Application Support; Remove image
the system picker (Android: the photo picker, or the document picker before
Android 13), which shares only the image you select. The app stores a
normalized copy up to 512 px in private Application Support (Android: the
app's private files); Remove image
removes that copy, preserving the original in Photos. Images are not uploaded
with EAS builds or sent to a server. Labels/image are display identity and
do not change protected timecode/source IDs or calibration.
2 changes: 1 addition & 1 deletion mobile-app/App.js
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ export default function App() {

// With calibrator: step 1 is a camera take (this phone's camera vs microphone), step 2 the
// generator's calibration pattern (this phone's screen vs speaker, sent in the AVS4 QR).
const supportsCalibrator = Platform.OS === 'ios' && Boolean(NativeCamera) && Boolean(native?.supportsGenerator?.());
const supportsCalibrator = Boolean(NativeCamera) && Boolean(native?.supportsGenerator?.());
const startCalibrator = () => { setSheet(null); setTab('measure'); setNotice(null); setFlow(null); setCal({ step: 'plug' }); };
const closeCalibrator = message => { setCal(null); restart(); setNotice(message ?? null); };
const calibratorStart = async base => {
Expand Down
Loading
Loading