feat(photo-exif): swap face stack from tfjs-node to onnxruntime-node by MSIH · Pull Request #273 · MSIH/life-context

MSIH · 2026-07-25T02:00:38Z

Closes #268

Summary

Replaces @tensorflow/tfjs-node + @vladmandic/face-api + canvas (no prebuilt Windows binary for current Node, needs a VS C++ workload to build from source — face detection is currently fully dead on this box) with onnxruntime-node + sharp (both prebuilt, no native compile), running SCRFD detection + ArcFace recognition (buffalo_l models).
Adds connectors/photo-exif/lib/face-align.js: pure, IO-free SCRFD box/keypoint decode, NMS, and a from-scratch Umeyama similarity transform + bilinear warp for 112x112 ArcFace alignment — no BLAS/dep, unit-tested without any model file.
Descriptor changes from a 128-d face-api embedding to a 512-d L2-normalized ArcFace embedding; FACE_MATCH_THRESHOLD/FACE_SEED_THRESHOLD re-tuned (1.0 / 1.15 euclidean) since the old 0.6 value doesn't transfer to the new space.
resolveDetector/fixtureDetector contract is unchanged — the swap is entirely inside loadModelDetector, so face-worker.js and the test suite's fixture-based coverage are untouched.

Changes

connectors/photo-exif/package.json — drop tfjs-node/face-api/canvas, add onnxruntime-node + sharp
connectors/photo-exif/lib/face-align.js (new) — Umeyama, invertAffine, warpTo112, SCRFD distance2bbox/kps, generateAnchorCenters, nms
connectors/photo-exif/lib/face-detect.js — loadModelDetector rewritten for ONNX; single decoded sharp pipeline shared (via .clone()) between the full-res alignment source and the letterboxed detection blob; SCRFD pad value normalized to match the model's calibration (not raw 0)
connectors/photo-exif/lib/face-cluster.js — comment noting descriptors are now unit-normalized (euclidean ≈ cosine)
connectors/photo-exif/face-worker.js, .env.example — new thresholds + FACE_DET_MODEL/FACE_REC_MODEL overrides
connectors/photo-exif/README.md — updated setup/model-download instructions, 128-d → 512-d note
connectors/photo-exif/test.mjs — new model-free unit tests for face-align.js (umeyama/invertAffine round-trip against a known transform, SCRFD decode helpers, warpTo112 shape/range) and l2Normalize

Verification

npm install in connectors/photo-exif — no native compile step (confirmed prebuilt onnxruntime-node/sharp binaries for win32-x64)
npm test — 24/25 passing; the 1 failure (face-worker: scan clusters...) is a pre-existing Windows/Node-v26 libuv crash in the test runner, reproduced identically on unmodified main — not introduced by this change
npm run check:boundary — clean
Ran a solo bug-hunt pass (multi-persona review) equivalent to /pre-pr-review's automated step and fixed two findings before this PR opened: the SCRFD letterbox pad value wasn't normalized like the rest of the canvas, and each photo was being decoded off disk twice (once per helper) instead of once via a shared sharp pipeline.

Test plan

npm install in connectors/photo-exif/ succeeds with no native compile
npm test passes (aside from the pre-existing unrelated flake noted above)
npm run check:boundary passes
Manually: download the buffalo_l ONNX models into FACE_MODELS_PATH, run node face-worker.js against a real photo library, confirm faces cluster and export-thumbnails/suggest-labels behave sanely

…268) Replaces face-api/tfjs-node/canvas (no prebuilt Windows binary for current Node) with onnxruntime-node + sharp running SCRFD detection and ArcFace recognition, keeping the existing detectFaces/resolveDetector seam intact. Adds lib/face-align.js (Umeyama alignment, SCRFD decode, bilinear warp) with model-free unit tests, and re-tunes FACE_MATCH_THRESHOLD/FACE_SEED_THRESHOLD for the new 512-d normalized descriptor space.

… once Pre-PR review caught two issues in the #268 ONNX swap: the letterboxed detection blob left padded pixels at raw 0 instead of the model's normalized pad value ((0-127.5)/128), and decodeRgb/letterbox each independently re-decoded the same file from disk. Fills the pad value correctly and shares one sharp pipeline (.clone()) across both outputs.

Copilot

Pull request overview

This PR updates the photo-exif connector’s face detection/recognition pipeline by replacing the prior tfjs/face-api/canvas stack with an ONNX Runtime + sharp implementation, adding pure alignment/decoding math utilities and accompanying unit tests to keep the connector runnable on modern Node/Windows without a native build toolchain.

Changes:

Swaps face detection/embedding runtime to onnxruntime-node + sharp, with SCRFD detection and ArcFace (512-d, L2-normalized) descriptors.
Adds a new pure math module (lib/face-align.js) for SCRFD decode + Umeyama alignment + 112×112 warping, and adds model-free unit tests for it.
Updates thresholds/config/docs to reflect the new embedding space (128-d → 512-d) and tuned match/seed defaults.

Reviewed changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
connectors/photo-exif/package.json	Replaces face-api/tfjs/canvas deps with `onnxruntime-node` + `sharp`.
connectors/photo-exif/package-lock.json	Locks the updated dependency tree for the new ONNX/sharp stack.
connectors/photo-exif/lib/face-detect.js	Rewrites the real detector to run SCRFD + ArcFace via ONNX Runtime, using sharp for preprocessing and returning normalized descriptors.
connectors/photo-exif/lib/face-align.js	New pure alignment/decoder utilities (NMS, SCRFD decode, Umeyama, warpTo112).
connectors/photo-exif/lib/face-cluster.js	Documents the new descriptor properties (unit-normalized 512-d) and why clustering logic remains valid.
connectors/photo-exif/face-worker.js	Updates default thresholds for the new normalized ArcFace descriptor space.
connectors/photo-exif/.env.example	Adds model filename overrides and documents new threshold semantics/defaults.
connectors/photo-exif/README.md	Updates setup/model download instructions and clarifies the new face stack and descriptor change.
connectors/photo-exif/test.mjs	Adds model-free unit tests for alignment/decoder utilities and `l2Normalize`.

Files not reviewed (1)

connectors/photo-exif/package-lock.json: Generated file

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    "exifr": "^7.1.3",
+    "onnxruntime-node": "^1.27.0",
+    "sharp": "^0.35.3"


      "dependencies": {
-        "@tensorflow/tfjs-node": "^4.22.0",
-        "@vladmandic/face-api": "^1.7.15",
-        "canvas": "^3.1.0",
-        "exifr": "^7.1.3"
+        "exifr": "^7.1.3",
+        "onnxruntime-node": "^1.27.0",
+        "sharp": "^0.35.3"
      },


+import { umeyama, invertAffine, applyAffine, distance2bbox, distance2kps, generateAnchorCenters, nms, warpTo112, DST_112 } from './lib/face-align.js';
+import { l2Normalize } from './lib/face-detect.js';

 const __dirname = path.dirname(fileURLToPath(import.meta.url)); // import.meta.dirname needs Node 20.11+; this connector declares >=18


+  DET_STRIDES.forEach((stride, idx) => {
+    const scores = outputs[idx];
+    const bboxPreds = outputs[idx + DET_STRIDES.length];
+    const kpsPreds = outputs[idx + DET_STRIDES.length * 2];
+    const anchors = generateAnchorCenters(DET_INPUT_SIZE, DET_INPUT_SIZE, stride, DET_NUM_ANCHORS);
+    for (let i = 0; i < anchors.length; i++) {
+      const score = scores[i];
+      if (score < DET_SCORE_THRESHOLD) continue;
+      const bd = [bboxPreds[i * 4] * stride, bboxPreds[i * 4 + 1] * stride, bboxPreds[i * 4 + 2] * stride, bboxPreds[i * 4 + 3] * stride];
+      const kd = [];
+      for (let k = 0; k < 10; k++) kd.push(kpsPreds[i * 10 + k] * stride);
+      candidates.push({ box: distance2bbox(anchors[i], bd), kps: distance2kps(anchors[i], kd), score });
+    }
+  });


…9.0) Copilot review on #273 caught that sharp@0.35.3 requires Node >=20.9.0, but package.json still declared >=18.12.0 — misleading metadata that could cause a confusing install failure on Node 18. Bumps engines.node and regenerates the lockfile's engines metadata to match.

MSIH · 2026-07-25T08:06:17Z

Resolving Copilot's review (4 comments):

Accepted (3) — package.json/package-lock.json/test.mjs: engines.node still said >=18.12.0, but sharp@0.35.3 actually requires >=20.9.0 (checked node_modules/sharp/package.json directly). Bumped engines.node to >=20.9.0, regenerated the lockfile's engines metadata, and updated the test.mjs comment that referenced the old floor. Fixed in f759c92.
Deferred (1) — lib/face-detect.js:91, the per-image generateAnchorCenters allocation: valid minor GC/allocation point (~16.8k [x,y] pairs rebuilt per photo scanned), but converting it to an index→point closure would change the exported, already-tested pure-math contract in lib/face-align.js for marginal benefit at this scale (a batch worker, not a request path). Not worth the structural churn in this PR; revisiting if face-worker throughput on a real library ever shows this as a hotspot.

…282) * chore(photo-exif): allowlist onnxruntime-node postinstall script Leftover from #273's tfjs-node -> onnxruntime-node swap; needed to unblock branching for #280. * fix(photo-exif): shared HEIC-capable image decode for both AI passes Add lib/decode-image.js (sharp-primary, heic-decode/libheif-WASM fallback, orientation-safe) so the face and caption workers can read pixels from the ~47% of this library's photos sharp's bundled libheif can't decode. Caption worker also survives an isolated VLM failure now, stopping only after VLM_MAX_CONSECUTIVE_FAILURES in a row instead of on the first one. Closes #280 * fix(photo-exif): clamp VLM_MAX_CONSECUTIVE_FAILURES to a minimum of 1 Found in pre-PR review: a value of 0 (or negative) broke the caption worker before its first VLM call, since the failure counter starts at 0 and 0 >= 0 is already true. Clamp to the default (5) below 1. Refs #280 * fix(photo-exif): force sharp's lazy decode so mislabeled files actually fall back Copilot review on PR #282: sharp defers real decode errors until an output op runs, so sharp(absPath).rotate() never threw for an undecodable file — the fallback to heic-decode for a mislabeled extension (e.g. a real HEIC saved as .jpg) never actually triggered, the same trap as sharp.metadata(). Force the decode on a clone before returning. Also fixes a misleading "fallback" log line for the .heic/.heif dispatch-first case. Refs #280 * fix(photo-exif): throttle VLM retries on failure; avoid a copy in the heic-decode path Second Copilot review round on PR #282: - caption-worker.js's VLM-failure catch continue'd past the throttle sleep, so repeated failures hammered the VLM endpoint back-to-back instead of pacing like a success does. Throttle applies on failure too now, verified by a timing-gap test. - decode-image.js's heic-decode path copied the whole decoded RGBA buffer via Buffer.from(typedArray); use a zero-copy Buffer view over the same ArrayBuffer instead (a 12MP RGBA decode is already ~48MB). Refs #280 --------- Co-authored-by: Barry Schneider <bschneider@msih.com>

barry8schneider added 2 commits July 24, 2026 21:56

MSIH requested a review from Copilot July 25, 2026 07:59

Copilot started reviewing on behalf of MSIH July 25, 2026 07:59 View session

Copilot AI reviewed Jul 25, 2026

View reviewed changes

MSIH merged commit 452ed00 into main Jul 25, 2026
5 checks passed

MSIH deleted the feat/268-onnx-face-detect branch July 25, 2026 08:12

This was referenced Jul 25, 2026

feat(photo-exif): suggest cluster labels from single-name Google Photos sidecars #274

Merged

chore(photo-exif): tag people in the photo library (face pass) #217

Open

fix(photo-exif): caption pass must preserve face enrichment #277

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(photo-exif): swap face stack from tfjs-node to onnxruntime-node#273

feat(photo-exif): swap face stack from tfjs-node to onnxruntime-node#273
MSIH merged 3 commits into
mainfrom
feat/268-onnx-face-detect

MSIH commented Jul 25, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

MSIH commented Jul 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

MSIH commented Jul 25, 2026

Summary

Changes

Verification

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

MSIH commented Jul 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants