chore(photo-exif): tag people in the photo library (face pass)

[MSIH]

Now actionable (2026-07-25). Both things that blocked this are resolved: the face-detection rewrite landed (#268 / PR #273), and the owner decided this pass runs first, ahead of photo descriptions. The description job was split out into #275.

⛔ Blocked by #280 — do not run this pass yet

Added 2026-07-25 after review. This pass cannot see HEIC photos, and it fails silently: detectFaces throws on every HEIC and face-worker.js:178-181 logs the error and continues. The job would finish, report a face count, and never tell you it skipped 62,614 photos — 47% of the library.

Nothing on this machine can decode a HEIC today (measured 2026-07-25): sharp's bundled libheif fails on all sampled files, and there is no ffmpeg, ImageMagick, or Windows HEIF codec. Ingestion worked only because exifr reads EXIF metadata without decoding pixels — this pass is the first code that needs the pixels.

The gap is concentrated in exactly the years with the most people in frame:

year	HEIC	other	HEIC share
2019	0	8,738	0%
2021	10,338	10,481	50%
2022	12,765	3,602	78%
2023	16,314	6,001	73%
2025	6,654	4,909	58%
2026	3,202	1,374	70%

Running before #280 lands means person search works well for 2015-2019 and goes progressively blind through the years you have the most photos of people — with no error to tell you so.

One piece of good news: the continue fires before the photo is written to the state file, so skipped photos are never marked done. Land #280 and re-run — everything is picked up, with no state to clear. But it is cheaper to land #280 first than to run this pass twice.

#280 also fixes the same blocker for #275, which needs it too.

Problem

The photo library — about 129,000 photos — can currently be searched only by when and where a photo was taken, plus the folder it came from. You can't ask for photos of a particular person.

There's an already-built job (connectors/photo-exif/face-worker.js) that fixes this. It finds faces in each photo, groups the ones that look like the same person into anonymous piles, and then — once you tell it "that pile is Amy" — tags every photo in that pile with her name. Those tags are what make "show me photos of Amy" work.

It has never run on this machine — verified 2026-07-25: zero photos have any face information.

Why it's runnable now

Two things were in the way, both now cleared:

1. The face-detection software wouldn't install. The old setup needed a component with no prebuilt version for this machine's Node version, and building it from scratch needed a Visual Studio C++ toolchain that isn't installed. feat(photo-exif): swap face stack from tfjs-node to onnxruntime-node #268 replaced that whole stack with one that ships ready-to-run binaries. No build step anymore.
2. It was waiting on the description job. That job turns out to be a months-long grind (see chore(photo-exif): write AI descriptions for the photo library (timing test first) #275). Rather than keep the fast, valuable half hostage to the slow half, the owner decided to run this pass first.

Ordering: this runs first, descriptions run second

Both jobs replace a photo's stored text and small metadata record rather than adding to them — so whichever runs second overwrites some of what the first one wrote. Understanding that tradeoff is the whole reason ordering matters.

Chosen order — faces first:

Week 1	This pass runs (hours) → person search works
Week 1+	You name the piles you recognize
Months after	Descriptions (#275) fill in gradually overnight
Months after	Nothing to redo — #276 made the two passes preserve each other

No longer overwritten (#276, merged 2026-07-25). The description job used to wipe this pass's bookkeeping — the detected-face count and the "Pictured: …" sentence. Both jobs now build their photo record through one shared component, so each carries the other's work through. Nothing is lost in either direction and no re-run is needed.

One rule remains: run the two passes one at a time, never overlapping. Each reads the other's state once when it starts, so naming a face while a description run is in flight would still be missed by that run.

(Faces-first was chosen on 2026-07-25 to avoid delaying person search by months. Since #276 the order no longer costs anything either way.)

Current state (verified 2026-07-25)

Thing	State
Photos in scope	129,143
— of those, with face info	0
— of those, from Google Photos (so they have Google's own people tags)	129,127 (99.99%)
Photos not in scope (came from Messages, outside the photo folder)	5,395
Face models downloaded?	no — FACE_MODELS_PATH is unset
Job's progress files	absent
Photo import finished?	yes — nothing new since 2026-07-17

How naming the piles works — two paths, cheap one first

Because 99.99% of the library came from Google Photos, most photos carry Google's own people tags. That makes the cheap path the primary one:

1. From Google's tags (feat(photo-exif): suggest cluster labels from single-name Google Photos sidecars #272 — free, no AI, no network). Where a photo has exactly one face and Google tagged exactly one person, that's a vote for "this pile is that person." Piles with a clear enough majority get a suggested name printed. This should name most piles for nothing.
2. From your contact photos (feat: suggest cluster labels from contact photos in photo-exif face-worker #84 — fallback). For piles the first path can't name, compare against the photos on your contacts.

Both paths only ever print suggestions. Neither ever applies a name by itself — you confirm each one. A wrong automatic name would pollute the contact graph worse than leaving the pile anonymous.

Implementation Plan

0. Land fix(photo-exif): shared HEIC-capable image decode for both AI passes #280 first (HEIC decode). Without it this pass silently skips 47% of the library — see the blocked-by note at the top. Everything below assumes fix(photo-exif): shared HEIC-capable image decode for both AI passes #280 has merged.

1. Download the two face models into a folder and point FACE_MODELS_PATH at it (currently unset in connectors/photo-exif/.env): det_10g.onnx (finds faces) and w600k_r50.onnx (recognizes them), the buffalo_l pair from the InsightFace model zoo. One-time download; fully offline afterward.

2. Install the connector's dependencies: npm install in connectors/photo-exif/. No compiler needed — both packages ship prebuilt binaries (this is what feat(photo-exif): swap face stack from tfjs-node to onnxruntime-node #268 fixed).

3. Run the pass: node face-worker.js from connectors/photo-exif/. No pause between photos by default, so it runs at full speed. Saves progress after every photo and resumes, so killing it mid-run is safe. Watch the log for face detection failed for ... lines — a run producing tens of thousands of them means fix(photo-exif): shared HEIC-capable image decode for both AI passes #280 didn't take, and the job will still exit reporting success.

4. Get name suggestions, cheap path first: node face-worker.js suggest-from-sidecars, then node face-worker.js export-thumbnails ./faces and suggest-labels for whatever's left unnamed.

5. Confirm the names you recognize: node face-worker.js label <id> "<name>" for each. This is the step that actually creates the person tags.

6. No follow-up re-run needed — fix(photo-exif): caption pass must preserve face enrichment #276 made the description job preserve this pass's output. Just don't run the two passes at the same time.

Acceptance Criteria

• The face pass completes over the library; the count of photos with face information climbs from 0 and is reported.
• HEIC photos were actually examined — roughly 47% of the photos with face information are .heic, and the run did not emit a large volume of face detection failed lines. Without this, a run that skipped half the library still passes the criterion above.
• suggest-from-sidecars runs and its suggestion count is reported (how many piles Google's tags named for free).
• At least one pile is confirmed with a real name, and that person's photos then surface when searching for them.
• No photo gets duplicated — face info attaches to the existing photo record.
• The date, location, place name, and original-file pointer on each photo are unchanged after the job touches it.
• Store→recall smoke test still green (server boots, 0 errors).

Out of Scope

• Photo descriptions — now chore(photo-exif): write AI descriptions for the photo library (timing test first) #275.
• Naming piles from a different machine than the server (needs a shared disk, feat: suggest cluster labels from contact photos in photo-exif face-worker #84).
• Tuning how aggressively faces get grouped, or description quality — separate follow-ups if results are noisy.
• The 5,395 photos that came from Messages. They live outside the photo folder this job walks, so it will never see them. Not unfinished backlog here.

[MSIH]

Status check (2026-07-23)

Queried life-context.db:

• Caption pass: extra_json LIKE '%captioned%' → 129,143 photo/video artifacts — the caption-worker pass has run and is essentially complete across the backlog.
• Face pass: extra_json LIKE '%faces_detected%' → 0 rows — the face-worker pass has not been run at all yet.

Wave order (scan → caption → face) is respected so far: captioning is done, so a subsequent face pass would correctly reconstruct text_repr as base + caption + "Pictured: …" per the design decision in this issue. Narrowing scope to the remaining work — retitling to track only the face-enrichment pass.

[MSIH]

Investigation (2026-07-24) — face pass BLOCKED; premise corrected

A worth-it triage picked this up as an activation task. A Sonnet ops agent tried to run the pass and found two things:

1. Native build blocker (tfjs-node). @tensorflow/tfjs-node@4.22.0 will not load on this box: no prebuilt Windows binary for Node v26.3.0 (napi-v8 / ABI 147) — node-pre-gyp GET → 404 — and the source-build fallback fails because VS2026 lacks the "Desktop development with C++" workload. lib/face-detect.js imports tfjs-node unconditionally (no pure-JS fallback), so face detection is a full stop as configured. (canvas rebuilt fine; face-api model weights are already bundled in node_modules/@vladmandic/face-api/model/.)

2. The "captions are complete" record is FALSE. Direct read-only DB query: 0 rows with extra_json.captioned = true; 129,143 = false, 5,395 = null. The 2026-07-23 status comment's LIKE '%captioned%' matched the captioned:false placeholder — a false positive. The caption-cache state file (~/.life-context/photo-exif-captions.json) does not exist. caption-worker has never completed on this machine. Per wave order (scan → caption → face), the face pass must NOT run until real captions land — otherwise caption-worker's later extra_json upsert would wipe any faces_detected / pictured data.

Disposition: parked (worth-it:DO → worth-it:DEFER). Durable fix = drop tfjs-node for onnxruntime-node (actively-maintained prebuilt binaries across recent Node/napi on Windows) — tracked in a dedicated migration issue (link to follow). Reactivate this face pass once (a) the ONNX migration lands, and (b) the caption pass has actually run to completion.

[MSIH]

Reconciliation (2026-07-25) — title corrected, body rewritten against live state

Re-verified this issue against the live DB and current code after #268/#272 merged. Several claims had gone stale or were outright wrong; body and title updated.

Verified, by direct query (not LIKE)

Using json_extract rather than the substring match that produced the 2026-07-23 false positive:

Check	Result
photo+video artifacts	134,538
— in scope (google-photos 129,127 + photo-exif 16)	129,143
— out of scope (imessage, outside PHOTO_ROOT)	5,395
extra.captioned = true	0
extra.captioned = false	129,143
extra.faces_detected present	0
Last photo ingest (scan drained?)	2026-07-17 ✅
~/.life-context/ caption cache / face state	absent (only the scan manifest)

The 2026-07-24 investigation comment was correct; the 2026-07-23 one was not.

What was wrong

1. The title claimed "(caption pass complete)". It never was — 0 captioned. Corrected; this was the most misleading thing in the issue.
2. Implementation Plan step 3 described the pre-feat(photo-exif): swap face stack from tfjs-node to onnxruntime-node #268 stack — ssdMobilenetv1/faceLandmark68Net/faceRecognitionNet weights and @vladmandic/face-api/@tensorflow/tfjs-node/canvas. All gone. Now: det_10g.onnx + w600k_r50.onnx into FACE_MODELS_PATH, deps onnxruntime-node + sharp, no native build. Design-Decisions bullet 2 was stale the same way.
3. No mention of feat(photo-exif): suggest cluster labels from single-name Google Photos sidecars #272. Since 129,127/129,143 in-scope artifacts are google-photos, essentially the whole library has Takeout sidecars — suggest-from-sidecars is the natural primary labeling path, with suggest-labels as fallback. Both are now in the plan in that order.
4. Backlog size was "~84k" — actual in-scope count is 129,143.
5. The 5,395 imessage photos were unaccounted for. Neither worker walks them (outside PHOTO_ROOT); now explicitly listed as out of scope so they aren't later mistaken for unenriched backlog.

What's new

Scale is the real blocker, and it was never stated. VLM_THROTTLE_MS defaults to 2000 ms → 129,143 × 2 s = 71.7 hours of throttle sleep alone, before any inference. With llava-7B on this box (Ryzen 8745H, integrated 780M, no discrete GPU), the README's 4-hour nightly window plausibly turns this into a months-long job. Added an explicit measure-first step: pull llava, time ~50 photos, record a real ETA on this issue, and only then choose throttle/model/cadence. No VLM is currently pulled at all.

Wave order is now an open question, not an inherited assumption. The face pass is cheap (FACE_THROTTLE_MS defaults to 0, no VLM) while captions are the multi-month grind. Verified the cost of running faces first: caption-worker.js posts extra: { captioned: true } and extra is replaced wholesale (doc 04 §3), so a later caption run would wipe faces_detected/pictured and the "Pictured: …" prose — but the entity_links rows survive (additive via resolveEntityHints), which is the primary recall path. Recorded as an explicit tradeoff; not reordering unilaterally.

Acceptance criteria are now progress-based rather than all-or-nothing, since this is a long-running job.

Label status

Keeping worth-it:DEFER, but the rationale has changed: of the two reactivation triggers, (a) ONNX migration — DONE (#268 / PR #273); (b) caption pass complete — still not met. The face pass is technically unblocked now; what holds it is the wave-order dependency plus the unmeasured caption runtime. Status banner added to the body.