feat(import): 1Password metadata-only pull + mapper (EF-ISS-8 P1)

scripts/import-1password.sh runs two read-only `op` calls (vault list,
item list), holds the result in memory, and pipes it through
scripts/map-1password.py — a pure JSON-in/JSON-out mapper that reads five
fields per item (title, category, urls, vault, updated_at) and drops
everything else, including additional_information's username/email hints.
Failures exit with their own codes and plain-English advice: 3 op absent,
4 no signed-in account, 5 refused, 6 no python3. A failed pull prints
nothing rather than a truncated candidate list.

Tests: stub `op` + fixtures + 38 assertions (mapping, ranking, dedup,
wrapper failure modes, imported entries validating clean on both tiers),
plus a literal grep guard so unmasking flags and per-item fetches can
never appear in the repo. Suite 146 pass / 0 fail, shellcheck clean.

Verified twice against the real 563-item vault, which found three defects
fixtures would not have: substring keyword matching ranked "Readwise" as
finance; a finance-looking secure note outranked real accounts; and 178
of 563 items share a title with another item, so they would share a
pointer and re-import would hide the twin (now flagged as
title_collisions for P2's wizard to resolve).

Spec corrections recorded in ideation/1password-import-spec.md §11: the
"1Password > {title}" pointer belongs in access_pointer, not identifier
(which the owner edits with a last-4, breaking dedup), and priority uses
the schema's `normal`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: TheWayWithin

CLAUDE.md

@@ -39,15 +39,26 @@ All specs live in `ideation/` (moved from repo root, 17 Jul 2026):
 - **EF-ISS-8 (open, project): 1Password import.** Spec committed at
   `ideation/1password-import-spec.md` — metadata-only `op` CLI pull, wizard
   + bulk triage in the browser editor, dedup for review-time re-import.
-  **P0 spike DONE 29 Jul** against the real vault (findings in spec §9:
-  list output is clean of secrets; drop `additional_information`; 94% of a
-  real vault is generic Login so ranking needs the finance heuristic).
-  Owner machine ready: op 2.35.0 + 1Password 8 CLI integration on.
-- **Pickup sequence (agreed 29 Jul 2026):**
-  1. Build EF-ISS-8 **P1** — `scripts/import-1password.sh` (pull + mapper,
-     candidate JSON, stub-`op` CI tests; spec §6-7).
-  2. **P2** — editor integration: /import endpoint, vault picker, wizard
-     with filter box + bulk mode, seal-time honesty gate.
+  **P0 spike DONE 29 Jul** against the real vault (spec §9: list output is
+  clean of secrets; drop `additional_information`; 94% of a real vault is
+  generic Login so ranking needs the finance heuristic).
+  **P1 BUILT 29 Jul** (spec §11): `scripts/import-1password.sh` (pull,
+  in-memory, exit codes 3/4/5/6 for op-absent / no-account / refused /
+  no-python) + `scripts/map-1password.py` (pure mapper → candidate JSON),
+  `tests/stub-op/op` + fixtures, 38 tests, suite 146 pass / 0 fail, plus
+  the metadata-only grep guard. Verified twice against the real 563-item
+  vault. Spec corrections made in P1: the `1Password > {title}` pointer
+  lives in **`access_pointer`** (not `identifier`), priority uses the
+  schema's `normal`, and `title_collisions` flags the 178-of-563 items
+  that share a title with another item.
+- **Pickup sequence (agreed 29 Jul 2026, P1 now done):**
+  1. ~~EF-ISS-8 **P1** — pull + mapper + stub-`op` tests.~~ DONE 29 Jul.
+  2. **P2 (next)** — editor integration: `GET /import/1password` in
+     edit-server.py, vault picker, wizard with filter box + bulk mode,
+     seal-time honesty gate. Must handle `title_collisions` (ask for a
+     last-4 in `identifier`) and default the wizard to `default_include`
+     with a "show everything" toggle. The importer needs no re-pull for
+     the toggle — hidden candidates are already in the payload.
   3. **T-159 Session A** — Jamie builds his REAL register, using the
      import wizard as its first genuine UAT (dogfood on 563 items).
   4. **Session B** (docs/UAT-PLAN.md) — seal + print, PLUS the 1Password

ideation/1password-import-spec.md

@@ -4,7 +4,8 @@
 categories, and URLs only, never secrets — then triage in the browser editor,
 either one item at a time (wizard) or all at once (bulk), before sealing.
 
-Status: SPEC. Written 2026-07-28. Owner: Jamie. Register row: EF-ISS-8.
+Status: **P1 BUILT 2026-07-29** (pull + mapper + tests; findings in §11).
+P2-P4 still spec. Written 2026-07-28. Owner: Jamie. Register row: EF-ISS-8.
 
 ---
 
@@ -126,8 +127,18 @@ Every field is a *guess presented for confirmation*, pre-filled never
 auto-final in wizard mode. Other mapped fields:
 
 - `provider` ← item title
-- `identifier` ← `1Password > {title}` — the pointer style the design wants;
-  never a number, never a URL with query strings
+- `access_pointer` ← `1Password > {title}` — the pointer style the design
+  wants (and the exact example the editor already shows for this field).
+  **Corrected in P1:** the first draft of this spec put the pointer in
+  `identifier`, which is the field for a last-4 or a reference; the pointer
+  belongs in `access_pointer`, and dedup (§5) is stronger there because the
+  owner edits `identifier` (adding last-4 digits) and would break the key.
+- `identifier` ← the URL host if the item has one, else the title. A
+  searchable reference, never a number, never a path or query string.
+- `priority` ← per the table; the schema's middle value is `normal`, not
+  "medium". A finance heuristic hit (§9.5) raises it to `high`.
+- `ownership` ← `sole` (the commonest case, and a required field — the
+  wizard shows it for correction)
 - `status` ← `active`; `last_confirmed` ← today (the owner is looking at it
   right now)
 - `action_notes` ← wizard: owner-written or a sensible template; bulk: the
@@ -138,26 +149,49 @@ not the engine — see the finance-heuristic ranking consequence there.
 
 ## 5. Dedup contract
 
-- Match key: normalised `identifier` equal to `1Password > {title}`
+- Match key: normalised `access_pointer` equal to `1Password > {title}`
   (case-insensitive, whitespace-collapsed). Already-present matches are
   filtered out of the candidate list before either mode shows anything.
+  The importer also matches the key against existing `identifier` values,
+  so registers written before this feature still dedup.
 - Soft warning on near-misses: candidate title ≈ existing `provider`
   (exact match after normalisation) shows "possibly already listed as A007"
   instead of silently duplicating.
+- **Title clashes (P1 finding, §11):** two vault items can carry the same
+  title, and then they carry the same pointer — so next year's re-import
+  would hide the second one. The mapper flags those candidates with
+  `title_collisions: N`; the wizard (P2) must ask for something
+  distinguishing in `identifier` (last 4 digits) rather than silently
+  accepting twins. On a real vault this is not an edge case: 178 of 563.
 - Dedup is what makes the review-time re-import (§3) work with zero extra
   machinery.
 
 ## 6. Architecture
 
 Three small pieces, following the existing edit-server pattern:
 
-1. **`scripts/import-1password.sh`** — the pull. Checks `op` exists and is
-   signed in, runs `op item list --format=json` (plus `op vault list` for the
-   picker), maps to a manager-agnostic **candidate JSON** on stdout:
-   `{source, pulled, vaults:[...], candidates:[{title, category, url_host,
-   vault, suggested:{provider, type, identifier, priority, status,
-   last_confirmed, preferred_action, action_notes}}]}`.
-   Pure, deterministic given `op` output → trivially testable with a stub.
+1. **`scripts/import-1password.sh`** — the pull (BUILT). Checks `op` exists
+   and is signed in, runs `op vault list --format=json` for the picker and
+   `op item list --format=json` for the items, and pipes the result through
+   the mapper to stdout. Nothing touches disk: the pull is held in memory,
+   and a failed pull prints nothing rather than a truncated list. Exit
+   codes carry the failure mode: 3 = op absent, 4 = no signed-in account,
+   5 = the pull was refused, 6 = python3 missing.
+   **`scripts/map-1password.py`** — the mapping (BUILT), split out because
+   it is pure: JSON in, JSON out, no `op`, no network, so the whole mapping
+   contract is testable from a fixture. It reads exactly five fields per
+   item (title, category, urls, vault, updated_at) and drops the rest,
+   including `additional_information`.
+   The **candidate JSON** it emits:
+   `{source, pulled, vaults:[{name, items}], counts:{items, candidates,
+   shown_by_default, hidden, deduped, other_vaults}, candidates:[{title,
+   category, url_host, vault, updated, rank, default_include,
+   possible_duplicate_of?, title_collisions?, suggested:{provider, type,
+   identifier, priority, ownership, status, last_confirmed,
+   preferred_action, action_notes, access_pointer}}]}`.
+   `rank` is the money-first ordering (0-3 accounts, 4-5 non-accounts) and
+   `default_include` is the "show everything" toggle — non-accounts are
+   emitted, not withheld, so the toggle needs no second pull.
    The manager-agnostic shape is deliberate: a future Bitwarden importer
    (`bw list items`) plugs in behind the same contract (parked, not built).
 2. **`web/edit-server.py`** — one new endpoint, `GET /import/1password`
@@ -179,17 +213,22 @@ the tail skippable).
 ## 7. Build phases
 
 - **P0 — empirical spike. DONE 2026-07-29, findings in §9.**
-- **P1 — pull + mapping.** `import-1password.sh`, candidate JSON contract,
-  stub-`op` CI fixture, mapping + dedup unit tests. Drops
-  `additional_information`, adds the finance-heuristic ranker (§9).
+- **P1 — pull + mapping. DONE 2026-07-29**, findings in §11.
+  `import-1password.sh` + `map-1password.py`, candidate JSON contract,
+  stub-`op` fixture, 38 tests. Drops `additional_information`, adds the
+  finance-heuristic ranker (§9). The §2.1 grep guard landed here rather
+  than in P4: a boundary is worth more from the moment the code exists.
 - **P2 — editor integration.** Endpoint, vault picker, both modes, the
   wizard filter box (§9), seal-time honesty gate. Deterministic test via the
   existing edit-server harness with the stub `op` on PATH.
 - **P3 — review-time re-import.** Same button in review mode, dedup against
   the decrypted register, "new since last pull" framing.
-- **P4 — docs + guards.** README + site get-started ("have 1Password? two
-  minutes instead of twenty"), SECURITY.md boundary note (§2), AGENTS.md
-  line, CI grep test that `--reveal`/concealed-field reads appear nowhere.
+- **P4 — docs.** README + site get-started ("have 1Password? two minutes
+  instead of twenty"), SECURITY.md boundary note (§2), AGENTS.md line, and
+  the 1Password 8 + CLI-integration prerequisite (§9.7). The CI grep guard
+  it used to own shipped with P1. Docs must state the boundary in plain
+  English **without quoting the forbidden flags** — the guard is a literal
+  grep, and it should stay dumb enough that nothing can talk it round.
 
 Rough size: comparable to the browser-seal build (EF-ISS-7) — a day of
 sessions.
@@ -255,3 +294,36 @@ category as a hint; P2's wizard gains the filter box.
 - Other password managers (Bitwarden etc.) — the candidate JSON contract is
   designed for them, but none are built now.
 - Windows-native `op` flows (owner tooling is macOS/Linux/WSL, as today).
+
+## 11. P1 build findings (2026-07-29)
+
+Built `scripts/import-1password.sh` + `scripts/map-1password.py`, a stub
+`op` (`tests/stub-op/op`) with fixtures, and 38 tests. Suite: 146 pass, 0
+fail. Verified twice against the real 563-item vault, not only fixtures —
+which is where three defects surfaced that fixtures never would have:
+
+1. **Substring keyword matching is too loose.** "Readwise" matched the
+   finance hint "wise" and ranked as money. Fixed: hints match from a word
+   boundary (so "hargreaveslansdown.com" still hits), and the short
+   ambiguous ones (isa, ira, irs, tax, wise, visa, loan) must match whole
+   words. Real-vault rank-1 count went 48 → 40.
+2. **Non-accounts could outrank accounts.** A secure note called "Trader-7
+   Coinbase API" scored a finance hit and sorted above real accounts while
+   being hidden by default. Fixed: accounts occupy ranks 0-3, non-accounts
+   4-5 — so the "show everything" toggle appends to the list instead of
+   reshuffling it.
+3. **Title clashes are common, not rare: 178 of 563 items share a title**
+   with another item (the same messy vault behind T-284). They would share
+   a pointer, so re-import would hide the twin. Mapper now emits
+   `title_collisions`; P2's wizard must act on it (§5).
+
+Also confirmed live: the not-signed-in and approval-timeout paths are real
+and exit cleanly (the vault session expired mid-session and the wrapper
+said so in plain English, printing nothing on stdout). Real-vault shape:
+563 items → 540 shown by default, 23 hidden, 7 at rank 0, 40 at rank 1,
+493 generic. That 493 is the tail the wizard's filter box (§9.6) exists
+for, and the strongest argument for P2 taking ergonomics seriously.
+
+Deviations from the spec as written, both deliberate and both above:
+`access_pointer` carries the pointer instead of `identifier` (§4), and the
+grep guard shipped in P1 instead of P4 (§7).

scripts/import-1password.sh

@@ -0,0 +1,127 @@
+#!/usr/bin/env bash
+# import-1password.sh — seed the register from your 1Password vault:
+# titles, categories and web addresses only, never passwords (EF-ISS-8).
+#
+# Usage:
+#   scripts/import-1password.sh [OPTIONS] > candidates.json
+#
+# Options:
+#   --vaults-only     list your vaults and stop (for the editor's picker)
+#   --vault NAME      only include items from this vault (repeatable)
+#   --existing FILE   JSON {"assets":[{id, provider, access_pointer}]} of the
+#                     register being edited, so items already listed are
+#                     dropped from the result
+#   --today DATE      override today's date (YYYY-MM-DD; tests)
+#
+# WHAT THIS READS: `op item list` returns item titles, categories, web
+# addresses, vault names and timestamps — it does NOT return passwords,
+# one-time codes, or any other concealed field, and this script never asks
+# for one. It makes exactly two read-only calls (the vault list and the item
+# list) and never fetches an individual item or asks 1Password to unmask
+# anything; a CI test greps the whole repo to keep it that way. The
+# 1Password CSV/1PUX export is deliberately NOT used: exporting writes your
+# secrets to disk, which is exactly what this project exists to avoid.
+#
+# Nothing is written to disk: the pull is piped straight into the mapper and
+# the candidate list goes to stdout. The browser editor holds it in the tab
+# until you accept entries into the register.
+#
+# Needs: the 1Password CLI (`op`) signed in — 1Password 8+ desktop app with
+# "Integrate with 1Password CLI" turned on, and a 1password.com account
+# (standalone vaults are not reachable by the CLI). This is owner-side
+# tooling; the executor recovery path gains no dependency from it.
+#
+# Exit codes: 0 = candidates written, 2 = usage, 3 = op not installed,
+# 4 = op installed but no account / not signed in, 5 = the pull failed
+# (approval denied, app locked, …), 6 = python3 missing.
+set -u
+set -o pipefail
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+MAPPER="$HERE/map-1password.py"
+
+VAULTS_ONLY=0
+MAP_ARGS=()
+while [ $# -gt 0 ]; do
+  case "$1" in
+    -h|--help) sed -n '2,34p' "$0"; exit 0 ;;
+    --vaults-only) VAULTS_ONLY=1; shift ;;
+    --vault)   [ $# -ge 2 ] || { echo "error: --vault needs a name" >&2; exit 2; }
+               MAP_ARGS+=(--vault "$2"); shift 2 ;;
+    --existing) [ $# -ge 2 ] || { echo "error: --existing needs a file" >&2; exit 2; }
+               MAP_ARGS+=(--existing "$2"); shift 2 ;;
+    --today)   [ $# -ge 2 ] || { echo "error: --today needs a date" >&2; exit 2; }
+               MAP_ARGS+=(--today "$2"); shift 2 ;;
+    *) echo "error: unknown option: $1" >&2; echo "try: $0 --help" >&2; exit 2 ;;
+  esac
+done
+
+if ! command -v op >/dev/null 2>&1; then
+  cat >&2 <<'EOF'
+error: the 1Password command-line tool (op) is not installed.
+
+  macOS:   brew install 1password-cli
+  other:   https://developer.1password.com/docs/cli/get-started/
+
+Then, in the 1Password 8 app: Settings > Developer > "Integrate with
+1Password CLI". You can also skip this and type your accounts in by hand —
+the importer only ever saves you typing.
+EOF
+  exit 3
+fi
+
+if ! command -v python3 >/dev/null 2>&1; then
+  echo "error: python3 is needed to map the pull (owner-side tooling only)." >&2
+  exit 6
+fi
+if [ ! -f "$MAPPER" ]; then
+  echo "error: mapper not found at $MAPPER" >&2
+  exit 6
+fi
+
+ERR="$(mktemp)"
+trap 'rm -f "$ERR"' EXIT
+
+# Vault list first: it is small, it is what the picker needs, and it is the
+# call that surfaces "not signed in" before we ask for anything else.
+if ! VAULTS_JSON="$(op vault list --format=json 2>"$ERR")"; then
+  MSG="$(cat "$ERR")"
+  case "$MSG" in
+    *"no account"*|*"not signed in"*)
+      cat >&2 <<'EOF'
+error: op is installed but no 1Password account is available to it.
+
+  1. Open the 1Password 8 app and unlock it.
+  2. Settings > Developer > tick "Integrate with 1Password CLI".
+  3. Sign in to a 1password.com account (standalone vaults cannot be
+     reached by the CLI).
+EOF
+      echo "op said: $MSG" >&2
+      exit 4 ;;
+    *)
+      echo "error: 1Password refused the request." >&2
+      echo "op said: $MSG" >&2
+      echo "If an approval dialog appeared, approve it and run this again." >&2
+      exit 5 ;;
+  esac
+fi
+
+if [ "$VAULTS_ONLY" -eq 1 ]; then
+  printf '[]' | python3 "$MAPPER" --vaults-json "$VAULTS_JSON" ${MAP_ARGS[@]+"${MAP_ARGS[@]}"}
+  exit $?
+fi
+
+# The pull itself. Piped straight into the mapper and held in memory: the
+# raw list never touches disk, and `additional_information` (username and
+# email hints) is dropped there rather than carried into the candidate
+# JSON. Held rather than streamed so a failed pull prints nothing at all on
+# stdout — a half-written candidate list would look like a short vault.
+if ! CANDIDATES="$(op item list --format=json 2>"$ERR" |
+     python3 "$MAPPER" --vaults-json "$VAULTS_JSON" ${MAP_ARGS[@]+"${MAP_ARGS[@]}"})"; then
+  MSG="$(cat "$ERR")"
+  echo "error: could not read your item list." >&2
+  [ -n "$MSG" ] && echo "op said: $MSG" >&2
+  echo "If 1Password asked for approval, approve it and run this again." >&2
+  exit 5
+fi
+printf '%s\n' "$CANDIDATES"