Add wizard step 7: interactive Duplicate Verification viewer

After saving the duplicate file (step 6 now auto-advances on a successful
save), every extracted row is re-validated:

- Row Key = the rule's duplicate key columns plus every cell to their
  right, under the scan's trim/case semantics.
- Condition A: the Row Key appears multiple times in the original file;
  the viewer lists the matching original row indices with their first
  cells (capped at 20).
- Condition B: it appears exactly once AND is absent from the reference
  file (over the mapped columns, reusing the extraction's exact key set -
  the ref-key db now survives the extraction for this).
- Anything else is INVALID: the audit trail for the extraction.

Occurrence lookups ride the scan store's dup_entries index; results live
in their own on-disk SQLite, so verification streams at any size.  The
viewer offers per-page size (preset or custom), Previous/Next, go-to
row/page, and live search over the first two cells (debounced LIKE).
Runs in its own subprocess (VerifyController); new scans/extractions
discard stale verification results.  New i18n keys in all five locales.

Focused adversarial review applied: the right-hand key is now
padding-insensitive (ragged source rows audit VALID instead of falsely
INVALID - trailing empty cells equal missing cells everywhere), the
verification reads the duplicate file without trailing-empty suppression
so fully-empty extracted rows are audited too, and unparseable dup-file
records surface as malformed_rows in the summary.

Local tmp/ data files are excluded and gitignored - user data never
belongs in the repository.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017es7LRoYVn9aCwZ8pKwffS

Author: SebastienDenooz

.gitignore

@@ -12,3 +12,4 @@ build/
 htmlcov/
 uv.lock
 tests/_fixtures_big/
+tmp/

README.md

@@ -40,6 +40,14 @@ Nothing ever leaves your machine.
   unique right-hand cells are kept only when an identical row (over the
   mapped columns) exists in the reference file. The extraction streams in
   its own process and the result is saved wherever you choose.
+- **Verification viewer** (step 7, entered automatically after saving):
+  every extracted row is re-checked against the original file and the
+  reference — condition A (its content appears several times in the
+  original, with the matching row indices and first cells listed) or
+  condition B (appears exactly once and is missing from the reference);
+  anything else is flagged INVALID. The viewer paginates (selectable or
+  custom page size), jumps to a row or page, and live-filters on the
+  first two columns.
 - **CLI**: the same engine headless, for pipelines
   (exit code `0` = no matches, `1` = matches found, `2` = error).
 - **Report**: per-rule match counts with lazily loaded sample rows

src/lsa/core/csv_stream.py

@@ -234,9 +234,17 @@ def _iter_raw(self) -> Iterator[Row]:
             yield Row(number, cells, offset)
             number += 1
 
-    def rows(self) -> Iterator[Row]:
-        """Yield data rows; the trailing run of blank lines is dropped."""
-        return suppress_trailing_empty(self._iter_raw())
+    def rows(self, *, keep_trailing_empty: bool = False) -> Iterator[Row]:
+        """Yield data rows; the trailing run of blank lines is dropped.
+
+        ``keep_trailing_empty`` disables that suppression — used when the
+        file is machine-written and every record is meaningful (e.g. the
+        verification step auditing a generated duplicate file).
+        """
+        iterator = self._iter_raw()
+        if keep_trailing_empty:
+            return iterator
+        return suppress_trailing_empty(iterator)
 
     def bytes_read(self) -> int:
         if self._lines is not None:

src/lsa/core/evaluate.py

@@ -137,6 +137,11 @@ def right_key_of(cells: Sequence[str], _start: int = right_start) -> str:
                 if trim:
                     value = value.strip()
                 parts.append(value if fold is None else fold(value))
+            # Padding-insensitive: a missing trailing cell and an empty one
+            # are the same content (a ragged source row and its padded copy
+            # in the duplicate file must produce identical keys).
+            while parts and parts[-1] == "":
+                parts.pop()
             return _KEY_SEPARATOR.join(parts)
 
         return DuplicateCondition(f"{rule_id}{_KEY_SEPARATOR}{index}", key_of, right_key_of)

src/lsa/core/extract.py

@@ -175,6 +175,7 @@ def extract_duplicates(
     transform = _value_transform(settings)
     source_indexes = mapping.source_indexes()
 
+    owns_ref_db = ref_db_path is None
     if ref_db_path is None:
         ref_db_path = make_temp_store_path(prefix="lsa-refkeys-")
     ref_db_path = Path(ref_db_path)
@@ -274,7 +275,10 @@ def flush_sub() -> None:
         source.close()
         store.close()
         ref_db.close()
-        ref_db_path.unlink(missing_ok=True)
+        if owns_ref_db:
+            # A caller-provided ref db is kept: the verification step reuses
+            # the exact key set the extraction ran against.
+            ref_db_path.unlink(missing_ok=True)
 
     return ExtractStats(
         reference_rows=reference_rows,

src/lsa/core/locales/de.json

@@ -103,5 +103,22 @@
   "dupfile.extracting.groups": "Gruppen werden verarbeitet... {done} / {total}",
   "dupfile.done": "{written} Duplikatzeile(n) extrahiert; {kept} Zeile(n) behalten; {groups} Gruppe(n) verarbeitet; Referenz: {reference} Zeile(n).",
   "dupfile.cancelled": "Extraktion abgebrochen - die Datei ist unvollständig.",
-  "dupfile.save": "Duplikatdatei speichern..."
+  "dupfile.save": "Duplikatdatei speichern...",
+  "step.verify.title": "Überprüfung",
+  "step.verify.help": "Jede Zeile der erzeugten Duplikatdatei wird erneut geprüft: Bedingung A bedeutet, ihr Inhalt (Schlüsselspalten plus alles rechts davon) kommt mehrfach in der Originaldatei vor; Bedingung B bedeutet, er kommt genau einmal vor und fehlt in der Referenzdatei. UNGÜLTIGE Zeilen hätten nicht extrahiert werden dürfen.",
+  "verify.no_data": "Extrahieren und speichern Sie zuerst eine Duplikatdatei - es gibt noch nichts zu überprüfen.",
+  "verify.running": "Duplikatzeilen werden überprüft... {done}",
+  "verify.summary": "{total} Zeile(n) überprüft: {valid_a} gültig (Bedingung A), {valid_b} gültig (Bedingung B), {invalid} UNGÜLTIG.",
+  "verify.status": "Status",
+  "verify.valid": "GÜLTIG",
+  "verify.invalid": "UNGÜLTIG",
+  "verify.condition": "Bedingung",
+  "verify.occurrences": "Vorkommen",
+  "verify.matches": "Originalzeilen (Zeile:erste Zelle)",
+  "verify.filter": "Filter (erste zwei Spalten)",
+  "verify.page_size": "Pro Seite",
+  "verify.goto": "Gehe zu",
+  "verify.goto.row": "Zeile",
+  "verify.goto.page": "Seite",
+  "verify.go": "Los"
 }

src/lsa/core/locales/en.json

@@ -103,5 +103,22 @@
   "dupfile.extracting.groups": "Processing groups... {done} / {total}",
   "dupfile.done": "{written} duplicate row(s) extracted; {kept} row(s) kept; {groups} group(s) processed; reference: {reference} row(s).",
   "dupfile.cancelled": "Extraction cancelled - the file is incomplete.",
-  "dupfile.save": "Save duplicate file..."
+  "dupfile.save": "Save duplicate file...",
+  "step.verify.title": "Verification",
+  "step.verify.help": "Every row of the generated duplicate file is checked again: condition A means its content (key columns plus everything to their right) appears several times in the original file; condition B means it appears exactly once and is missing from the reference file. INVALID rows should not have been extracted.",
+  "verify.no_data": "Extract and save a duplicate file first - there is nothing to verify yet.",
+  "verify.running": "Verifying duplicate rows... {done}",
+  "verify.summary": "{total} row(s) verified: {valid_a} valid (condition A), {valid_b} valid (condition B), {invalid} INVALID.",
+  "verify.status": "Status",
+  "verify.valid": "VALID",
+  "verify.invalid": "INVALID",
+  "verify.condition": "Condition",
+  "verify.occurrences": "Occurrences",
+  "verify.matches": "Original rows (row:first cell)",
+  "verify.filter": "Filter (first two columns)",
+  "verify.page_size": "Per page",
+  "verify.goto": "Go to",
+  "verify.goto.row": "Row",
+  "verify.goto.page": "Page",
+  "verify.go": "Go"
 }

src/lsa/core/locales/es.json

@@ -103,5 +103,22 @@
   "dupfile.extracting.groups": "Procesando grupos... {done} / {total}",
   "dupfile.done": "{written} fila(s) duplicada(s) extraída(s); {kept} fila(s) conservada(s); {groups} grupo(s) procesado(s); referencia: {reference} fila(s).",
   "dupfile.cancelled": "Extracción cancelada: el archivo está incompleto.",
-  "dupfile.save": "Guardar archivo de duplicados..."
+  "dupfile.save": "Guardar archivo de duplicados...",
+  "step.verify.title": "Verificación",
+  "step.verify.help": "Cada fila del archivo de duplicados se comprueba de nuevo: la condición A significa que su contenido (columnas clave más todo lo que está a su derecha) aparece varias veces en el archivo original; la condición B significa que aparece exactamente una vez y falta en el archivo de referencia. Las filas NO VÁLIDAS no deberían haberse extraído.",
+  "verify.no_data": "Primero extraiga y guarde un archivo de duplicados: aún no hay nada que verificar.",
+  "verify.running": "Verificando filas duplicadas... {done}",
+  "verify.summary": "{total} fila(s) verificada(s): {valid_a} válida(s) (condición A), {valid_b} válida(s) (condición B), {invalid} NO VÁLIDA(S).",
+  "verify.status": "Estado",
+  "verify.valid": "VÁLIDA",
+  "verify.invalid": "NO VÁLIDA",
+  "verify.condition": "Condición",
+  "verify.occurrences": "Apariciones",
+  "verify.matches": "Filas originales (fila:primera celda)",
+  "verify.filter": "Filtro (dos primeras columnas)",
+  "verify.page_size": "Por página",
+  "verify.goto": "Ir a",
+  "verify.goto.row": "Fila",
+  "verify.goto.page": "Página",
+  "verify.go": "Ir"
 }

src/lsa/core/locales/fr.json

@@ -103,5 +103,22 @@
   "dupfile.extracting.groups": "Traitement des groupes... {done} / {total}",
   "dupfile.done": "{written} ligne(s) en double extraite(s) ; {kept} ligne(s) conservée(s) ; {groups} groupe(s) traité(s) ; référence : {reference} ligne(s).",
   "dupfile.cancelled": "Extraction annulée - le fichier est incomplet.",
-  "dupfile.save": "Enregistrer le fichier des doublons..."
+  "dupfile.save": "Enregistrer le fichier des doublons...",
+  "step.verify.title": "Vérification",
+  "step.verify.help": "Chaque ligne du fichier des doublons est recontrôlée : la condition A signifie que son contenu (colonnes clés et tout ce qui est à leur droite) apparaît plusieurs fois dans le fichier d'origine ; la condition B signifie qu'il apparaît exactement une fois et est absent du fichier de référence. Les lignes INVALIDES n'auraient pas dû être extraites.",
+  "verify.no_data": "Extrayez et enregistrez d'abord un fichier des doublons - il n'y a encore rien à vérifier.",
+  "verify.running": "Vérification des lignes en double... {done}",
+  "verify.summary": "{total} ligne(s) vérifiée(s) : {valid_a} valide(s) (condition A), {valid_b} valide(s) (condition B), {invalid} INVALIDE(S).",
+  "verify.status": "Statut",
+  "verify.valid": "VALIDE",
+  "verify.invalid": "INVALIDE",
+  "verify.condition": "Condition",
+  "verify.occurrences": "Occurrences",
+  "verify.matches": "Lignes d'origine (ligne:première cellule)",
+  "verify.filter": "Filtre (deux premières colonnes)",
+  "verify.page_size": "Par page",
+  "verify.goto": "Aller à",
+  "verify.goto.row": "Ligne",
+  "verify.goto.page": "Page",
+  "verify.go": "OK"
 }

src/lsa/core/locales/pt.json

@@ -103,5 +103,22 @@
   "dupfile.extracting.groups": "A processar grupos... {done} / {total}",
   "dupfile.done": "{written} linha(s) duplicada(s) extraída(s); {kept} linha(s) mantida(s); {groups} grupo(s) processado(s); referência: {reference} linha(s).",
   "dupfile.cancelled": "Extração cancelada - o ficheiro está incompleto.",
-  "dupfile.save": "Guardar ficheiro de duplicados..."
+  "dupfile.save": "Guardar ficheiro de duplicados...",
+  "step.verify.title": "Verificação",
+  "step.verify.help": "Cada linha do ficheiro de duplicados é verificada novamente: a condição A significa que o seu conteúdo (colunas-chave mais tudo o que está à direita) aparece várias vezes no ficheiro original; a condição B significa que aparece exatamente uma vez e está ausente do ficheiro de referência. As linhas INVÁLIDAS não deveriam ter sido extraídas.",
+  "verify.no_data": "Extraia e guarde primeiro um ficheiro de duplicados - ainda não há nada para verificar.",
+  "verify.running": "A verificar linhas duplicadas... {done}",
+  "verify.summary": "{total} linha(s) verificada(s): {valid_a} válida(s) (condição A), {valid_b} válida(s) (condição B), {invalid} INVÁLIDA(S).",
+  "verify.status": "Estado",
+  "verify.valid": "VÁLIDA",
+  "verify.invalid": "INVÁLIDA",
+  "verify.condition": "Condição",
+  "verify.occurrences": "Ocorrências",
+  "verify.matches": "Linhas originais (linha:primeira célula)",
+  "verify.filter": "Filtro (duas primeiras colunas)",
+  "verify.page_size": "Por página",
+  "verify.goto": "Ir para",
+  "verify.goto.row": "Linha",
+  "verify.goto.page": "Página",
+  "verify.go": "Ir"
 }