AuditFile.group_refs is computed from valid lines only, but audit_file.groups (M2M) is populated from every parsed line — a group referenced only by an invalid line is linked yet absent from group_refs

[erskingardner]

Summary

create_events resolves and links a file→group membership for every parsed line (valid or invalid, and duplicates), but the denormalized AuditFile.group_refs list is built from valid lines only. So a group referenced only by an invalid line is linked via the groups M2M yet never appears in group_refs — two persisted representations of the same file→group relationship disagree.

Code

forensics/ingest.py:846-867 — M2M membership from all parsed lines:

for parsed in parsed_lines:                       # all lines, valid or not
    group_key = group_key_for_parsed_line(parsed, fallback_group_slug=fallback_group_slug)
    group = groups_by_key.get(group_key)
    if group is not None:
        group_ids.add(group.id)                   # invalid-line groups added too
    if duplicate_event_exists(...):
        duplicate_count += 1
        continue
    ...
if group_ids:
    audit_file.groups.add(*group_ids)             # M2M includes invalid-line groups

forensics/ingest.py:1138-1144 — group_refs from valid lines only:

group_refs = sorted({
    line.normalized.get("group_ref")
    for line in valid_lines                        # valid_lines only
    if line.normalized.get("group_ref")
})

Concrete failure scenario

A V2 line carries a well-formed group_ref "aabb…" but is marked INVALID for an unrelated reason (e.g. a bad seq). create_events still resolves its group, adds it to group_ids, and links it via audit_file.groups.add(...). But AuditFile.group_refs will not list "aabb…" unless some valid line also references it. The file-detail summary (reading group_refs) then under-reports a group the file is demonstrably linked to, while that group's own detail view (via the M2M) shows the file and its invalid event.

Why not a duplicate

• Empty-string group_ref yields a VALID but silently un-grouped event (marked valid, orphaned with no fallback group → invisible in every view) #204 is about empty group_ref orphaning a valid event.
• audit_files_for_group.group_event_count counts INVALID and structurally-quarantined events, contradicting every other group view #203 is group_event_count counting invalid/quarantined events.
• file_metadata derives schema_versions from all parsed lines, while every sibling aggregate uses valid lines only #267 is schema_versions derived from all parsed lines while sibling aggregates use valid lines.

This is a distinct, opposite-direction inconsistency: a valid-only aggregate (group_refs) undercounts relative to the all-lines M2M membership for the same file.

Suggested fix

Reconcile the two representations against one convention. If "aggregates reflect valid evidence only" is the intended rule (per #267's reasoning), the M2M group_ids should be derived from valid lines too (except the deliberate dedup carve-out for goggles#37); otherwise group_refs should be widened to match the M2M. Either way a test should assert set(group_refs) and the linked-group ref set agree.

Severity: LOW — a persisted parity gap between two representations of file→group membership (fits the existing #203/#204/#267 parity class).