Skip to content

Lazy-index drop on duplicate selections: switch to uniform warning + join-internal suppression (option b)Β #14

Description

@espg

πŸ€– from Claude

Follow-up from PR #12's standing judgment call (espg: merge as-is, revisit later β€” 2026-07-19). Behavior change proposal recorded with the full decision space so the revisit can pick up cold.

Current behavior (as merged in #12)

Phase 5's contract: a selection unrepresentable by MortonMocIndex's interval substrate keeps correct values but silently becomes a pandas-backed dataset, with a UserWarning making the index-type change discoverable. Phase 6b carved out duplicate-containing picks: silent drop, no warning β€” because join_coarse maps many fine cells to repeated coarse rows on every call (interval sets fundamentally cannot hold duplicates) and warning per join would be noise. The carve-out's true breadth is documented in the code (moc_index.py, fold commit fbff4c3): it also silences user-initiated ds.sel(morton=[w, w]) β€” the manual broadcast idiom β€” where PandasIndex would keep a non-unique index and phase-5 posture says the drop should signal.

The proposed change β€” option (b)

  1. Remove the duplicate carve-out: ALL lazy-index drops warn uniformly (restores the phase-5 contract without exceptions).
  2. join_coarse suppresses the warning locally around its own internal coarse-row materialization (warnings.catch_warnings block or an internal flag threaded through the isel path β€” ~5 lines) β€” the join owns its silence because it knows the drop is intentional; users keep full observability.
  3. Tests: user duplicate-sel warns; join_coarse emits zero warnings (pin with simplefilter("error") the way the phase-5 tests do); reorder-drop warning unchanged.

Why it's close (the honest tradeoff, from the PR #12 review + session discussion)

  • For (b): the phase-5 warning exists precisely because a silent index-type change is a footgun β€” downstream ops shift semantics (a later join using the result as coarse falls back to word-inference instead of index.level; alignment behavior changes), discoverable today only via xindexes inspection. The carve-out was expedience, not principle.
  • For (a), the as-merged state (the PR Cross-resolution truncation join (phase 6 of issue #1)Β #12 reviewer's lean): duplicates are definitionally unrepresentable in an interval set; the affected user pattern is rare; values are never wrong; users doing deliberate broadcast-sel would find the warning noise (mitigable with standard filterwarnings).
  • (c) a positions-backed lazy variant for duplicated domains = over-engineering (pandas fallback IS the right representation); (d) raising = hostile (pandas permits duplicate labels). Both rejected in-session.

Implementation notes for whoever picks this up: the join's internal path is _coarse_lookup β†’ ranges.rank β†’ positional isel on the coarse dataset β€” the suppression wraps only that materialization; catch_warnings is process-global state (fine for the synchronous join; note if the #5-era concurrency ever reaches this path). Severity low either way; behavior change is user-visible (warning appears where none fired), so it should ride a minor release note.

Refs: PR #12 (judgment call 1; review finding r3611444087 + fold fbff4c3), phase-5 warning ratification (PR #10 fold 94b8a0e), issue #1.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions