Skip to content

Overlay translation: ephemeral NumBig handles for never-indexed decimal/bigint values #1324

Description

@bplatz

Context

Typed xsd:decimal and overflow xsd:integer values are NumBig-arena-keyed per (graph, predicate), with handles minted at index build. Overlay translation now resolves values already in the arena read-only (BinaryIndexStore::find_numbig_handle, used by value_to_otype_okey's BigInt/Decimal arms in fluree-db-query/src/binary_scan.rs), which covers every retract and every re-assert of an indexed value.

A value the index has never seen still has no handle, so translation returns Unsupported and collect_resolved_overlay_ops declines the predicate's whole batched probe lane (fast_path_common.rs) until the next reindex. On decimal-assert-heavy novelty (e.g. frequent updates writing new budgetAmount values), the batched join lanes never engage for that predicate.

Severity note: the TranslatedOverlayCache memo (landed via #1308) removed the old O(rows × novelty) per-row re-translation cliff, so the cost of a decline today is "memoized per-row scans" rather than quadratic blowup. This issue is the remaining step to keep the batched lanes fully live.

Proposal

Mirror the ephemeral-p_id pattern used for novelty-only predicates:

  • During per-predicate ops collection, assign ephemeral handles above the arena watermark for arena-miss values, keyed by value so the same decimal maps to one handle within a translation.
  • Thread a decode side-table (ephemeral handle → value) to consumers, the way ephemeral_preds / ephemeral_p_id_to_sid already flows through binary_range and TranslatedOverlayOps.

Soundness: the arena is equality-only (no value order in o_key), and a value absent from the arena can never identity-match a base row — so ephemeral handles only ever participate in overlay-only assert injection and intra-overlay identity, never in base-row reconciliation.

Acceptance

  • A novelty assert of a never-indexed decimal keeps the batched subject/object lanes engaged (extend it_join_batched_overlay.rs::batched_join_decimal_novelty phase B, which currently asserts the decline-and-fallback behavior).
  • Ephemeral handles decode back to the exact BigDecimal/BigInt for projection and FILTER evaluation.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions