Skip to content

fast-stats over-count: novelty asserts not reconciled against the base index #1391

Description

@bplatz

Summary

assemble_fast_stats applies a blind ±1 per novelty flake on top of the indexed base counts, with no reconciliation against whether the asserted fact already exists in the base index. An idempotent re-assert of a fact that has already been reindexed into the base therefore transiently inflates fast-stats counts until the next reindex collapses the duplicate.

Detail

  • fluree-db-novelty/src/runtime_stats.rs (assemble_fast_stats_inner): delta = if flake.op { 1 } else { -1 }, added to indexed.flakes and the per-graph / class / property counts, with no base-presence check.
  • Within-novelty dedup (NoveltyFactState) prevents re-asserting a fact already asserted in the novelty window, but it is novelty-window-scoped and cannot see the base index. Once a fact is reindexed into base (and dropped from novelty), a subsequent assert of the same (s, p, o, dt, m) is kept as a novelty flake and counted again.
  • Net effect: counts (total flakes, class instance counts, property counts) can over-report by the number of such idempotent re-asserts, until the next reindex merges novelty into base under set semantics and the count self-corrects.

Scope / severity

  • Pre-existing; affects the normal reindex path. Not introduced by the segmented-novelty rework (perf: append-only segmented novelty — O(batch) write path #1302) — surfaced during its review.
  • Transient (self-corrects at the next reindex), and query results are unaffected — set semantics make the duplicate assert idempotent at read time. Only the stat counts drift.

Note on whether to fix

These stats are used primarily for query optimization / planning, not as an authoritative user-facing count. Reconciling every novelty assert against base presence (e.g. a base-index existence probe per asserted fact) could add non-trivial cost to the stats-assembly path. If making the counts 100% accurate carries a significant performance implication, it may not be worth doing — the planner tolerates approximate counts. Recording here for future consideration / measurement rather than as an urgent fix.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions