Skip to content

Mark DST fall-back days incomplete when repeated-hour capture evidence is absent #104

Description

@flamboh

Observed behavior

On the 2025-11-02 America/Los_Angeles fall-back day, each collector in /research/tango_cis/uonet-in has 288 nfcapd files, while the local day spans 25 elapsed hours. The capture files use wall-clock filenames, so the repeated 01:00 hour has no second filename. Internal nfdump receive timestamps show the repeated-hour records collide with the first 01:00 filename.

data/uoregon-v4/netflow.sqlite currently records 288 complete 5m rows per collector and a complete 1d row (expected_units = observed_units = 288; the combined uoregon_all row is 576/576). The missing interval is the repeated 01:00 hour: there is a 65-minute timestamp gap from 2025-11-02 08:55 UTC to 10:00 UTC, corresponding to 12 five-minute buckets.

When the web coverage API builds a continuous 25-hour 5m timeline, it synthesizes those 12 buckets as unknown, but the persisted day is still reported complete. The pipeline’s current regression test explicitly codifies 288 buckets and a complete day in tools/netflow-db/src/pipeline.rs (fall_back_tree_uses_wall_clock_filenames_without_repeating_one_am).

Expected behavior

A fall-back local day whose source evidence does not distinguish or cover the repeated hour should be marked incomplete (partial/unknown according to the coverage contract), including its daily rollup. The web API and charts should not have to discover the incompleteness only by synthesizing unknown timeline buckets.

Evidence / code references

  • tools/netflow-db/src/pipeline.rs:1240-1280 iterates local days and five-minute buckets.
  • tools/netflow-db/src/pipeline.rs:3024-3040 advances by local wall-clock five-minute labels.
  • tools/netflow-db/src/pipeline.rs:4443-4508 tests the fall-back tree as 288 5m buckets with 287 unknown and no incomplete daily state.
  • apps/web/src/lib/server/db/coverage.ts:227-255 fills missing stored intervals as unknown, exposing the 12 repeated-hour gaps only at API timeline construction.
  • data/uoregon-v4/netflow.sqlite: 2025-11-02 bucket_coverage has 288 rows per collector, all complete, and a complete 1d row.

Scope

Define and persist truthful DST fall-back completeness for native nfcapd tree inputs and rollups. This is a tracking issue only; do not mix it into current user-traffic work.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions