Skip to content

Scale processing beyond the in-memory session limit #11

Description

@zrack

Operator outcome

Large telemetry sessions remain responsive, cancellable, and deterministic under documented memory and latency budgets.

Scope

  • Move validation, decoding, aggregation, and bundle construction off the UI thread with Web Workers.
  • Add streaming JSON/NDJSON ingestion or a chunked container beyond the current 32 MiB UTF-8 .nlsession limit.
  • Evaluate IndexedDB or another local indexed store for frame and metric access.
  • Bound downsampling, memory growth, marker lookup, and evidence generation for multi-million-record sessions.
  • Add cancellation, progress reporting, and deterministic failure recovery.

Invariants

  • Integer-microsecond offsets and half-open ranges remain exact.
  • Raw records remain immutable and failure diagnostics remain visible.
  • Worker and streaming paths produce the same decoded and evidence results as the current in-memory pipeline.

Exit criteria

  • A documented large fixture imports, replays, seeks, selects incidents, and exports without blocking the main UI.
  • Memory and latency budgets are measured in automated tests.
  • Cancellation cannot produce a falsely complete session or bundle.

Metadata

Metadata

Assignees

No one assigned

    Labels

    architectureArchitecture, runtime, storage, and core design decisionsenhancementNew feature or request

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions