Skip to content

Fix timeline virtualization limit on large datasets - #7

Open
jaybird1291 wants to merge 43 commits into
r3nzsec:mainfrom
jaybird1291:fix/fix-timeline-scroll-limit
Open

Fix timeline virtualization limit on large datasets#7
jaybird1291 wants to merge 43 commits into
r3nzsec:mainfrom
jaybird1291:fix/fix-timeline-scroll-limit

Conversation

@jaybird1291

Copy link
Copy Markdown

Summary

Fixes a renderer-side virtualization issue in the base timeline view that prevented navigation beyond ~645k rows on large datasets.

Root cause

The table virtualization relied on a scroll container sized as totalRows * ROW_HEIGHT, which becomes too large for Chromium/Electron on multi-million-row datasets.

Changes

  • bound the physical scroll height
  • map physical scroll position to logical row position
  • update virtual window calculations and row positioning
  • keep scrollToRow and context menu positioning aligned with the new mapping

Validation

Tested with output_timeline_stream.csv (cf. #6):

  • no filters
  • no search
  • chronological sort
  • reverse chronological sort

The base view can now navigate beyond the previous ~645k-row ceiling:

Fix.mp4

Fixes #6

r3nzsec and others added 30 commits March 1, 2026 23:42
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add Open Graph and Twitter Card meta tags for rich social previews
- Add sitemap generation and lastUpdated support
- Fix favicon (switch from missing .ico to logo.svg)
- Fix version mismatch in HeroGraphic (v2.1.2 → v1.0.3-beta)
- Add cross-links in DFIR Tips (brute-force, log-tampering, kape-triage)
- Add "Who is this for?" section to homepage
- Add FAQ & Troubleshooting page (reference/faq)
- Add Roadmap page (about/roadmap)
- Add Download button and social links (X, LinkedIn) to nav bar
- Add FAQ and Roadmap to sidebar navigation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Enable cleanUrls to remove .html extensions from all page URLs
- Enable markdown.image.lazyLoading for native lazy loading on doc images
- Add prefers-reduced-motion support to HeroGraphic (JS + CSS)
- Fix accessibility contrast in HeroGraphic: bump label colors from #555
  to #777 (~4.5:1 WCAG AA on dark backgrounds), tree branches #333 to #555
- Add performance benchmarks section to Performance Tips: import+indexing,
  query response times, and analytics tool benchmarks across 3 dataset sizes

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix copyright year (2025 → 2025-2026)
- Fix search modes to match actual app (Mixed, FTS, LIKE, Fuzzy, Regex)
- Fix nav logo path (logo.png → logo.svg)
- Fix macOS casing in homepage tagline
- Fix virtual scrolling window size (5K → 10K) across all pages
- Fix build instructions (cd path, remove nonexistent npm run dist)
- Add missing .xls/.xlsm file associations
- Update stale source file line counts in architecture page

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Redesign lateral movement SVG: 15 nodes (hosts, IPs, suspicious outliers),
  17 edges with connection counts, red dashed circles with warning triangles
- Fix Mimikatz label from LOLBIN to CREDENTIAL DUMP in HeroGraphic
- Add release dates to all changelog entries
- Add large file re-import warning to sessions page
- Fix FAQ debug log link (remove misleading architecture page link)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…Store

- Fix detection rules count inconsistency (344 → 342 to match source)
- Expand sparse v0.9.1 changelog with detailed descriptions
- Add Histogram brush selection screenshot to histogram feature page
- Remove .DS_Store files from git tracking (already in .gitignore)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add description frontmatter to all 40 docs pages for meta description tags
- Add SoftwareApplication JSON-LD structured data to VitePress config
- Add warm off-white light mode palette and component styles in custom.css
- Add "What's New in v1.0.3-beta" highlights section to homepage

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
50-row forensic attack scenario with text/regex/fuzzy search,
sortable columns, match highlighting, sparkline, and row expansion.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Network graph shows KALI flagged as threat actor workstation via hostname
pattern matching. Includes attack chain visualization, clickable nodes,
pulsing outlier ring, and finding callout explaining the detection.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
After the initial entrance animation (~2.4s), auto-cycles through 4 steps
(Import, Search, Process Tree, Lateral Movement) every 5s with opacity-based
section focus, floating captions, and step indicator dots with progress bars.
Hover pauses the tour, clicking a dot jumps to that step. Desktop-only steps
are excluded on mobile. Supports prefers-reduced-motion.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Drop generic intro line, lead bullets with pain points, make closing
paragraph more direct with concrete 1M-row number and active voice.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds 3-5 contextual links at the bottom of each page connecting
related features and workflows (17 pages, 117 lines total).
- docFooter: prev/next links at the bottom of every page
- outline: 'deep' for nested heading TOC on long pages
- externalLinkIcon: visual indicator for external links
- returnToTopLabel: back-to-top button
- Add light mode styles for tour caption, dots, and graphic shadow
- Tighten all 8 feature card descriptions for better scannability
…e roadmap/shortcuts/KAPE profiles

Add new NTFS Analysis docs page covering all 6 MFT/USN tools. Update
roadmap to move shipped features (MFT support, auto-update, VT, analyst
profiles) to completed. Add VirusTotal enrichment section to IOC matching,
view modes and analyst profiles to Process Inspector, 8 missing KAPE
profiles, missing keyboard shortcuts, advanced filter editor and find
duplicates to search docs, and bulk actions to bookmarks/tags.
Update line counts (App.jsx 19K, db.js 12.8K, main.js 1.9K, parser.js
2.5K), IPC handler count (69), preload methods (132). Add new modules:
updater.js, logger.js. Add MFT/USN binary parsers, NTFS analysis engine,
VirusTotal cache DB, 5 custom SQL functions, electron-updater dependency.
Update security model for VT API and auto-update network access.
SVG diagram now shows 7 parsers (added $MFT and $J boxes), 5 main
process boxes (added VT & Updater), 5 SQLite engine boxes (added NTFS
Analysis), updated IPC handler count (69), detection rules (342 chain +
13 standalone), and file list (updater.js, logger.js).
…rsistence detection expansion

- Add raw NTFS $MFT and $UsnJrnl ($J) direct import with full path reconstruction, SI/FN timestamps, and change reason mapping
- Add ransomware impact analytics from $MFT data: bulk rename detection, entropy-based extension analysis, ransom note identification, temporal clustering
- Add VirusTotal integration for IOC matching: bulk API lookups with rate limiting, local caching, malware family extraction, verdict badges, per-IOC retry, relationship pivoting (contacted domains, communicating files, DNS resolutions), URL enrichment support
- Expand persistence analyzer to 30+ techniques: account persistence chains, cross-technique correlation, PowerShell 4104 script block reassembly
- Add VT enrichment to CSV export (scores, verdicts, threat labels, timestamps) and HTML reports
- Fix SQL UNION ALL LIMIT syntax, VT filter case sensitivity, CSV falsy score export, listener leak on modal close
- Update Quick Help and About dialog with all new features
- Add auto-updater, electron-builder config, macOS notarization, release workflow

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… enrichment highlights

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
r3nzsec and others added 13 commits March 9, 2026 09:00
… extraction

Add UsnJrnl Rewind section to NTFS analysis docs explaining reverse-
chronological path reconstruction for deleted files (credit CyberCX).
Expand resident data extraction with threat actor recovery use cases.
Update homepage and roadmap references. Fix VT auto-tag names.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…s pages

New standalone documentation pages for features that were previously
undocumented or buried in other pages. Updates sidebar navigation and
bumps version badge to v1.0.4-beta.
Bug fixes:
- Fix Plaso import crash: LIMIT clause in UNION ALL query must be inside
  a subquery — wraps first SELECT in (SELECT ... LIMIT 300) to fix SQLite
  error on all Plaso files
- Fix better-sqlite3 architecture mismatch on Intel Macs: CI now runs on
  macos-13 (Intel) and explicitly builds x64 + arm64 slices then lipo-merges
  into a universal fat binary before packaging

UI improvements:
- Cell context menu: Cmd+Click any cell to Filter in, Filter out, or Hide column
- Right-click context menu: Filter in / Filter out added under FILTERS section
- Tags collapsed into hover submenu (Tags ▸) to keep context menu compact
- Multi-row tagging: right-clicking with multiple rows selected applies tag
  to all selected rows; menu shows count e.g. "Tags (4 rows)"
- Fixed context menu background bleed-through: opacity raised to 0.97 so
  grid rows no longer show through semi-transparent menus
- Cmd+C now correctly copies selected text in detail panel instead of
  intercepting native copy when a DOM selection exists

CI / release:
- Switch CI runner from macos-14 (arm64) to macos-13 (Intel) for reliable
  universal native module cross-compilation
- Clean release/ directory before each build to prevent stale artifact pickup
- Artifact collection now matches exact version from package.json instead of
  first-found glob

Quick Help updated with filter in/out and multi-row tagging documentation

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…niversal binary

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add v1.0.5 changelog, document cell context menu (Cmd+Click filter
in/out), multi-row tagging, Tags hover submenu, .timeline format
support, and updated keyboard shortcuts reference.
…rmat table

Update architecture.md line counts (App.jsx ~19.9K, main.js ~2K,
db.js ~14.1K, parser.js ~2.6K), correct IPC handler/method counts,
update virtual-grid context menu docs for v1.0.5, add .timeline to
homepage format table, note Cmd+C detail panel behavior.
RMM_SIGS in db.js has 33 RMM tools and 7 tunnel tools, not 30.
Add missing tools (GoTo/LogMeIn, BeyondTrust, Dameware, Supremo,
FixMe.IT) and separate tunnel tools into their own list.

@r3nzsec r3nzsec left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the detailed PR — the approach is exactly right and the analysis in #6 is spot-on (the ~16.7M-px Chromium LayoutUnit ceiling is the actual root cause). I went down the same path on my end and arrived at essentially the same design: bounded physical rail + linear physical↔logical mapping + per-row top adjustment.

A few notes on the change set:

👍 The visibleStartAi = Math.max(si, rowOffset) correction for the non-grouped path is a separate, real bug that I'd missed — good catch.
👍 Adding viewportH to the scroll-fetch effect deps is correct; without it the fetch trigger goes stale on window resize.
The cap at 16M − 4KB is aggressive — it sits just under Blink's saturation point. A more conservative value (e.g. 12–15M) leaves headroom for sub-pixel rounding and zoom-related layout shifts. The explicit renderBaseTop clamp probably makes 4KB margin safe in practice, but worth considering.
Heads up on a merge conflict: a recent local refactor moved the virtual-grid render into src/components/VirtualGrid.jsx and pulled the layout constants into src/constants/grid.js. The four top: rowTopForIndex(ai) updates will need to land in the component file rather than App.jsx, and MAX_VIRTUAL_SCROLL_HEIGHT belongs alongside ROW_HEIGHT in the constants module. Happy to handle the rebase if that's easier.
Validated locally against the same 2.88M-row dataset — the previous ~645K cap is gone in both sort directions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Base timeline view cannot display more than ~645k rows on large datasets

2 participants