diff --git a/CHANGELOG.md b/CHANGELOG.md index 10b0d30..2219bce 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,9 +7,35 @@ that, minor versions may change interfaces and the manifest format. ## [Unreleased] +### Packaging +- One package, `gridsift`, now carries both binaries: `cargo install + gridsift` installs the desktop application and the command-line tool, + and `cargo run` starts the application (`default-run`). The + `gridsift-desktop` package is gone; the binary names are unchanged. + +### Analysis cache +- Whole-file results are kept in the user cache directory beside the index + sidecar: the file-wide column profile and value counts (`*.gsan`) and + the timelines over all records (`*.gstl`, read only when a timeline is + asked for). The next open of the same bytes — same + size, modification time and SHA-256 — shows the dashboard and timeline + at once and starts no profile pass; `freq`, `timeline` and `profile` + report `cached`. Counts over a selection or of derived columns are + always recomputed, and a changed file never uses the cache. + +### Files without a header +- The sniffer's reading of the first record can be overridden everywhere: + `--header` / `--no-header` on the desktop launch line as on the command + line, and a click on *header* / *no header* in the sidebar reopens the + file the other way. `--names a,b,c` (command line and desktop launch + line) and the sidebar's *name columns…* dialog name the columns of a + file without a header; the names resolve `-c` and are recorded in the + manifest of every export (`source.dialect.names`), never in the file. + ### Desktop (`gridsift-desktop`) -- The analysis dock opens on *Values*, now the first tab and with its own - column picker. The dashboard is built from the file-wide profile and +- The analysis dock is open from the start, on *Values*, now the first tab + and with its own column picker; *Analysis* in the command bar hides and + shows it. The dashboard is built from the file-wide profile and counted in the background, on screen or not, so opening it shows current charts at once; the panels are counted one after another (one count's memory instead of six), and a timeline the analyst moved to another diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 349214e..bc93140 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -28,7 +28,7 @@ CI runs the same on Linux, macOS and Windows with the stable toolchain and a `cargo check` on the minimum supported Rust version (1.88). The scanner is checked against the `csv` crate on a torture corpus and on random input (`crates/gridsift-core/tests/differential.rs`); the command-line tool has -end-to-end tests (`crates/gridsift-cli/tests/cli.rs`) that run the real +end-to-end tests (`crates/gridsift/tests/cli.rs`) that run the real binary with the index cache redirected into a temporary home. Synthetic data for manual testing and benchmarks comes from the tool diff --git a/Cargo.lock b/Cargo.lock index c988b6a..afdffa9 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1618,8 +1618,13 @@ version = "0.1.0" dependencies = [ "anyhow", "clap", + "eframe", + "egui", + "egui_extras", + "egui_plot", "gridsift-core", "indicatif", + "rfd", "serde_json", ] @@ -1650,19 +1655,6 @@ dependencies = [ "thiserror 2.0.21", ] -[[package]] -name = "gridsift-desktop" -version = "0.1.0" -dependencies = [ - "anyhow", - "eframe", - "egui", - "egui_extras", - "egui_plot", - "gridsift-core", - "rfd", -] - [[package]] name = "half" version = "2.7.1" diff --git a/Cargo.toml b/Cargo.toml index 57ce613..8346d64 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,6 +1,8 @@ [workspace] resolver = "3" members = ["crates/*"] +# `cargo build` / `cargo run` at the root build the application package +default-members = ["crates/gridsift"] [workspace.package] version = "0.1.0" diff --git a/NOTICE b/NOTICE index 5d2b0e1..e6eebe3 100644 --- a/NOTICE +++ b/NOTICE @@ -5,7 +5,7 @@ This project is dual-licensed under the MIT License and the Apache License, Version 2.0. See LICENSE-MIT and LICENSE-APACHE for the full texts. Third-party dependencies and their declared licenses (direct dependencies -of the three workspace crates; versions as pinned in Cargo.lock at the time +of the two workspace crates; versions as pinned in Cargo.lock at the time of writing): gridsift-core @@ -30,19 +30,16 @@ gridsift-core sha2 0.11 MIT OR Apache-2.0 https://github.com/RustCrypto/hashes thiserror 2.0 MIT OR Apache-2.0 https://github.com/dtolnay/thiserror -gridsift (command-line tool) +gridsift (the desktop application and the command-line tool) anyhow 1.0 MIT OR Apache-2.0 https://github.com/dtolnay/anyhow clap 4.6 MIT OR Apache-2.0 https://github.com/clap-rs/clap - indicatif 0.18 MIT https://github.com/console-rs/indicatif - serde_json 1.0 MIT OR Apache-2.0 https://github.com/serde-rs/json - -gridsift-desktop - anyhow 1.0 MIT OR Apache-2.0 https://github.com/dtolnay/anyhow eframe 0.32 MIT OR Apache-2.0 https://github.com/emilk/egui egui 0.32 MIT OR Apache-2.0 https://github.com/emilk/egui egui_extras 0.32 MIT OR Apache-2.0 https://github.com/emilk/egui egui_plot 0.33 MIT OR Apache-2.0 https://github.com/emilk/egui_plot + indicatif 0.18 MIT https://github.com/console-rs/indicatif rfd 0.17 MIT https://github.com/PolyMeilex/rfd + serde_json 1.0 MIT OR Apache-2.0 https://github.com/serde-rs/json Test-only dependencies proptest 1.11 MIT OR Apache-2.0 https://github.com/proptest-rs/proptest diff --git a/README.md b/README.md index e08ce7d..05e5a77 100644 --- a/README.md +++ b/README.md @@ -33,16 +33,18 @@ application for Linux, macOS and Windows. | Interface | Binary | Description | |-----------|--------|-------------| +| **Desktop** | `gridsift-desktop` | The interactive investigation: instant rows, search and pivot with a visible selection lineage, value counts, dashboard and timeline, enrichment, export with redaction. | | **CLI** | `gridsift` | Index, search, count, timeline, profile, export, verify; `--json` on every command for scripting and batch use. | -| **Desktop** | `gridsift-desktop` | The interactive investigation: instant rows, search and pivot with a visible selection lineage, timeline and value counts, enrichment, export with redaction. | + +Both binaries come from the one `gridsift` package; the desktop application +is what `cargo run` starts. ## Quick install -Rust 1.88 or newer. From crates.io: +Rust 1.88 or newer. From crates.io, one package installs both binaries: ``` -cargo install gridsift --locked # command-line tool -cargo install gridsift-desktop --locked # desktop application +cargo install gridsift --locked # gridsift-desktop and gridsift ``` Or build from source: diff --git a/crates/gridsift-cli/NOTICE b/crates/gridsift-cli/NOTICE deleted file mode 100644 index 5d2b0e1..0000000 --- a/crates/gridsift-cli/NOTICE +++ /dev/null @@ -1,81 +0,0 @@ -gridsift -Copyright 2026 gridsift contributors - -This project is dual-licensed under the MIT License and the Apache License, -Version 2.0. See LICENSE-MIT and LICENSE-APACHE for the full texts. - -Third-party dependencies and their declared licenses (direct dependencies -of the three workspace crates; versions as pinned in Cargo.lock at the time -of writing): - -gridsift-core - blake3 1.8 CC0-1.0 OR Apache-2.0 https://github.com/BLAKE3-team/BLAKE3 - bytecount 0.6 Apache-2.0 OR MIT https://github.com/llogiq/bytecount - csv 1.4 Unlicense OR MIT https://github.com/BurntSushi/rust-csv - dirs 7.0 MIT OR Apache-2.0 https://codeberg.org/dirs/dirs-rs - hmac 0.13 MIT OR Apache-2.0 https://github.com/RustCrypto/MACs - itoa 1.0 MIT OR Apache-2.0 https://github.com/dtolnay/itoa - libc 0.2 MIT OR Apache-2.0 https://github.com/rust-lang/libc - maxminddb 0.32 ISC https://github.com/oschwald/maxminddb-rust - memchr 2.8 Unlicense OR MIT https://github.com/BurntSushi/memchr - memmap2 0.9 MIT OR Apache-2.0 https://github.com/RazrFalcon/memmap2-rs - psl 2.1 MIT OR Apache-2.0 https://github.com/addr-rs/psl - rand 0.10 MIT OR Apache-2.0 https://github.com/rust-random/rand - rand_chacha 0.10 MIT OR Apache-2.0 https://github.com/rust-random/rand - regex 1.13 MIT OR Apache-2.0 https://github.com/rust-lang/regex - roaring 0.11 MIT OR Apache-2.0 https://github.com/RoaringBitmap/roaring-rs - same-file 1.0 Unlicense OR MIT https://github.com/BurntSushi/same-file - serde 1.0 MIT OR Apache-2.0 https://github.com/serde-rs/serde - serde_json 1.0 MIT OR Apache-2.0 https://github.com/serde-rs/json - sha2 0.11 MIT OR Apache-2.0 https://github.com/RustCrypto/hashes - thiserror 2.0 MIT OR Apache-2.0 https://github.com/dtolnay/thiserror - -gridsift (command-line tool) - anyhow 1.0 MIT OR Apache-2.0 https://github.com/dtolnay/anyhow - clap 4.6 MIT OR Apache-2.0 https://github.com/clap-rs/clap - indicatif 0.18 MIT https://github.com/console-rs/indicatif - serde_json 1.0 MIT OR Apache-2.0 https://github.com/serde-rs/json - -gridsift-desktop - anyhow 1.0 MIT OR Apache-2.0 https://github.com/dtolnay/anyhow - eframe 0.32 MIT OR Apache-2.0 https://github.com/emilk/egui - egui 0.32 MIT OR Apache-2.0 https://github.com/emilk/egui - egui_extras 0.32 MIT OR Apache-2.0 https://github.com/emilk/egui - egui_plot 0.33 MIT OR Apache-2.0 https://github.com/emilk/egui_plot - rfd 0.17 MIT https://github.com/PolyMeilex/rfd - -Test-only dependencies - proptest 1.11 MIT OR Apache-2.0 https://github.com/proptest-rs/proptest - -Bundled data and fonts: - - The `psl` crate compiles a snapshot of the Public Suffix List - (https://publicsuffix.org/), which is published by Mozilla under the - Mozilla Public License 2.0. The list is used unmodified as data; the - snapshot version is recorded in every manifest that uses it. - - The desktop application embeds the fonts shipped with egui: Hack (MIT), - Ubuntu (Ubuntu Font Licence 1.0), Noto Emoji (SIL Open Font License 1.1) - and emoji-icon-font (MIT). - - No GeoIP / ASN database is bundled. MaxMind GeoLite2 and similar - databases are subject to their own licences and must be obtained and - imported by the user. - -Images in the repository (images/, shown in README.md and docs/demo.md): - - gridsift-dashboard.png, gridsift-desktop.png, gridsift-values.png and - gridsift-export.png show synthetic data from `gridsift gen --profile - demo` and are covered by the project licence above. - - demo-ctu13-dashboard.png and demo-ctu13-timeline.png show records of the - CTU-13 dataset, filtered, counted and charted by gridsift. CTU-13 was - published by the Stratosphere Laboratory, Czech Technical University in - Prague (https://www.stratosphereips.org/datasets-ctu13 ; files at - https://mcfp.felk.cvut.cz/publicDatasets/CTU-13-Dataset/ ) under the - Creative Commons Attribution 2.0 licence (CC BY 2.0, - https://creativecommons.org/licenses/by/2.0/ ). Reference: S. García, - M. Grill, J. Stiborek, A. Zunino, "An empirical comparison of botnet - detection methods", Computers & Security 45 (2014) 100–123. The data - visible in those two images remains under CC BY 2.0; the MIT / Apache-2.0 - licence of this project does not relicense it. diff --git a/crates/gridsift-core/NOTICE b/crates/gridsift-core/NOTICE index 5d2b0e1..e6eebe3 100644 --- a/crates/gridsift-core/NOTICE +++ b/crates/gridsift-core/NOTICE @@ -5,7 +5,7 @@ This project is dual-licensed under the MIT License and the Apache License, Version 2.0. See LICENSE-MIT and LICENSE-APACHE for the full texts. Third-party dependencies and their declared licenses (direct dependencies -of the three workspace crates; versions as pinned in Cargo.lock at the time +of the two workspace crates; versions as pinned in Cargo.lock at the time of writing): gridsift-core @@ -30,19 +30,16 @@ gridsift-core sha2 0.11 MIT OR Apache-2.0 https://github.com/RustCrypto/hashes thiserror 2.0 MIT OR Apache-2.0 https://github.com/dtolnay/thiserror -gridsift (command-line tool) +gridsift (the desktop application and the command-line tool) anyhow 1.0 MIT OR Apache-2.0 https://github.com/dtolnay/anyhow clap 4.6 MIT OR Apache-2.0 https://github.com/clap-rs/clap - indicatif 0.18 MIT https://github.com/console-rs/indicatif - serde_json 1.0 MIT OR Apache-2.0 https://github.com/serde-rs/json - -gridsift-desktop - anyhow 1.0 MIT OR Apache-2.0 https://github.com/dtolnay/anyhow eframe 0.32 MIT OR Apache-2.0 https://github.com/emilk/egui egui 0.32 MIT OR Apache-2.0 https://github.com/emilk/egui egui_extras 0.32 MIT OR Apache-2.0 https://github.com/emilk/egui egui_plot 0.33 MIT OR Apache-2.0 https://github.com/emilk/egui_plot + indicatif 0.18 MIT https://github.com/console-rs/indicatif rfd 0.17 MIT https://github.com/PolyMeilex/rfd + serde_json 1.0 MIT OR Apache-2.0 https://github.com/serde-rs/json Test-only dependencies proptest 1.11 MIT OR Apache-2.0 https://github.com/proptest-rs/proptest diff --git a/crates/gridsift-core/src/analysis.rs b/crates/gridsift-core/src/analysis.rs new file mode 100644 index 0000000..2e2eff0 --- /dev/null +++ b/crates/gridsift-core/src/analysis.rs @@ -0,0 +1,449 @@ +//! Cached whole-file analyses: the column profile, value counts and +//! timelines computed over all records of a file, kept in the user's cache +//! directory next to the index sidecar so that the next open of the same +//! bytes shows its charts without a scan. +//! +//! Only whole-file results are cached. Counts over a selection or of a +//! derived (enrichment) column are rarely repeated and would need the whole +//! lineage as their key, so they are recomputed. The cache is bound to the +//! source's size and modification time, and to its SHA-256 when both sides +//! know it; a file that changed gets nothing from it. +//! +//! Timelines are large (one bucket per second over the file's span) and +//! rarely wanted by whoever asks for a count, so they live in a companion +//! file (`.gstl` beside the `.gsan`) that is read only when a timeline is +//! asked for; the profile and the counts stay small and load with the file. + +use std::fs; +use std::io::{self, Write as _}; +use std::path::{Path, PathBuf}; + +use serde::{Deserialize, Serialize}; + +use crate::dialect::Dialect; +use crate::frequency::FrequencyResult; +use crate::semantic::Profile; +use crate::sidecar; +use crate::source::{Source, SourceId}; +use crate::timeline::TimelineResult; + +/// Format version; a cache written by another version is ignored. +pub const VERSION: u32 = 1; + +#[derive(Clone, Debug, Serialize, Deserialize)] +pub struct AnalysisCache { + pub version: u32, + /// Size and modification time of the source the results came from. + pub size: u64, + pub mtime_secs: u64, + pub mtime_nanos: u32, + /// SHA-256 of the source, lowercase hex, when it was known. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub sha256: Option, + /// The file-wide column profile. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub profile: Option, + /// Value counts over all records, one per column. + #[serde(default)] + pub counts: Vec, + /// Timelines computed in this session (per column and reference year); + /// stored in the companion file, not in this one. + #[serde(skip)] + pub timelines: Vec, + #[serde(skip)] + timelines_changed: bool, +} + +/// The companion file: the timelines, bound to the same bytes. +#[derive(Clone, Debug, Serialize, Deserialize)] +struct TimelineFile { + version: u32, + size: u64, + mtime_secs: u64, + mtime_nanos: u32, + #[serde(default, skip_serializing_if = "Option::is_none")] + sha256: Option, + #[serde(default)] + timelines: Vec, +} + +impl TimelineFile { + fn matches(&self, id: SourceId, sha256: Option<&str>) -> bool { + self.version == VERSION + && self.size == id.size + && self.mtime_secs == id.mtime_secs + && self.mtime_nanos == id.mtime_nanos + && match (&self.sha256, sha256) { + (Some(a), Some(b)) => a == b, + _ => true, + } + } +} + +#[derive(Clone, Debug, Serialize, Deserialize)] +pub struct CachedCount { + /// How many top values were asked for; the entry serves requests up to + /// that many (or any number when the list is already complete). + pub top: usize, + pub result: FrequencyResult, +} + +impl AnalysisCache { + pub fn new(id: SourceId, sha256: Option) -> AnalysisCache { + AnalysisCache { + version: VERSION, + size: id.size, + mtime_secs: id.mtime_secs, + mtime_nanos: id.mtime_nanos, + sha256, + profile: None, + counts: Vec::new(), + timelines: Vec::new(), + timelines_changed: false, + } + } + + /// The companion file holding the timelines of the cache at `path`. + pub fn timelines_path(path: &Path) -> PathBuf { + path.with_extension("gstl") + } + + /// Default location: the user's cache directory, keyed like the index + /// sidecar by canonical path and parser settings. + pub fn default_path(source: &Path, dialect: Dialect) -> io::Result { + sidecar::default_analysis_path(source, dialect) + } + + /// The cache describes these bytes: same size and modification time, + /// and the same SHA-256 when both sides know it. + pub fn matches(&self, id: SourceId, sha256: Option<&str>) -> bool { + self.version == VERSION + && self.size == id.size + && self.mtime_secs == id.mtime_secs + && self.mtime_nanos == id.mtime_nanos + && match (&self.sha256, sha256) { + (Some(a), Some(b)) => a == b, + _ => true, + } + } + + /// Read a cache file; a missing or unreadable one is simply absent. + pub fn load(path: &Path) -> Option { + serde_json::from_slice(&fs::read(path).ok()?).ok() + } + + /// Read the cache file if it describes these bytes. + pub fn load_for(path: &Path, id: SourceId, sha256: Option<&str>) -> Option { + Self::load(path).filter(|c| c.matches(id, sha256)) + } + + /// Learn the digest once it is known, so later loads can check it. + pub fn set_sha256(&mut self, sha256: String) { + self.sha256 = Some(sha256); + } + + pub fn is_empty(&self) -> bool { + self.profile.is_none() && self.counts.is_empty() && self.timelines.is_empty() + } + + /// The timeline of `column` for `reference_year` from the companion + /// file of the cache at `path`, if that file describes these bytes. + pub fn load_timeline( + path: &Path, + id: SourceId, + sha256: Option<&str>, + column: usize, + reference_year: i64, + ) -> Option { + let bytes = fs::read(Self::timelines_path(path)).ok()?; + let file: TimelineFile = serde_json::from_slice(&bytes).ok()?; + if !file.matches(id, sha256) { + return None; + } + file.timelines + .into_iter() + .find(|t| t.column == column && t.reference_year == reference_year) + } + + /// The cached count of `column`, cut to `top` values, if the cached + /// entry was asked for at least that many (or holds the complete list). + pub fn count(&self, column: usize, top: usize) -> Option { + let c = self.counts.iter().find(|c| c.result.column == column)?; + let complete_list = c.result.top.len() < c.top; + if c.top < top && !complete_list { + return None; + } + let mut r = c.result.clone(); + r.top.truncate(top); + Some(r) + } + + /// Remember a complete count asked for `top` values; a wider request + /// replaces a narrower one for the same column, never the reverse. + pub fn put_count(&mut self, top: usize, result: FrequencyResult) { + if !result.complete { + return; + } + match self + .counts + .iter_mut() + .find(|c| c.result.column == result.column) + { + Some(c) if top >= c.top => { + c.top = top; + c.result = result; + } + Some(_) => {} + None => self.counts.push(CachedCount { top, result }), + } + } + + /// A timeline computed in this session (the companion file is read by + /// [`AnalysisCache::load_timeline`]). + pub fn timeline(&self, column: usize, reference_year: i64) -> Option<&TimelineResult> { + self.timelines + .iter() + .find(|t| t.column == column && t.reference_year == reference_year) + } + + pub fn put_timeline(&mut self, result: TimelineResult) { + if !result.complete { + return; + } + self.timelines_changed = true; + match self + .timelines + .iter_mut() + .find(|t| t.column == result.column && t.reference_year == result.reference_year) + { + Some(t) => *t = result, + None => self.timelines.push(result), + } + } + + /// Remember the file-wide profile (one drawn from the whole file, not + /// from the first rows). + pub fn put_profile(&mut self, profile: Profile) { + if profile.spans_file { + self.profile = Some(profile); + } + } + + /// Write the cache (temp file + rename) after checking that neither the + /// target nor its temporary file is the source. Timelines computed in + /// this session go to the companion file, merged with the ones already + /// there for the same bytes. + pub fn save_for(&mut self, source: &Source, path: &Path) -> io::Result<()> { + let bytes = serde_json::to_vec(self).map_err(io::Error::other)?; + write_atomically(source, path, &bytes)?; + if self.timelines_changed { + let tpath = Self::timelines_path(path); + let id = SourceId { + size: self.size, + mtime_secs: self.mtime_secs, + mtime_nanos: self.mtime_nanos, + }; + let mut file = fs::read(&tpath) + .ok() + .and_then(|b| serde_json::from_slice::(&b).ok()) + .filter(|f| f.matches(id, self.sha256.as_deref())) + .unwrap_or(TimelineFile { + version: VERSION, + size: self.size, + mtime_secs: self.mtime_secs, + mtime_nanos: self.mtime_nanos, + sha256: self.sha256.clone(), + timelines: Vec::new(), + }); + file.sha256 = self.sha256.clone().or(file.sha256); + for t in &self.timelines { + match file + .timelines + .iter_mut() + .find(|f| f.column == t.column && f.reference_year == t.reference_year) + { + Some(f) => *f = t.clone(), + None => file.timelines.push(t.clone()), + } + } + let bytes = serde_json::to_vec(&file).map_err(io::Error::other)?; + write_atomically(source, &tpath, &bytes)?; + self.timelines_changed = false; + } + Ok(()) + } +} + +/// Temp file + rename, refusing a path (or its temporary) that is the source. +fn write_atomically(source: &Source, path: &Path, bytes: &[u8]) -> io::Result<()> { + let tmp = temp_path(path); + source.guard_not_source(path)?; + source.guard_not_source(&tmp)?; + if let Some(dir) = path.parent() { + fs::create_dir_all(dir)?; + } + { + let mut f = fs::File::create(&tmp)?; + f.write_all(bytes)?; + f.sync_all()?; + } + fs::rename(&tmp, path) +} + +fn temp_path(path: &Path) -> PathBuf { + let mut name = path + .file_name() + .map(|s| s.to_os_string()) + .unwrap_or_else(|| "analysis".into()); + name.push(".tmp"); + path.with_file_name(name) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::frequency::FrequencyEntry; + use std::time::Duration; + + fn count(column: usize, values: &[(&str, u64)]) -> FrequencyResult { + FrequencyResult { + column, + counted: values.iter().map(|v| v.1).sum(), + empty: 0, + distinct: values.len() as u64, + exact: true, + error_bound: 0, + top: values + .iter() + .map(|(v, n)| FrequencyEntry { + value: v.as_bytes().to_vec(), + count: *n, + }) + .collect(), + complete: true, + elapsed: Duration::from_millis(1), + threads: 1, + } + } + + fn id(size: u64) -> SourceId { + SourceId { + size, + mtime_secs: 1_700_000_000, + mtime_nanos: 5, + } + } + + #[test] + fn counts_serve_requests_up_to_what_was_asked() { + let mut c = AnalysisCache::new(id(10), None); + let twelve: Vec<(String, u64)> = (0..12).map(|i| (format!("v{i}"), 20 - i)).collect(); + let twelve: Vec<(&str, u64)> = twelve.iter().map(|(v, n)| (v.as_str(), *n)).collect(); + c.put_count(12, count(3, &twelve)); + assert_eq!(c.count(3, 5).unwrap().top.len(), 5); + assert_eq!(c.count(3, 12).unwrap().top.len(), 12); + assert!(c.count(3, 20).is_none(), "asked for more than was counted"); + assert!(c.count(4, 1).is_none()); + // a narrower request never replaces a wider entry … + c.put_count(5, count(3, &twelve[..5])); + assert_eq!(c.count(3, 12).unwrap().top.len(), 12); + // … a wider one does + c.put_count(500, count(3, &twelve)); + assert_eq!( + c.count(3, 1000).unwrap().top.len(), + 12, + "the list is complete" + ); + // incomplete results are not kept + let mut partial = count(7, &twelve[..2]); + partial.complete = false; + c.put_count(12, partial); + assert!(c.count(7, 1).is_none()); + } + + #[test] + fn cache_is_bound_to_the_bytes() { + let c = AnalysisCache::new(id(10), Some("ab".into())); + assert!(c.matches(id(10), Some("ab"))); + assert!( + c.matches(id(10), None), + "an unknown digest is not a mismatch" + ); + assert!(!c.matches(id(11), Some("ab"))); + assert!(!c.matches(id(10), Some("cd"))); + let mut other_time = id(10); + other_time.mtime_nanos = 6; + assert!(!c.matches(other_time, Some("ab"))); + let unknown = AnalysisCache::new(id(10), None); + assert!(unknown.matches(id(10), Some("ab"))); + } + + #[test] + fn round_trip_through_the_file_and_the_source_guard() { + let dir = std::env::temp_dir().join(format!("gridsift-analysis-{}", std::process::id())); + fs::create_dir_all(&dir).unwrap(); + let src_path = dir.join("src.csv"); + fs::write(&src_path, "a,b\n1,2\n").unwrap(); + let src = Source::open(&src_path).unwrap(); + let mut c = AnalysisCache::new(src.id(), None); + c.put_count(10, count(1, &[("2", 1)])); + c.put_timeline(TimelineResult { + column: 0, + reference_year: 2026, + counted: 1, + parsed: 1, + unparsed: 0, + resolution: 1, + buckets: vec![(0, 1)], + min: Some(0), + max: Some(0), + complete: true, + elapsed: Duration::from_millis(1), + threads: 1, + }); + let path = dir.join("cache").join("src.csv.key.gsan"); + c.save_for(&src, &path).unwrap(); + let back = AnalysisCache::load_for(&path, src.id(), None).expect("same bytes"); + assert_eq!(back.count(1, 10).unwrap().top[0].value, b"2"); + // timelines live in the companion file, read on demand + assert!(back.timelines.is_empty()); + assert!(AnalysisCache::timelines_path(&path).is_file()); + let t = AnalysisCache::load_timeline(&path, src.id(), None, 0, 2026).unwrap(); + assert_eq!(t.buckets, [(0, 1)]); + assert!( + AnalysisCache::load_timeline(&path, src.id(), None, 0, 2027).is_none(), + "another year parses differently" + ); + let mut changed = src.id(); + changed.size += 1; + assert!(AnalysisCache::load_for(&path, changed, None).is_none()); + assert!(AnalysisCache::load_timeline(&path, changed, None, 0, 2026).is_none()); + // a later session adds another column's timeline without losing this one + let mut later = AnalysisCache::load_for(&path, src.id(), None).unwrap(); + later.put_timeline(TimelineResult { + column: 1, + reference_year: 2026, + counted: 1, + parsed: 1, + unparsed: 0, + resolution: 1, + buckets: vec![(5, 1)], + min: Some(5), + max: Some(5), + complete: true, + elapsed: Duration::from_millis(1), + threads: 1, + }); + later.save_for(&src, &path).unwrap(); + assert!(AnalysisCache::load_timeline(&path, src.id(), None, 0, 2026).is_some()); + assert_eq!( + AnalysisCache::load_timeline(&path, src.id(), None, 1, 2026) + .unwrap() + .buckets, + [(5, 1)] + ); + // never over the evidence + assert!(c.save_for(&src, &src_path).is_err()); + assert_eq!(fs::read(&src_path).unwrap(), b"a,b\n1,2\n"); + } +} diff --git a/crates/gridsift-core/src/frequency.rs b/crates/gridsift-core/src/frequency.rs index 68112ce..0cae8d4 100644 --- a/crates/gridsift-core/src/frequency.rs +++ b/crates/gridsift-core/src/frequency.rs @@ -14,6 +14,8 @@ use std::sync::Mutex; use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering}; use std::time::{Duration, Instant}; +use serde::{Deserialize, Serialize}; + use crate::enrich::Enrichment; use crate::export::Selection; use crate::index::{Checkpoint, SparseIndex}; @@ -86,14 +88,14 @@ impl FrequencyShared { } } -#[derive(Clone, Debug, PartialEq, Eq)] +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] pub struct FrequencyEntry { /// The field bytes exactly as stored (unquoted). pub value: Vec, pub count: u64, } -#[derive(Clone, Debug)] +#[derive(Clone, Debug, Serialize, Deserialize)] pub struct FrequencyResult { pub column: usize, /// Records in the selection that were examined. diff --git a/crates/gridsift-core/src/lib.rs b/crates/gridsift-core/src/lib.rs index 82e9f29..1de8f5a 100644 --- a/crates/gridsift-core/src/lib.rs +++ b/crates/gridsift-core/src/lib.rs @@ -26,6 +26,7 @@ //! - [`synth`] deterministic synthetic datasets for benchmarks //! - [`sys`] process probes and formatting helpers for reports +pub mod analysis; pub mod dialect; pub mod enrich; pub mod export; diff --git a/crates/gridsift-core/src/manifest.rs b/crates/gridsift-core/src/manifest.rs index 2cc8575..a89cf84 100644 --- a/crates/gridsift-core/src/manifest.rs +++ b/crates/gridsift-core/src/manifest.rs @@ -99,6 +99,10 @@ pub struct DialectInfo { pub delimiter: String, pub quote: Option, pub header: bool, + /// Column names the analyst gave a file without a header (`--names`); + /// absent when the names came from the file. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub names: Option>, } impl From for DialectInfo { @@ -107,6 +111,7 @@ impl From for DialectInfo { delimiter: (d.delimiter as char).to_string(), quote: d.quote.map(|q| (q as char).to_string()), header: d.has_header, + names: None, } } } diff --git a/crates/gridsift-core/src/semantic.rs b/crates/gridsift-core/src/semantic.rs index 46f0d46..b20e40e 100644 --- a/crates/gridsift-core/src/semantic.rs +++ b/crates/gridsift-core/src/semantic.rs @@ -354,8 +354,8 @@ pub struct ColumnProfile { pub struct Profile { pub columns: Vec, pub sampled_records: u64, - /// The sample was drawn from positions across the whole file, not just - /// the head. + /// The sample was drawn from positions across the whole file (or the + /// head was the whole file), not just from the first rows. pub spans_file: bool, } @@ -559,6 +559,9 @@ pub fn profile( let recs = locate_many(source, index, &ordinals); spans_file = !recs.is_empty(); add_records(&recs, &mut accs); + } else if index.stats.complete && total <= opts.head_rows as u64 { + // the head was the whole file: nothing was left unsampled + spans_file = true; } finish_all(accs, header, n, spans_file) } diff --git a/crates/gridsift-core/src/sidecar.rs b/crates/gridsift-core/src/sidecar.rs index 238db1c..5e59818 100644 --- a/crates/gridsift-core/src/sidecar.rs +++ b/crates/gridsift-core/src/sidecar.rs @@ -21,13 +21,14 @@ pub fn cache_root() -> PathBuf { .join("gridsift") } -/// Default location of the sparse index for `source` parsed with `dialect`. +/// File stem and key of `source` parsed with `dialect`: the canonical path +/// and the parser settings, hashed. /// /// The dialect is part of the key: an index records where records start, /// and that depends on the delimiter, quoting and header settings, so a /// file opened with `--no-header` must not pick up the index that was built -/// with a header. -pub fn default_index_path(source: &Path, dialect: Dialect) -> io::Result { +/// with a header; the same goes for the analyses computed over its records. +fn sidecar_key(source: &Path, dialect: Dialect) -> io::Result<(String, String)> { let canon = fs::canonicalize(source)?; let mut keyed = canon.to_string_lossy().into_owned().into_bytes(); keyed.push(0); @@ -39,9 +40,31 @@ pub fn default_index_path(source: &Path, dialect: Dialect) -> io::Result io::Result { + let (stem, key) = sidecar_key(source, dialect)?; + Ok(root.join("index").join(format!("{stem}.{key}.gsix"))) +} + +/// Default location of the sparse index for `source` parsed with `dialect`. +pub fn default_index_path(source: &Path, dialect: Dialect) -> io::Result { + index_path_in(&cache_root(), source, dialect) +} + +/// Location of the cached whole-file analyses (column profile, value +/// counts, timelines) of `source` parsed with `dialect` under `root`. +pub fn analysis_path_in(root: &Path, source: &Path, dialect: Dialect) -> io::Result { + let (stem, key) = sidecar_key(source, dialect)?; + Ok(root.join("analysis").join(format!("{stem}.{key}.gsan"))) +} + +/// Default location of the cached whole-file analyses of `source`. +pub fn default_analysis_path(source: &Path, dialect: Dialect) -> io::Result { + analysis_path_in(&cache_root(), source, dialect) } #[cfg(test)] @@ -76,5 +99,14 @@ mod tests { ); assert!(pa.extension().is_some_and(|e| e == "gsix")); assert!(default_index_path(&dir.join("missing.csv"), d).is_err()); + // the analyses share the key and sit beside the index tree + let an = default_analysis_path(&a, d).unwrap(); + assert!(an.extension().is_some_and(|e| e == "gsan")); + assert_eq!(an.file_stem(), pa.file_stem()); + assert_ne!(an, default_analysis_path(&a, no_header).unwrap()); + assert_eq!( + index_path_in(Path::new("/x"), &a, d).unwrap(), + Path::new("/x").join("index").join(pa.file_name().unwrap()) + ); } } diff --git a/crates/gridsift-core/src/timeline.rs b/crates/gridsift-core/src/timeline.rs index 712c8e0..efb1366 100644 --- a/crates/gridsift-core/src/timeline.rs +++ b/crates/gridsift-core/src/timeline.rs @@ -10,6 +10,8 @@ use std::collections::HashMap; use std::io; + +use serde::{Deserialize, Serialize}; use std::sync::Mutex; use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; use std::time::{Duration, Instant}; @@ -391,9 +393,13 @@ impl Buckets { } } -#[derive(Clone, Debug)] +#[derive(Clone, Debug, Serialize, Deserialize)] pub struct TimelineResult { pub column: usize, + /// Year assumed for formats without one (part of a cached result's + /// identity: the same bytes parse differently in another year). + #[serde(default)] + pub reference_year: i64, /// Records in the selection that were examined. pub counted: u64, pub parsed: u64, @@ -562,6 +568,7 @@ pub fn timeline( buckets.sort_unstable(); Ok(TimelineResult { column: opts.column, + reference_year: opts.reference_year, counted: counted.load(Ordering::Relaxed), parsed, unparsed, diff --git a/crates/gridsift-desktop/Cargo.toml b/crates/gridsift-desktop/Cargo.toml deleted file mode 100644 index 7e49933..0000000 --- a/crates/gridsift-desktop/Cargo.toml +++ /dev/null @@ -1,25 +0,0 @@ -[package] -name = "gridsift-desktop" -description = "Offline, evidence-safe desktop workbench for very large CSV security data (egui)" -version.workspace = true -edition.workspace = true -rust-version.workspace = true -license.workspace = true -repository.workspace = true -homepage.workspace = true -readme.workspace = true -keywords.workspace = true -categories.workspace = true - -[[bin]] -name = "gridsift-desktop" -path = "src/main.rs" - -[dependencies] -anyhow = "1.0.104" -eframe = "0.32.3" -egui = "0.32.3" -egui_extras = "0.32.3" -egui_plot = "0.33" -gridsift-core = { path = "../gridsift-core", version = "0.1.0" } -rfd = "0.17.2" diff --git a/crates/gridsift-desktop/LICENSE-APACHE b/crates/gridsift-desktop/LICENSE-APACHE deleted file mode 100644 index d645695..0000000 --- a/crates/gridsift-desktop/LICENSE-APACHE +++ /dev/null @@ -1,202 +0,0 @@ - - Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "[]" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright [yyyy] [name of copyright owner] - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. diff --git a/crates/gridsift-desktop/LICENSE-MIT b/crates/gridsift-desktop/LICENSE-MIT deleted file mode 100644 index b026aec..0000000 --- a/crates/gridsift-desktop/LICENSE-MIT +++ /dev/null @@ -1,21 +0,0 @@ -MIT License - -Copyright (c) 2026 gridsift contributors - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all -copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -SOFTWARE. diff --git a/crates/gridsift-cli/Cargo.toml b/crates/gridsift/Cargo.toml similarity index 58% rename from crates/gridsift-cli/Cargo.toml rename to crates/gridsift/Cargo.toml index ca8c85c..289fe8a 100644 --- a/crates/gridsift-cli/Cargo.toml +++ b/crates/gridsift/Cargo.toml @@ -1,6 +1,8 @@ [package] name = "gridsift" -description = "Offline, evidence-safe command-line workbench for very large CSV security data" +description = "Offline, evidence-safe workbench for very large CSV security data: desktop application and command-line tool" +# `cargo run` starts the desktop application; the command line is `--bin gridsift` +default-run = "gridsift-desktop" version.workspace = true edition.workspace = true rust-version.workspace = true @@ -11,13 +13,14 @@ readme.workspace = true keywords.workspace = true categories.workspace = true -[[bin]] -name = "gridsift" -path = "src/main.rs" - [dependencies] anyhow = "1.0.104" clap = { version = "4.6.7", features = ["derive"] } +eframe = "0.32.3" +egui = "0.32.3" +egui_extras = "0.32.3" +egui_plot = "0.33" gridsift-core = { path = "../gridsift-core", version = "0.1.0" } indicatif = "0.18.6" +rfd = "0.17.2" serde_json = "1.0.151" diff --git a/crates/gridsift-cli/LICENSE-APACHE b/crates/gridsift/LICENSE-APACHE similarity index 100% rename from crates/gridsift-cli/LICENSE-APACHE rename to crates/gridsift/LICENSE-APACHE diff --git a/crates/gridsift-cli/LICENSE-MIT b/crates/gridsift/LICENSE-MIT similarity index 100% rename from crates/gridsift-cli/LICENSE-MIT rename to crates/gridsift/LICENSE-MIT diff --git a/crates/gridsift-desktop/NOTICE b/crates/gridsift/NOTICE similarity index 95% rename from crates/gridsift-desktop/NOTICE rename to crates/gridsift/NOTICE index 5d2b0e1..e6eebe3 100644 --- a/crates/gridsift-desktop/NOTICE +++ b/crates/gridsift/NOTICE @@ -5,7 +5,7 @@ This project is dual-licensed under the MIT License and the Apache License, Version 2.0. See LICENSE-MIT and LICENSE-APACHE for the full texts. Third-party dependencies and their declared licenses (direct dependencies -of the three workspace crates; versions as pinned in Cargo.lock at the time +of the two workspace crates; versions as pinned in Cargo.lock at the time of writing): gridsift-core @@ -30,19 +30,16 @@ gridsift-core sha2 0.11 MIT OR Apache-2.0 https://github.com/RustCrypto/hashes thiserror 2.0 MIT OR Apache-2.0 https://github.com/dtolnay/thiserror -gridsift (command-line tool) +gridsift (the desktop application and the command-line tool) anyhow 1.0 MIT OR Apache-2.0 https://github.com/dtolnay/anyhow clap 4.6 MIT OR Apache-2.0 https://github.com/clap-rs/clap - indicatif 0.18 MIT https://github.com/console-rs/indicatif - serde_json 1.0 MIT OR Apache-2.0 https://github.com/serde-rs/json - -gridsift-desktop - anyhow 1.0 MIT OR Apache-2.0 https://github.com/dtolnay/anyhow eframe 0.32 MIT OR Apache-2.0 https://github.com/emilk/egui egui 0.32 MIT OR Apache-2.0 https://github.com/emilk/egui egui_extras 0.32 MIT OR Apache-2.0 https://github.com/emilk/egui egui_plot 0.33 MIT OR Apache-2.0 https://github.com/emilk/egui_plot + indicatif 0.18 MIT https://github.com/console-rs/indicatif rfd 0.17 MIT https://github.com/PolyMeilex/rfd + serde_json 1.0 MIT OR Apache-2.0 https://github.com/serde-rs/json Test-only dependencies proptest 1.11 MIT OR Apache-2.0 https://github.com/proptest-rs/proptest diff --git a/crates/gridsift-desktop/src/cache.rs b/crates/gridsift/src/bin/gridsift-desktop/cache.rs similarity index 100% rename from crates/gridsift-desktop/src/cache.rs rename to crates/gridsift/src/bin/gridsift-desktop/cache.rs diff --git a/crates/gridsift-desktop/src/dashboard.rs b/crates/gridsift/src/bin/gridsift-desktop/dashboard.rs similarity index 100% rename from crates/gridsift-desktop/src/dashboard.rs rename to crates/gridsift/src/bin/gridsift-desktop/dashboard.rs diff --git a/crates/gridsift-desktop/src/dialogs.rs b/crates/gridsift/src/bin/gridsift-desktop/dialogs.rs similarity index 89% rename from crates/gridsift-desktop/src/dialogs.rs rename to crates/gridsift/src/bin/gridsift-desktop/dialogs.rs index 7fa65c6..270ca9a 100644 --- a/crates/gridsift-desktop/src/dialogs.rs +++ b/crates/gridsift/src/bin/gridsift-desktop/dialogs.rs @@ -189,6 +189,56 @@ pub fn export_window(ctx: &egui::Context, d: &mut Document) { } /// The enrichment dialog: assemble rules, apply them to the document. +/// Column names for a file without a header. Returns the names to reopen +/// the file with once the analyst applies them. +pub fn names_window(ctx: &egui::Context, d: &mut Document) -> Option> { + if !d.names_ui.open { + return None; + } + let mut open = true; + let mut applied = None; + egui::Window::new("Column names") + .collapsible(false) + .resizable(false) + .default_width(520.0) + .open(&mut open) + .show(ctx, |ui| { + ui.label( + RichText::new( + "This file has no header record. Name its columns here, comma-separated and \ + in order; a missing name stays col0, col1, … The names live in gridsift and \ + in the manifests of exports, never in the source file.", + ) + .color(DIM), + ); + ui.add_space(6.0); + ui.add( + egui::TextEdit::singleline(&mut d.names_ui.text) + .desired_width(f32::INFINITY) + .hint_text("time,src_user,dst_computer,…"), + ); + ui.add_space(6.0); + ui.horizontal(|ui| { + if ui.button("Apply").clicked() { + applied = Some( + d.names_ui + .text + .split(',') + .map(|s| s.trim().to_string()) + .collect::>(), + ); + } + if ui.button("Cancel").clicked() { + d.names_ui.open = false; + } + }); + }); + if !open || applied.is_some() { + d.names_ui.open = false; + } + applied +} + pub fn enrich_window(ctx: &egui::Context, d: &mut Document) { if !d.enrich_ui.open { return; diff --git a/crates/gridsift-desktop/src/document.rs b/crates/gridsift/src/bin/gridsift-desktop/document.rs similarity index 82% rename from crates/gridsift-desktop/src/document.rs rename to crates/gridsift/src/bin/gridsift-desktop/document.rs index de63e51..d08b4a1 100644 --- a/crates/gridsift-desktop/src/document.rs +++ b/crates/gridsift/src/bin/gridsift-desktop/document.rs @@ -8,10 +8,11 @@ use std::time::{Duration, Instant}; use eframe::egui; use gridsift_core::Source; +use gridsift_core::analysis::AnalysisCache; use gridsift_core::dialect::sniff; use gridsift_core::enrich::{EnrichRule, Enrichment, GeoIpDb, GeoProvider, LookupTable, Provider}; use gridsift_core::export::{ExportOptions, Selection, export_pending}; -use gridsift_core::frequency::{FrequencyOptions, FrequencyShared, frequency}; +use gridsift_core::frequency::{FrequencyOptions, FrequencyResult, FrequencyShared, frequency}; use gridsift_core::hash::{Digests, HashSelection, hash_source, hex}; use gridsift_core::index::{IndexParams, SparseIndex, bootstrap}; use gridsift_core::manifest::{Manifest, Operation, OutputInfo, SelectionInfo, SourceInfo}; @@ -23,7 +24,9 @@ use gridsift_core::search::{ MatchSet, PatternKind, SearchOptions, SearchOutcome, SearchQuery, SearchShared, search, }; use gridsift_core::semantic::{Profile, ProfileOptions, SemanticType, profile, profile_rows}; -use gridsift_core::sidecar::default_index_path; +use gridsift_core::sidecar::{ + analysis_path_in, default_analysis_path, default_index_path, index_path_in, +}; use gridsift_core::sys::{boost_current_thread, group_thousands, human_bytes}; use gridsift_core::timeline::{TimelineOptions, current_year, select_time_range, timeline}; @@ -344,12 +347,36 @@ pub enum DockTab { // --------------------------------------------------------------------------- // the document +/// How to read a file: the sniffer's choices, or the analyst's. +#[derive(Clone, Debug, Default)] +pub struct OpenOptions { + /// Treat the first record as a header (`Some(true)`) or as data + /// (`Some(false)`); `None` leaves it to the sniffer. + pub header: Option, + /// Column names for a file without a header (implies no header). + pub names: Option>, + /// Root of the cache tree (index sidecars, analysis caches) instead of + /// the user's; for tests. + pub cache_root: Option, +} + +/// The column-naming dialog. +#[derive(Default)] +pub struct NamesUi { + pub open: bool, + pub text: String, +} + pub struct Document { pub path: PathBuf, pub source: Arc, pub params: IndexParams, pub index: Arc>, pub header: Vec, + /// Column names the analyst gave a file without a header; exports + /// record them in the manifest. + pub names: Option>, + pub names_ui: NamesUi, /// Rows known to exist so far (probe, then build progress). pub known_rows: u64, pub build: Option, @@ -393,6 +420,11 @@ pub struct Document { /// A cached index without a digest gets one from this pass. pub digest_job: Option, + /// Whole-file analyses (profile, counts, timelines) kept between + /// sessions in the cache directory; filled as results come in. + pub analysis: Option, + pub analysis_path: Option, + analysis_dirty: bool, /// The source's size or mtime changed after it was opened: the index, /// digest and every selection are stale. Set by [`Document::poll_all`]. pub source_changed: bool, @@ -409,20 +441,42 @@ pub struct DigestJob { } impl Document { - pub fn open(ctx: &egui::Context, path: &Path) -> Result { + /// Open a file. `opts` overrides the sniffer's reading: header or not, + /// and names for the columns of a file without one. + pub fn open_with( + ctx: &egui::Context, + path: &Path, + opts: OpenOptions, + ) -> Result { let t0 = Instant::now(); let source = Arc::new(Source::open(path).map_err(|e| format!("{}: {e}", path.display()))?); let head = source.slice(0, 1 << 20); let sn = sniff(head, source.len()); + let mut dialect = sn.dialect; + if let Some(h) = opts.header { + dialect.has_header = h; + } + if opts.names.is_some() { + dialect.has_header = false; + } let params = IndexParams { - dialect: sn.dialect, + dialect, scan_start: sn.scan_start, ..IndexParams::default() }; // A complete sidecar for this file *and* this dialect means no // build is needed; a missing digest is computed separately below. - let sidecar = default_index_path(path, params.dialect).ok(); + let (sidecar, analysis_path) = match &opts.cache_root { + Some(root) => ( + index_path_in(root, path, params.dialect).ok(), + analysis_path_in(root, path, params.dialect).ok(), + ), + None => ( + default_index_path(path, params.dialect).ok(), + default_analysis_path(path, params.dialect).ok(), + ), + }; let cached = sidecar .as_ref() .and_then(|p| SparseIndex::load(p).ok()) @@ -445,6 +499,37 @@ impl Document { .map(|i| format!("col{i}")) .collect(), }; + // the analyst's names for a header-less file; gaps keep col + let header: Vec = match (&opts.names, params.dialect.has_header) { + (Some(names), false) => (0..header.len().max(names.len())) + .map(|i| { + names + .get(i) + .filter(|n| !n.is_empty()) + .cloned() + .unwrap_or_else(|| format!("col{i}")) + }) + .collect(), + _ => header, + }; + // whole-file analyses from an earlier session, if they describe + // these very bytes; the profile is by column index, so it takes + // the current names + let sha = index.digests.sha256.map(|d| hex(&d)); + let analysis = analysis_path + .as_ref() + .and_then(|p| AnalysisCache::load_for(p, source.id(), sha.as_deref())); + let cached_profile = analysis + .as_ref() + .and_then(|a| a.profile.clone()) + .filter(|p| p.spans_file && p.columns.len() == header.len()) + .map(|mut p| { + for (c, name) in p.columns.iter_mut().zip(&header) { + c.name = name.clone(); + } + p + }); + let profile_from_cache = cached_profile.is_some(); let mut cache = RowCache::default(); let probed = cache.fill_window(&source, &index, None, 0, PROBE_ROWS); @@ -459,7 +544,7 @@ impl Document { // With a complete index already on disk there is no build to wait // for, so the file-wide profile starts right away. - let profile_job = from_sidecar.then(|| { + let profile_job = (from_sidecar && !profile_from_cache).then(|| { let (s, i, h) = (source.clone(), index.clone(), header.clone()); std::thread::Builder::new() .name("gridsift-profile".into()) @@ -486,7 +571,7 @@ impl Document { handle: Some(handle), } }); - Ok(Document { + let mut doc = Document { path: path.to_path_buf(), source, params, @@ -504,7 +589,7 @@ impl Document { selection: None, export: None, export_ui: ExportUi::default(), - profile: Some(quick_profile), + profile: Some(cached_profile.unwrap_or(quick_profile)), profile_job, enrichment: None, derived_names: Vec::new(), @@ -514,7 +599,7 @@ impl Document { freq_job: None, freq: None, dock_tab: DockTab::Values, - dock_open: false, + dock_open: true, dashboard: Dashboard::default(), first_rows_in, index_elapsed: None, @@ -525,7 +610,16 @@ impl Document { source_changed: false, last_source_check: Instant::now(), enrichment_gen: 0, - }) + names: opts.names, + names_ui: NamesUi::default(), + analysis, + analysis_path, + analysis_dirty: false, + }; + if profile_from_cache { + doc.auto_build_dashboard(); + } + Ok(doc) } // -- names and columns ------------------------------------------------- @@ -645,6 +739,7 @@ impl Document { self.poll_timeline(); self.poll_dashboard(); self.poll_export(); + self.save_analysis(false); } fn poll_digest(&mut self) { @@ -658,10 +753,15 @@ impl Document { self.digest_job = None; match handle.join() { Ok(Ok(Some(d))) => { + let sha = d.sha256.map(|s| hex(&s)); let saved = self.index.write().ok().map(|mut w| { w.digests = d; w.clone() }); + if let (Some(sha), Some(a)) = (sha, &mut self.analysis) { + a.set_sha256(sha); + self.analysis_dirty = true; + } if let (Some(idx), Some(p)) = (saved, &self.sidecar) && let Err(e) = idx.save_for(&self.source, p) { @@ -690,7 +790,9 @@ impl Document { self.known_rows = idx.stats.records; let complete = idx.stats.complete; if complete { - self.spawn_profile(idx.clone()); + if !self.profile.as_ref().is_some_and(|p| p.spans_file) { + self.spawn_profile(idx.clone()); + } if let Some(p) = &self.sidecar && let Err(e) = idx.save_for(&self.source, p) { @@ -726,7 +828,10 @@ impl Document { && let Some(h) = self.profile_job.take() && let Ok(p) = h.join() { - self.profile = Some(p); + self.profile = Some(p.clone()); + if let Some(a) = self.analysis_mut() { + a.put_profile(p); + } // the dashboard is built from the file-wide profile and counted // in the background, so it is ready when its tab is opened if !self.dashboard.auto_built { @@ -735,6 +840,59 @@ impl Document { } } + // -- analysis cache ---------------------------------------------------- + + /// The cache to record whole-file results in, created on first use; + /// none while the source is known to have changed or when there is + /// nowhere to keep it. Asking for it marks it dirty. + fn analysis_mut(&mut self) -> Option<&mut AnalysisCache> { + if self.source_changed || self.analysis_path.is_none() { + return None; + } + let sha = self + .index + .read() + .ok() + .and_then(|i| i.digests.sha256.map(|d| hex(&d))); + let id = self.source.id(); + self.analysis_dirty = true; + Some( + self.analysis + .get_or_insert_with(|| AnalysisCache::new(id, sha)), + ) + } + + /// Keep a whole-file count of a source column for the next session. + fn remember_count(&mut self, top: usize, result: &FrequencyResult) { + if result.column < self.header.len() + && let Some(a) = self.analysis_mut() + { + a.put_count(top, result.clone()); + } + } + + /// Write the analysis cache when something new is in it: once the scans + /// are idle, or right away when `now` (closing the file). + fn save_analysis(&mut self, now: bool) { + if !self.analysis_dirty || self.source_changed { + return; + } + let idle = self.freq_job.is_none() + && self.timeline_job.is_none() + && !self.dashboard.running() + && self.profile_job.is_none(); + if !(now || idle) { + return; + } + if let (Some(a), Some(p)) = (&mut self.analysis, &self.analysis_path) + && !a.is_empty() + && let Err(e) = a.save_for(&self.source, p) + { + self.status = Some(format!("analysis cache not saved: {e}")); + } + self.analysis_dirty = false; + } + // -- selections -------------------------------------------------------- /// The selection that analyses (counts, timelines, exports) are @@ -969,6 +1127,13 @@ impl Document { fn spawn_freq(&self, column: usize, top: usize) -> Option { let index = self.index_snapshot()?; let base = self.scan_base(); + // whole-file counts of source columns come from the analysis cache + if base.is_none() + && column < self.header.len() + && let Some(cached) = self.analysis.as_ref().and_then(|a| a.count(column, top)) + { + return Some(FreqJob::cached(cached, self.source.len())); + } let shared = Arc::new(FrequencyShared::new(self.source.len())); let cancel = Arc::new(AtomicBool::new(false)); let enrichment = self.enrichment.clone(); @@ -1012,6 +1177,8 @@ impl Document { handle: Some(handle), column, base, + top, + cached: false, }) } @@ -1023,16 +1190,21 @@ impl Document { return; } let handle = job.handle.take().expect("handle present until joined"); - let column = job.column; + let (column, top, cached) = (job.column, job.top, job.cached); + let whole_file = job.base.is_none(); let base = job.base.as_ref().map(|n| n.count()); self.freq_job = None; match handle.join() { Ok(Ok(result)) if result.complete => { + if whole_file && !cached { + self.remember_count(top, &result); + } self.freq = Some(FreqView { column, rows: freq_rows(&result), result, base, + cached, }); } Ok(Ok(_)) => self.status = Some("count cancelled".into()), @@ -1060,6 +1232,7 @@ impl Document { self.dock_open = true; if let Some(c) = self.timestamp_column() && self.dashboard_ready() + && self.build.is_none() && self.timeline_column_in_use() != Some(c) && !self.source_changed && self.guard_source() @@ -1255,22 +1428,28 @@ impl Document { } fn poll_dashboard(&mut self) { + let mut remember: Vec<(usize, FrequencyResult)> = Vec::new(); for p in &mut self.dashboard.panels { let Some(job) = &mut p.job else { continue }; if !job.handle.as_ref().is_some_and(|h| h.is_finished()) { continue; } let handle = job.handle.take().expect("handle present until joined"); - let column = job.column; + let (column, top, cached) = (job.column, job.top, job.cached); + let whole_file = job.base.is_none(); let base = job.base.as_ref().map(|n| n.count()); p.job = None; match handle.join() { Ok(Ok(result)) if result.complete => { + if whole_file && !cached { + remember.push((top, result.clone())); + } p.view = Some(FreqView { column, rows: freq_rows(&result), result, base, + cached, }); } Ok(Ok(_)) => {} @@ -1278,6 +1457,9 @@ impl Document { Err(_) => p.error = Some("count thread panicked".into()), } } + for (top, result) in remember { + self.remember_count(top, &result); + } // a new selection step (or a change of view) makes the counts stale; // they are recomputed in the background, on screen or not, so the // tab shows current charts the moment it is opened @@ -1289,7 +1471,12 @@ impl Document { // the timeline card exists whenever there is a timestamp column, with // or without value panels let has_content = !self.dashboard.panels.is_empty() || self.timestamp_column().is_some(); - if self.dashboard.stale && self.dashboard_ready() && has_content && !self.source_changed { + if self.dashboard.stale + && self.dashboard_ready() + && has_content + && !self.source_changed + && self.build.is_none() + { self.dashboard.stale = false; self.refresh_dashboard(); } else if !self.source_changed { @@ -1316,6 +1503,36 @@ impl Document { return; }; let base = self.scan_base(); + // the whole-file timeline comes from the analysis cache: from memory + // when it was computed this session, else from the companion file, + // which the worker reads so that nothing large is parsed up front + let reference_year = TimelineOptions::default().reference_year; + if base.is_none() + && let Some(cached) = self + .analysis + .as_ref() + .and_then(|a| a.timeline(column, reference_year)) + .cloned() + { + self.timeline_job = Some(TimelineJob::cached(cached, self.source.len())); + if switch { + self.dock_tab = DockTab::Timeline; + self.dock_open = true; + } + return; + } + let cached_file = base + .is_none() + .then(|| self.analysis_path.clone()) + .flatten() + .map(|p| { + let sha = self + .index + .read() + .ok() + .and_then(|i| i.digests.sha256.map(|d| hex(&d))); + (p, self.source.id(), sha) + }); let shared = Arc::new(FrequencyShared::new(self.source.len())); let cancel = Arc::new(AtomicBool::new(false)); let (source, s2, c2, b2) = ( @@ -1345,7 +1562,18 @@ impl Document { cancel: Some(&c2), ..TimelineOptions::default() }; - timeline(&source, &index, selection, opts, &s2) + if let Some((p, id, sha)) = &cached_file + && let Some(t) = AnalysisCache::load_timeline( + p, + *id, + sha.as_deref(), + column, + opts.reference_year, + ) + { + return Ok((t, true)); + } + timeline(&source, &index, selection, opts, &s2).map(|t| (t, false)) }) .expect("spawn timeline thread"); self.timeline_job = Some(TimelineJob { @@ -1370,10 +1598,17 @@ impl Document { } let handle = job.handle.take().expect("handle present until joined"); let column = job.column; + let whole_file = job.base.is_none(); let base = job.base.as_ref().map(|n| n.count()); self.timeline_job = None; match handle.join() { - Ok(Ok(result)) if result.complete => { + Ok(Ok((result, from_cache))) if result.complete => { + if whole_file + && !from_cache + && let Some(a) = self.analysis_mut() + { + a.put_timeline(result.clone()); + } self.timeline = Some(TimelineView::new(column, result, base)) } Ok(Ok(_)) => self.status = Some("timeline cancelled".into()), @@ -1469,6 +1704,7 @@ impl Document { cancel.clone(), out.clone(), ); + let names = self.names.clone(); let handle = std::thread::Builder::new() .name("gridsift-export".into()) .spawn(move || { @@ -1514,13 +1750,15 @@ impl Document { )); } let out_path = std::path::absolute(&out2).unwrap_or_else(|_| out2.clone()); + let mut source_info = SourceInfo::from_source( + &source, + index.params.dialect, + index.digests, + Some(index.stats.records), + ); + source_info.dialect.names = names; let manifest = Manifest::new( - SourceInfo::from_source( - &source, - index.params.dialect, - index.digests, - Some(index.stats.records), - ), + source_info, operations, match &matches { Some(m) => SelectionInfo::Matches { records: m.len() }, @@ -1595,6 +1833,7 @@ impl Document { /// Stop every background job (on close). pub fn cancel_all(&mut self) { + self.save_analysis(true); if let Some(j) = &self.build { j.cancel.store(true, Ordering::Relaxed); } @@ -1686,15 +1925,21 @@ mod tests { } } + /// Open with the cache tree beside the fixture, never the user's. fn open(path: &Path) -> (egui::Context, Document) { let ctx = egui::Context::default(); - let mut d = Document::open(&ctx, path).expect("open fixture"); - // never publish a test index into the user's cache directory - d.sidecar = None; + let mut d = Document::open_with(&ctx, path, opts_for(path)).expect("open fixture"); settle(&mut d); (ctx, d) } + fn opts_for(path: &Path) -> OpenOptions { + OpenOptions { + cache_root: Some(path.parent().unwrap().join("cache")), + ..OpenOptions::default() + } + } + const HOSTS: &str = "ts,host,alternate_host\n\ 2026-09-21T14:13:20Z,alpha.example.com,alpha.example.org\n\ 2026-09-21T14:13:21Z,alpha.example.com,alpha.example.org\n\ @@ -1737,12 +1982,91 @@ mod tests { ); let (_, d) = open(&p); assert_eq!(d.timestamp_column(), Some(0)); - assert!(!d.dock_open); + assert_eq!(d.dock_tab, DockTab::Values); assert!(d.dashboard.auto_built); assert!(d.dashboard.panels.is_empty()); assert_eq!(d.timeline.as_ref().map(|t| t.result.parsed), Some(4)); } + /// Names given for a header-less file become the column names and the + /// first record stays data; forcing a header reads the file the other + /// way. + #[test] + fn names_and_header_override_the_sniffer() { + let p = fixture("bare.csv", "1,10.0.0.1\n2,10.0.0.2\n3,10.0.0.3\n"); + let ctx = egui::Context::default(); + let opts = OpenOptions { + header: None, + names: Some(vec!["n".into(), "ip".into()]), + ..opts_for(&p) + }; + let mut d = Document::open_with(&ctx, &p, opts).unwrap(); + settle(&mut d); + assert!(!d.params.dialect.has_header); + assert_eq!(d.header, ["n", "ip"]); + assert_eq!(d.index.read().unwrap().stats.records, 3); + assert_eq!( + d.names.as_deref(), + Some(&["n".to_string(), "ip".to_string()][..]) + ); + + let opts = OpenOptions { + header: Some(true), + names: None, + ..opts_for(&p) + }; + let mut d = Document::open_with(&ctx, &p, opts).unwrap(); + settle(&mut d); + assert!(d.params.dialect.has_header); + assert_eq!(d.header, ["1", "10.0.0.1"]); + assert_eq!(d.index.read().unwrap().stats.records, 2); + assert!(d.names.is_none()); + } + + /// Whole-file counts, the timeline and the profile are kept in the cache + /// tree: the next open of the same bytes shows them without a scan or a + /// profile pass, selections are still counted, and a changed file gets + /// nothing from the cache. + #[test] + fn whole_file_analyses_are_cached_between_opens() { + let p = fixture("hosts.csv", HOSTS); + let ctx = egui::Context::default(); + let mut d = Document::open_with(&ctx, &p, opts_for(&p)).unwrap(); + settle(&mut d); + assert!(d.dashboard.auto_built); + assert!(!d.dashboard.panels[0].view.as_ref().unwrap().cached); + let path = d.analysis_path.clone().unwrap(); + assert!(path.is_file(), "saved once the scans were idle"); + drop(d); + + let mut d = Document::open_with(&ctx, &p, opts_for(&p)).unwrap(); + assert!( + d.profile_job.is_none(), + "the file-wide profile came from the cache" + ); + assert!(d.dashboard.auto_built, "built from it at once"); + settle(&mut d); + for panel in &d.dashboard.panels { + assert!(panel.view.as_ref().unwrap().cached); + } + assert_eq!(d.timeline.as_ref().map(|t| t.result.parsed), Some(4)); + select_host(&ctx, &mut d, "alpha.example.com"); + let v = d.dashboard.panels[0].view.as_ref().unwrap(); + assert!(!v.cached && v.base == Some(2)); + drop(d); + + std::fs::write( + &p, + format!("{HOSTS}2026-09-21T14:13:24Z,gamma.example.com,gamma.example.org\n"), + ) + .unwrap(); + let mut d = Document::open_with(&ctx, &p, opts_for(&p)).unwrap(); + assert!(d.analysis.is_none(), "other bytes, nothing from the cache"); + settle(&mut d); + assert!(!d.dashboard.panels[0].view.as_ref().unwrap().cached); + assert_eq!(d.timeline.as_ref().map(|t| t.result.parsed), Some(5)); + } + fn select_host(ctx: &egui::Context, d: &mut Document, host: &str) { d.search_ui.column = Some(1); d.search_ui.pattern = host.into(); @@ -1759,8 +2083,7 @@ mod tests { #[test] fn dashboard_is_built_and_recounted_in_the_background() { let (ctx, mut d) = open(&fixture("hosts.csv", HOSTS)); - assert_eq!(d.dock_tab, DockTab::Values); - assert!(!d.dock_open); + assert!(d.dock_open && d.dock_tab == DockTab::Values); assert!(d.dashboard.auto_built); let columns: Vec = d.dashboard.panels.iter().map(|p| p.column).collect(); assert_eq!(columns, [1, 2]); @@ -1772,7 +2095,7 @@ mod tests { assert_eq!(d.timeline.as_ref().map(|t| t.result.parsed), Some(4)); select_host(&ctx, &mut d, "alpha.example.com"); - assert!(!d.dock_open); + assert_eq!(d.dock_tab, DockTab::Values); let v = d.dashboard.panels[0].view.as_ref().unwrap(); assert_eq!(v.base, Some(2)); assert_eq!(v.rows, [("alpha.example.com".to_string(), 2, 1.0)]); diff --git a/crates/gridsift-desktop/src/jobs.rs b/crates/gridsift/src/bin/gridsift-desktop/jobs.rs similarity index 92% rename from crates/gridsift-desktop/src/jobs.rs rename to crates/gridsift/src/bin/gridsift-desktop/jobs.rs index adcd666..3723445 100644 --- a/crates/gridsift-desktop/src/jobs.rs +++ b/crates/gridsift/src/bin/gridsift-desktop/jobs.rs @@ -368,6 +368,26 @@ pub struct FreqJob { pub column: usize, /// The selection the count is restricted to, if any. pub base: Option>, + /// Top values asked for (what the analysis cache keeps). + pub top: usize, + /// Served from the analysis cache rather than counted. + pub cached: bool, +} + +impl FreqJob { + /// A whole-file count from the analysis cache: finished before it starts. + pub fn cached(result: FrequencyResult, source_len: u64) -> FreqJob { + let (column, top) = (result.column, result.top.len()); + FreqJob { + shared: Arc::new(FrequencyShared::new(source_len)), + cancel: Arc::new(AtomicBool::new(false)), + handle: Some(std::thread::spawn(move || Ok(result))), + column, + base: None, + top, + cached: true, + } + } } pub struct FreqView { @@ -377,6 +397,8 @@ pub struct FreqView { pub rows: Vec<(String, u64, f32)>, /// Records in the selection the count was restricted to. pub base: Option, + /// Counted in an earlier session over the same bytes. + pub cached: bool, } // --------------------------------------------------------------------------- @@ -455,12 +477,27 @@ impl Dashboard { pub struct TimelineJob { pub shared: Arc, pub cancel: Arc, - pub handle: Option>>, + /// The result and whether it came from the analysis cache. + pub handle: Option>>, pub column: usize, /// The selection the timeline is restricted to, if any. pub base: Option>, } +impl TimelineJob { + /// A whole-file timeline already in memory: finished before it starts. + pub fn cached(result: TimelineResult, source_len: u64) -> TimelineJob { + let column = result.column; + TimelineJob { + shared: Arc::new(FrequencyShared::new(source_len)), + cancel: Arc::new(AtomicBool::new(false)), + handle: Some(std::thread::spawn(move || Ok((result, true)))), + column, + base: None, + } + } +} + pub struct TimelineView { pub column: usize, pub result: TimelineResult, diff --git a/crates/gridsift-desktop/src/main.rs b/crates/gridsift/src/bin/gridsift-desktop/main.rs similarity index 77% rename from crates/gridsift-desktop/src/main.rs rename to crates/gridsift/src/bin/gridsift-desktop/main.rs index d3f306a..6daa473 100644 --- a/crates/gridsift-desktop/src/main.rs +++ b/crates/gridsift/src/bin/gridsift-desktop/main.rs @@ -25,13 +25,18 @@ use std::time::Duration; use eframe::egui::{self, Key, Modifiers}; use gridsift_core::enrich::{EnrichRule, Provider}; -use crate::document::{Document, RuleChoice}; +use crate::document::{Document, OpenOptions, RuleChoice}; use crate::ui::Action; /// `gridsift-desktop [FILE] [--search PATTERN] [--regex] [--count COLUMN] -/// [--domain COLUMN] [--timeline]` +/// [--domain COLUMN] [--timeline] [--dashboard] [--header | --no-header] +/// [--names a,b,c]` struct Launch { file: Option, + /// Read the first record as a header / as data; `None` asks the sniffer. + header: Option, + /// Column names for a file without a header. + names: Option>, search: Option, regex: bool, count: Option, @@ -46,6 +51,8 @@ struct Launch { fn parse_args() -> Launch { let mut l = Launch { file: None, + header: None, + names: None, search: None, regex: false, count: None, @@ -67,6 +74,13 @@ fn parse_args() -> Launch { .extend(args.next().and_then(|c| c.parse::().ok())), "--timeline" => l.timeline = true, "--dashboard" => l.dashboard = true, + "--header" => l.header = Some(true), + "--no-header" => l.header = Some(false), + "--names" => { + l.names = args + .next() + .map(|s| s.split(',').map(|n| n.trim().to_string()).collect()) + } "--export-dialog" => l.export_dialog = true, "--enrich-dialog" => l.enrich_dialog = true, _ if l.file.is_none() => l.file = Some(PathBuf::from(a)), @@ -96,7 +110,12 @@ fn main() -> Result<(), eframe::Error> { cc.egui_ctx.set_theme(egui::Theme::Dark); let mut app = App::default(); if let Some(p) = &launch.file { - app.open(&cc.egui_ctx, p); + let opts = OpenOptions { + header: launch.header, + names: launch.names.clone(), + cache_root: None, + }; + app.open_with(&cc.egui_ctx, p, opts); if let Some(d) = &mut app.doc { if !launch.domain.is_empty() { for &c in &launch.domain { @@ -145,8 +164,12 @@ struct App { impl App { fn open(&mut self, ctx: &egui::Context, path: &Path) { + self.open_with(ctx, path, OpenOptions::default()); + } + + fn open_with(&mut self, ctx: &egui::Context, path: &Path, opts: OpenOptions) { self.close(); - match Document::open(ctx, path) { + match Document::open_with(ctx, path, opts) { Ok(d) => { self.doc = Some(d); self.error = None; @@ -178,6 +201,40 @@ impl App { match a { Action::OpenDialog => self.open_dialog(ctx), Action::CloseDocument => self.close(), + // the index is keyed by the header setting, so a different + // reading of the first record means reopening the file + Action::ToggleHeader => { + if let Some(d) = &self.doc { + let path = d.path.clone(); + let opts = OpenOptions { + header: Some(!d.params.dialect.has_header), + names: d.names.clone(), + cache_root: None, + }; + self.open_with(ctx, &path, opts); + } + } + Action::NameColumns => { + if let Some(d) = &mut self.doc { + d.names_ui.text = d + .names + .clone() + .unwrap_or_else(|| d.header.clone()) + .join(","); + d.names_ui.open = true; + } + } + Action::SetNames(names) => { + if let Some(d) = &self.doc { + let path = d.path.clone(); + let opts = OpenOptions { + header: Some(false), + names: Some(names), + cache_root: None, + }; + self.open_with(ctx, &path, opts); + } + } other => { if let Some(d) = &mut self.doc { apply_to_document(ctx, d, other); @@ -190,7 +247,11 @@ impl App { fn apply_to_document(ctx: &egui::Context, d: &mut Document, action: Action) { match action { - Action::OpenDialog | Action::CloseDocument => {} + Action::OpenDialog + | Action::CloseDocument + | Action::ToggleHeader + | Action::NameColumns + | Action::SetNames(_) => {} Action::Find => { if !d.search_ui.pattern.is_empty() { d.start_search(ctx); @@ -287,6 +348,9 @@ impl eframe::App for App { .show(ctx, |ui| ui::grid(ui, d)); dialogs::export_window(ctx, d); dialogs::enrich_window(ctx, d); + if let Some(names) = dialogs::names_window(ctx, d) { + actions.push(Action::SetNames(names)); + } } None => { egui::CentralPanel::default().show(ctx, ui::empty_state); diff --git a/crates/gridsift-desktop/src/theme.rs b/crates/gridsift/src/bin/gridsift-desktop/theme.rs similarity index 100% rename from crates/gridsift-desktop/src/theme.rs rename to crates/gridsift/src/bin/gridsift-desktop/theme.rs diff --git a/crates/gridsift-desktop/src/ui.rs b/crates/gridsift/src/bin/gridsift-desktop/ui.rs similarity index 94% rename from crates/gridsift-desktop/src/ui.rs rename to crates/gridsift/src/bin/gridsift-desktop/ui.rs index 89ee5e6..8d26e16 100644 --- a/crates/gridsift-desktop/src/ui.rs +++ b/crates/gridsift/src/bin/gridsift-desktop/ui.rs @@ -42,6 +42,12 @@ pub enum Action { /// Filter to a time range of `column`. FilterRange(usize, i64, i64), GoTo(u64), + /// Reopen the file reading the first record the other way (header ↔ data). + ToggleHeader, + /// Open the column-naming dialog (files without a header). + NameColumns, + /// Reopen the file with these column names and no header. + SetNames(Vec), /// Show the dashboard (built from the profile the first time). Dashboard, /// Rebuild the dashboard from the column profile. @@ -118,7 +124,7 @@ pub fn sidebar(ctx: &egui::Context, d: &mut Document) -> Vec { .id_salt("sidebar-scroll") .auto_shrink([false, false]) .show(ui, |ui| { - evidence_section(ui, d); + evidence_section(ui, d, &mut actions); columns_section(ui, d, &mut actions); enrichment_section(ui, d, &mut actions); ui.add_space(12.0); @@ -138,7 +144,7 @@ pub fn sidebar(ctx: &egui::Context, d: &mut Document) -> Vec { actions } -fn evidence_section(ui: &mut egui::Ui, d: &Document) { +fn evidence_section(ui: &mut egui::Ui, d: &Document, actions: &mut Vec) { let (records, complete, mismatches, lenient, unterminated, sha) = { let idx = d.index.read().expect("index lock"); ( @@ -207,17 +213,39 @@ fn evidence_section(ui: &mut egui::Ui, d: &Document) { }; theme::fact(ui, "Index", index_text); let dl = d.params.dialect; - theme::fact( - ui, - "Dialect", - RichText::new(format!( - "{:?} {} {}", - dl.delimiter as char, - dl.quote - .map_or("no-quote".to_string(), |q| format!("{:?}", q as char)), - if dl.has_header { "header" } else { "no header" } - )), - ); + ui.label(RichText::new("Dialect").color(DIM)); + ui.horizontal(|ui| { + ui.label( + RichText::new(format!( + "{:?} {}", + dl.delimiter as char, + dl.quote + .map_or("no-quote".to_string(), |q| format!("{:?}", q as char)), + )) + .monospace(), + ); + // the sniffer's reading of the first record, flippable: the + // file is reopened (its index is keyed by this setting) + let (label, hint) = if dl.has_header { + ( + "header", + "the first record is the header — click to reopen treating it as data", + ) + } else { + ( + "no header", + "the first record is data — click to reopen treating it as the header", + ) + }; + if ui + .small_button(RichText::new(label).monospace()) + .on_hover_text(hint) + .clicked() + { + actions.push(Action::ToggleHeader); + } + }); + ui.end_row(); let cols = if d.derived_names.is_empty() { format!("{}", d.header.len()) } else { @@ -256,6 +284,20 @@ fn evidence_section(ui: &mut egui::Ui, d: &Document) { fn columns_section(ui: &mut egui::Ui, d: &Document, actions: &mut Vec) { theme::section(ui, "COLUMNS"); + if !d.params.dialect.has_header { + ui.horizontal(|ui| { + if ui + .small_button("name columns…") + .on_hover_text("names for a file without a header; exports record them") + .clicked() + { + actions.push(Action::NameColumns); + } + if d.names.is_some() { + ui.label(RichText::new("named").color(GREEN).size(11.0)); + } + }); + } let n_source = d.header.len(); for i in 0..d.column_count() { let name = d.column_name(i); @@ -355,7 +397,7 @@ pub fn command_bar(ctx: &egui::Context, d: &mut Document) -> Vec { ui.label(RichText::new("🔍").color(DIM)); let resp = ui.add( egui::TextEdit::singleline(&mut d.search_ui.pattern) - .desired_width(420.0) + .desired_width(360.0) .font(egui::TextStyle::Monospace) .hint_text(format!("literal text, or a regex ({}F)", theme::CMD)), ); @@ -391,6 +433,9 @@ pub fn command_bar(ctx: &egui::Context, d: &mut Document) -> Vec { if ui.button("Find").clicked() { actions.push(Action::Find); } + ui.add_space(8.0); + // the analysis dock is open from the start; this hides and shows it + theme::toggle_chip(ui, &mut d.dock_open, "Analysis"); if let Some(e) = &d.search_ui.error { ui.colored_label(RED, format!("invalid pattern: {e}")); } @@ -398,8 +443,8 @@ pub fn command_bar(ctx: &egui::Context, d: &mut Document) -> Vec { ui.label(RichText::new("Go to row").color(DIM)); let r = ui.add( egui::TextEdit::singleline(&mut d.goto) - .desired_width(90.0) - .hint_text("0-based"), + .desired_width(80.0) + .hint_text("row"), ); if r.lost_focus() && ui.input(|i| i.key_pressed(Key::Enter)) @@ -929,6 +974,10 @@ fn values_tab(ui: &mut egui::Ui, d: &mut Document, actions: &mut Vec) { .color(DIM), ); scope_label(ui, v.base); + if v.cached { + ui.label(RichText::new("cached").color(GREEN).size(11.0)) + .on_hover_text("counted in an earlier session over the same bytes"); + } if !r.exact { ui.label( RichText::new(format!( diff --git a/crates/gridsift-cli/src/main.rs b/crates/gridsift/src/main.rs similarity index 91% rename from crates/gridsift-cli/src/main.rs rename to crates/gridsift/src/main.rs index facf01c..dbc2878 100644 --- a/crates/gridsift-cli/src/main.rs +++ b/crates/gridsift/src/main.rs @@ -14,6 +14,7 @@ use std::time::{Duration, Instant}; use anyhow::{Context, Result, bail}; use clap::{Args, Parser, Subcommand, ValueEnum}; +use gridsift_core::analysis::AnalysisCache; use gridsift_core::dialect::sniff; use gridsift_core::enrich::{EnrichRule, Enrichment, GeoIpDb, GeoProvider, LookupTable, Provider}; use gridsift_core::export::{ExportOptions, Selection, Terminator, export_pending}; @@ -347,6 +348,9 @@ struct DialectArgs { /// Force the first record to be treated as a header #[arg(long)] header: bool, + /// Column names for a file without a header, comma-separated (implies `--no-header`) + #[arg(long, value_name = "NAMES", value_delimiter = ',')] + names: Vec, } /// Offline enrichment sources, shared by the commands that can derive columns. @@ -740,6 +744,9 @@ fn resolve_dialect(src: &Source, args: &DialectArgs) -> Result<(Sniff, Dialect)> if args.header { d.has_header = true; } + if !args.names.is_empty() { + d.has_header = false; + } Ok((sn, d)) } @@ -850,7 +857,7 @@ fn cmd_info(file: &Path, rows: usize, args: &DialectArgs, json: bool) -> Result< ..IndexParams::default() }; let idx = bootstrap(&src, params); - let header = header_fields(&src, &idx); + let header = column_names(&src, &idx, args); let recs = locate_records(&src, &idx, 0, rows); let elapsed = t0.elapsed(); @@ -1065,7 +1072,7 @@ fn cmd_rows( "records": records, })); } - if let Some(h) = header_fields(&src, &idx) { + if let Some(h) = column_names(&src, &idx, args) { println!("{:>10} {}", "#", render_fields(&h, 24)); } for r in &recs { @@ -1307,6 +1314,24 @@ fn load_index( } /// Resolve `--column` values (header names or 0-based indexes). +/// Column names: the header record, or the `--names` given for a file +/// without one. +fn column_names<'s>( + src: &'s Source, + idx: &SparseIndex, + args: &DialectArgs, +) -> Option>> { + if !args.names.is_empty() { + return Some( + args.names + .iter() + .map(|n| Cow::Owned(n.as_bytes().to_vec())) + .collect(), + ); + } + header_fields(src, idx) +} + fn resolve_columns( specs: &[String], header: Option<&[Cow<'_, [u8]>]>, @@ -1349,7 +1374,7 @@ fn cmd_search( "note: no index for this file; searching sequentially (run `gridsift index` first for a parallel search)" ); } - let header = header_fields(&src, &idx); + let header = column_names(&src, &idx, args); let columns = resolve_columns(&o.columns, header.as_deref())?; let query = SearchQuery { pattern: pattern.to_string(), @@ -1594,7 +1619,7 @@ fn cmd_export(file: &Path, args: &DialectArgs, o: ExportArgs, json: bool) -> Res ..IndexParams::default() }; let idx = ensure_full_index(file, &src, params, o.index_path.clone(), json)?; - let header = header_fields(&src, &idx); + let header = column_names(&src, &idx, args); // optional search → selection let mut operations = Vec::new(); @@ -1705,13 +1730,15 @@ fn cmd_export(file: &Path, args: &DialectArgs, o: ExportArgs, json: bool) -> Res // manifest next to the output, committed before the output itself let out_path = std::path::absolute(&o.output).unwrap_or_else(|_| o.output.clone()); + let mut source_info = SourceInfo::from_source( + &src, + idx.params.dialect, + idx.digests, + Some(idx.stats.records), + ); + source_info.dialect.names = (!args.names.is_empty()).then(|| args.names.clone()); let manifest = Manifest::new( - SourceInfo::from_source( - &src, - idx.params.dialect, - idx.digests, - Some(idx.stats.records), - ), + source_info, operations, selection_info, OutputInfo { @@ -1978,14 +2005,49 @@ fn cmd_profile( ..IndexParams::default() }; let (idx, from_index) = load_index(file, &src, params, index_path)?; - let header: Vec = match header_fields(&src, &idx) { + let header: Vec = match column_names(&src, &idx, args) { Some(h) => h.iter().map(|f| field_str(f).into_owned()).collect(), None => (0..idx.stats.expected_fields) .map(|i| format!("col{i}")) .collect(), }; let t0 = Instant::now(); - let prof = profile_columns(&src, &idx, &header, ProfileOptions::default()); + // the file-wide profile is kept between runs (by column index: names + // given since are applied) + let analysis_path = AnalysisCache::default_path(file, idx.params.dialect).ok(); + let sha = idx.digests.sha256.map(|d| hex(&d)); + let mut analysis = analysis_path + .as_ref() + .and_then(|p| AnalysisCache::load_for(p, src.id(), sha.as_deref())); + let cached = analysis + .as_ref() + .and_then(|a| a.profile.clone()) + .filter(|p| p.spans_file && p.columns.len() == header.len()) + .map(|mut p| { + for (c, name) in p.columns.iter_mut().zip(&header) { + c.name = name.clone(); + } + p + }); + let from_cache = cached.is_some(); + let prof = match cached { + Some(p) => p, + None => { + let p = profile_columns(&src, &idx, &header, ProfileOptions::default()); + if let Some(path) = &analysis_path + && p.spans_file + { + let a = analysis.get_or_insert_with(|| AnalysisCache::new(src.id(), sha.clone())); + a.put_profile(p.clone()); + if let Err(e) = a.save_for(&src, path) + && !json + { + eprintln!("note: analysis cache not saved: {e}"); + } + } + p + } + }; let elapsed = t0.elapsed(); if json { @@ -1994,6 +2056,7 @@ fn cmd_profile( "sampled_records": prof.sampled_records, "spans_file": prof.spans_file, "from_index": from_index, + "cached": from_cache, "columns": prof.columns, "elapsed_ms": elapsed.as_secs_f64() * 1000.0, })); @@ -2115,7 +2178,7 @@ fn cmd_freq(file: &Path, args: &DialectArgs, o: FreqArgs, json: bool) -> Result< "note: no index for this file; counting sequentially (run `gridsift index` first for a parallel count)" ); } - let header = header_fields(&src, &idx); + let header = column_names(&src, &idx, args); let enrichment = build_enrichment(&o.enrich, idx.params.dialect, header.as_deref())?; let names = all_column_names(header.as_deref(), enrichment.as_ref()); let column = resolve_columns(std::slice::from_ref(&o.column), Some(&names))? @@ -2125,6 +2188,20 @@ fn cmd_freq(file: &Path, args: &DialectArgs, o: FreqArgs, json: bool) -> Result< .get(column) .map(|f| field_str(f).into_owned()) .unwrap_or_else(|| format!("col{column}")); + // whole-file counts of source columns are kept between runs + let cacheable = o.search.is_none() + && (enrichment.is_none() || header.as_ref().is_some_and(|h| column < h.len())); + let analysis_path = if cacheable { + AnalysisCache::default_path(file, idx.params.dialect).ok() + } else { + None + }; + let sha = idx.digests.sha256.map(|d| hex(&d)); + let mut analysis = analysis_path + .as_ref() + .and_then(|p| AnalysisCache::load_for(p, src.id(), sha.as_deref())); + let cached = analysis.as_ref().and_then(|a| a.count(column, o.top)); + let from_cache = cached.is_some(); let matches = match &o.search { Some(pattern) => Some(search_matches( @@ -2148,27 +2225,42 @@ fn cmd_freq(file: &Path, args: &DialectArgs, o: FreqArgs, json: bool) -> Result< }; let shared = FrequencyShared::new(src.len()); - let opts = FrequencyOptions { - column, - top: o.top, - threads: o.threads, - enrichment: enrichment.as_ref(), - ..FrequencyOptions::default() - }; - let pb = progress_bar(src.len(), json); - let result = std::thread::scope(|s| { - let h = s.spawn(|| frequency(&src, &idx, selection, opts, &shared)); - while !h.is_finished() { - pb.set_position(shared.bytes.load(std::sync::atomic::Ordering::Relaxed)); - std::thread::sleep(Duration::from_millis(50)); + let result = match cached { + Some(r) => r, + None => { + let opts = FrequencyOptions { + column, + top: o.top, + threads: o.threads, + enrichment: enrichment.as_ref(), + ..FrequencyOptions::default() + }; + let pb = progress_bar(src.len(), json); + let result = std::thread::scope(|s| { + let h = s.spawn(|| frequency(&src, &idx, selection, opts, &shared)); + while !h.is_finished() { + pb.set_position(shared.bytes.load(std::sync::atomic::Ordering::Relaxed)); + std::thread::sleep(Duration::from_millis(50)); + } + h.join().expect("frequency thread") + }) + .context("counting")?; + pb.finish_and_clear(); + if !result.complete { + bail!("count did not complete"); + } + if let Some(p) = &analysis_path { + let a = analysis.get_or_insert_with(|| AnalysisCache::new(src.id(), sha.clone())); + a.put_count(o.top, result.clone()); + if let Err(e) = a.save_for(&src, p) + && !json + { + eprintln!("note: analysis cache not saved: {e}"); + } + } + result } - h.join().expect("frequency thread") - }) - .context("counting")?; - pb.finish_and_clear(); - if !result.complete { - bail!("count did not complete"); - } + }; let rss = peak_rss_bytes(); let share = |n: u64| { if result.counted == 0 { @@ -2184,6 +2276,7 @@ fn cmd_freq(file: &Path, args: &DialectArgs, o: FreqArgs, json: bool) -> Result< "column": column, "column_name": column_name, "search": o.search, + "cached": from_cache, "counted": result.counted, "empty": result.empty, "distinct": result.distinct, @@ -2225,8 +2318,21 @@ fn cmd_freq(file: &Path, args: &DialectArgs, o: FreqArgs, json: bool) -> Result< share(e.count) ); } + let timing = if from_cache { + "from the analysis cache".to_string() + } else { + format!( + "{:.2} s, {:.0} MiB/s, {} thread(s)", + result.elapsed.as_secs_f64(), + mib_per_s( + shared.bytes.load(std::sync::atomic::Ordering::Relaxed), + result.elapsed + ), + result.threads + ) + }; eprintln!( - "({} records, {} empty, {} distinct{}; {:.2} s, {:.0} MiB/s, {} thread(s), peak RSS {})", + "({} records, {} empty, {} distinct{}; {timing}, peak RSS {})", group_thousands(result.counted), group_thousands(result.empty), group_thousands(result.distinct), @@ -2238,12 +2344,6 @@ fn cmd_freq(file: &Path, args: &DialectArgs, o: FreqArgs, json: bool) -> Result< group_thousands(result.error_bound) ) }, - result.elapsed.as_secs_f64(), - mib_per_s( - shared.bytes.load(std::sync::atomic::Ordering::Relaxed), - result.elapsed - ), - result.threads, rss_text(rss) ); Ok(()) @@ -2300,7 +2400,7 @@ fn cmd_timeline(file: &Path, args: &DialectArgs, o: TimelineArgs, json: bool) -> "note: no index for this file; scanning sequentially (run `gridsift index` first for a parallel scan)" ); } - let header = header_fields(&src, &idx); + let header = column_names(&src, &idx, args); let column = resolve_columns(std::slice::from_ref(&o.column), header.as_deref())? .and_then(|c| c.first().copied()) .expect("one column"); @@ -2310,6 +2410,21 @@ fn cmd_timeline(file: &Path, args: &DialectArgs, o: TimelineArgs, json: bool) -> .map(|f| field_str(f).into_owned()) .unwrap_or_else(|| format!("col{column}")); let width_spec = parse_bucket(&o.bucket)?; + let reference_year = o.year.unwrap_or_else(current_year); + // the whole-file timeline is kept between runs + let analysis_path = if o.search.is_none() { + AnalysisCache::default_path(file, idx.params.dialect).ok() + } else { + None + }; + let sha = idx.digests.sha256.map(|d| hex(&d)); + let mut analysis = analysis_path + .as_ref() + .and_then(|p| AnalysisCache::load_for(p, src.id(), sha.as_deref())); + let cached = analysis_path.as_ref().and_then(|p| { + AnalysisCache::load_timeline(p, src.id(), sha.as_deref(), column, reference_year) + }); + let from_cache = cached.is_some(); let matches = match &o.search { Some(pattern) => Some(search_matches( @@ -2333,26 +2448,41 @@ fn cmd_timeline(file: &Path, args: &DialectArgs, o: TimelineArgs, json: bool) -> }; let shared = FrequencyShared::new(src.len()); - let opts = TimelineOptions { - column, - threads: o.threads, - reference_year: o.year.unwrap_or_else(current_year), - ..TimelineOptions::default() - }; - let pb = progress_bar(src.len(), json); - let t = std::thread::scope(|s| { - let h = s.spawn(|| timeline(&src, &idx, selection, opts, &shared)); - while !h.is_finished() { - pb.set_position(shared.bytes.load(std::sync::atomic::Ordering::Relaxed)); - std::thread::sleep(Duration::from_millis(50)); + let t = match cached { + Some(t) => t, + None => { + let opts = TimelineOptions { + column, + threads: o.threads, + reference_year, + ..TimelineOptions::default() + }; + let pb = progress_bar(src.len(), json); + let t = std::thread::scope(|s| { + let h = s.spawn(|| timeline(&src, &idx, selection, opts, &shared)); + while !h.is_finished() { + pb.set_position(shared.bytes.load(std::sync::atomic::Ordering::Relaxed)); + std::thread::sleep(Duration::from_millis(50)); + } + h.join().expect("timeline thread") + }) + .context("timeline")?; + pb.finish_and_clear(); + if !t.complete { + bail!("timeline did not complete"); + } + if let Some(p) = &analysis_path { + let a = analysis.get_or_insert_with(|| AnalysisCache::new(src.id(), sha.clone())); + a.put_timeline(t.clone()); + if let Err(e) = a.save_for(&src, p) + && !json + { + eprintln!("note: analysis cache not saved: {e}"); + } + } + t } - h.join().expect("timeline thread") - }) - .context("timeline")?; - pb.finish_and_clear(); - if !t.complete { - bail!("timeline did not complete"); - } + }; let width = width_spec.unwrap_or_else(|| t.auto_width(o.max_buckets)); let buckets = t.rebucket(width); let fmt = |secs: i64| { @@ -2369,6 +2499,7 @@ fn cmd_timeline(file: &Path, args: &DialectArgs, o: TimelineArgs, json: bool) -> "column": column, "column_name": column_name, "search": o.search, + "cached": from_cache, "counted": t.counted, "parsed": t.parsed, "unparsed": t.unparsed, @@ -2394,11 +2525,14 @@ fn cmd_timeline(file: &Path, args: &DialectArgs, o: TimelineArgs, json: bool) -> let bar = "█".repeat((count * 40 / max) as usize); println!("{} {:>12} {bar}", fmt(*start), group_thousands(*count)); } + let timing = if from_cache { + "from the analysis cache".to_string() + } else { + format!("{:.2} s, {} thread(s)", t.elapsed.as_secs_f64(), t.threads) + }; eprintln!( - "({} records, {:.2} s, {} thread(s){})", + "({} records, {timing}{})", group_thousands(t.counted), - t.elapsed.as_secs_f64(), - t.threads, if t.resolution > 1 { format!( "; base resolution coarsened to {}", diff --git a/crates/gridsift-cli/tests/cli.rs b/crates/gridsift/tests/cli.rs similarity index 74% rename from crates/gridsift-cli/tests/cli.rs rename to crates/gridsift/tests/cli.rs index 305755c..ad83bea 100644 --- a/crates/gridsift-cli/tests/cli.rs +++ b/crates/gridsift/tests/cli.rs @@ -189,6 +189,89 @@ fn exact_search_selects_whole_fields() { ); } +#[test] +fn whole_file_counts_are_served_from_the_analysis_cache_until_the_file_changes() { + let dir = workdir(); + let csv = dir.join("c.csv"); + fs::write(&csv, "ip,host\n10.0.0.1,a\n10.0.0.2,b\n10.0.0.1,c\n").unwrap(); + ok_json(&dir, &["index", s(&csv), "--json"]); + let first = ok_json(&dir, &["freq", s(&csv), "-c", "ip", "--json"]); + assert_eq!(first["cached"], false); + assert_eq!(first["top"][0]["count"], 2); + let again = ok_json(&dir, &["freq", s(&csv), "-c", "ip", "--json"]); + assert_eq!(again["cached"], true); + assert_eq!(again["top"], first["top"]); + // a narrower request is served, a wider one is counted again + assert_eq!( + ok_json(&dir, &["freq", s(&csv), "-c", "ip", "-n", "1", "--json"])["cached"], + true + ); + // a selection is never served from the cache + let sel = ok_json(&dir, &["freq", s(&csv), "-c", "ip", "-s", "a", "--json"]); + assert_eq!(sel["cached"], false); + assert_eq!(sel["counted"], 1); + // the profile too, and the timeline + assert_eq!( + ok_json(&dir, &["profile", s(&csv), "--json"])["cached"], + false + ); + assert_eq!( + ok_json(&dir, &["profile", s(&csv), "--json"])["cached"], + true + ); + // a changed file gets nothing from the cache + fs::write(&csv, "ip,host\n10.0.0.1,a\n10.0.0.2,b\n10.0.0.2,c\n").unwrap(); + let changed = ok_json(&dir, &["freq", s(&csv), "-c", "ip", "--json"]); + assert_eq!(changed["cached"], false); + assert_eq!(changed["top"][0]["value"], "10.0.0.2"); +} + +#[test] +fn names_label_a_headerless_file_and_reach_the_manifest() { + let dir = workdir(); + let csv = dir.join("bare.csv"); + fs::write(&csv, "1,10.0.0.1\n2,10.0.0.2\n3,10.0.0.1\n").unwrap(); + // the names resolve `-c`, and the first record is counted as data + let v = ok_json( + &dir, + &["freq", s(&csv), "-c", "ip", "--names", "n,ip", "--json"], + ); + assert_eq!(v["top"][0]["value"], "10.0.0.1"); + assert_eq!(v["top"][0]["count"], 2); + // a file the sniffer would read with a header is read as data with --names + let out = dir.join("named.csv"); + let v = ok_json( + &dir, + &[ + "export", + s(&csv), + "-o", + s(&out), + "--names", + "n,ip", + "-s", + "10.0.0.1", + "-c", + "ip", + "--json", + ], + ); + assert_eq!(v["records"], 2); + let m: serde_json::Value = + serde_json::from_slice(&fs::read(dir.join("named.csv.manifest.json")).unwrap()).unwrap(); + assert_eq!(m["source"]["dialect"]["header"], false); + assert_eq!( + m["source"]["dialect"]["names"], + serde_json::json!(["n", "ip"]) + ); + // without names the sniffer's reading stands and the manifest says nothing about names + let out2 = dir.join("plain.csv"); + ok_json(&dir, &["export", s(&csv), "-o", s(&out2), "--json"]); + let m2: serde_json::Value = + serde_json::from_slice(&fs::read(dir.join("plain.csv.manifest.json")).unwrap()).unwrap(); + assert!(m2["source"]["dialect"].get("names").is_none()); +} + #[test] fn export_publishes_output_with_manifest_and_verify_states_its_scope() { let dir = workdir(); diff --git a/docs/design/README.md b/docs/design/README.md index 8d247ab..92b332a 100644 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -1,6 +1,6 @@ # Desktop UI design — v1 -Status: implemented in `crates/gridsift-desktop` (2026-09-28); `mock-v1.html` +Status: implemented in `crates/gridsift/src/bin/gridsift-desktop` (2026-09-28); `mock-v1.html` next to this file is the static mock the implementation was checked against. Deviations from the proposal are listed at the end. diff --git a/docs/installation.md b/docs/installation.md index a42b0cb..e8602f5 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -6,14 +6,14 @@ graph needs — the Linux file-dialog backend). ## From crates.io ``` -cargo install gridsift --locked # the command-line tool -cargo install gridsift-desktop --locked # the desktop application (egui) +cargo install gridsift --locked # gridsift-desktop (the application) and gridsift (the command line) ``` `--locked` builds with the dependency versions the release was tested -with. Both binaries go into `~/.cargo/bin`. The engine is published separately as -[`gridsift-core`](https://crates.io/crates/gridsift-core) for use as a -library. Releases are listed in [CHANGELOG.md](../CHANGELOG.md). +with. Both binaries go into `~/.cargo/bin`. The engine is published +separately as [`gridsift-core`](https://crates.io/crates/gridsift-core) +for use as a library. Releases are listed in +[CHANGELOG.md](../CHANGELOG.md). ## Build from source @@ -23,6 +23,9 @@ cd GridSift cargo build --release ``` +`cargo run` starts the desktop application (`cargo run -- file.csv` opens +a file); the command line is `cargo run --bin gridsift -- …`. + This produces two binaries: | binary | what it is | @@ -66,16 +69,18 @@ console window. ## Where gridsift writes gridsift never writes next to the evidence. The only files it creates on -its own are index sidecars (`*.gsix`) in the user cache directory: +its own are index sidecars (`*.gsix`) and analysis caches (`*.gsan` with +the column profile and whole-file value counts, `*.gstl` with the +timelines) in the user cache directory: -| platform | index cache | +| platform | cache directory | |---|---| -| Linux | `$XDG_CACHE_HOME/gridsift/index/` (default `~/.cache/gridsift/index/`) | -| macOS | `~/Library/Caches/gridsift/index/` | -| Windows | `%LOCALAPPDATA%\gridsift\index\` | +| Linux | `$XDG_CACHE_HOME/gridsift/{index,analysis}/` (default `~/.cache/gridsift/`) | +| macOS | `~/Library/Caches/gridsift/{index,analysis}/` | +| Windows | `%LOCALAPPDATA%\gridsift\{index,analysis}\` | -A sidecar is bound to the source's size and modification time and is -ignored when either changes. `gridsift index --index PATH` writes it +Both are bound to the source's size and modification time (and to its +SHA-256 once known) and are ignored when the file changes. `gridsift index --index PATH` writes it somewhere else instead. Exports and manifests are written only where the analyst asks for them. diff --git a/docs/usage.md b/docs/usage.md index 4f222e2..9cafa43 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -5,8 +5,13 @@ Every command accepts `--json` for machine-readable output and prints elapsed time, throughput and peak RSS. Dialect overrides (`-d ,` / `-d tab`, `--no-quote`, `--header` / `--no-header`) are accepted wherever a file is -read. The index sidecar is found in the user cache directory automatically; -`--index PATH` uses another one. +read, as is `--names a,b,c`: names for the columns of a file without a +header (it implies `--no-header`); they resolve `-c`, label the output and +are recorded in the manifest of an export. The index sidecar is found in +the user cache directory automatically; `--index PATH` uses another one. +Whole-file results are kept there too: `freq` and `timeline` without `-s` +and `profile` answer from the analysis cache the second time (`cached` +in their output) and a file that changed never uses it. ``` gridsift info FILE # dialect + first rows, no scan @@ -146,17 +151,22 @@ gridsift-desktop proxy.csv --search '/c2/beacon' --dashboard Launch options (`--search PATTERN [--regex]`, `--count COLUMN`, `--timeline`, `--dashboard`, `--domain COLUMN`) reproduce a state on -start; they exist for demos and screenshots. +start; they exist for demos and screenshots. `--header` / `--no-header` +and `--names a,b,c` read the file the way the command line would. ### The window - **Title bar** — file name, size, and the two badges: *EVIDENCE · READ ONLY* and *STRICT OFFLINE*. - **Evidence sidebar** — facts only: records, SHA-256 (✔ once computed), - index state and time, dialect, malformed counts. Below it every column - with its detected type; a click opens the column menu: *Count values · - Timeline · Search in this column · Enrich… · Redact on export…*. Then the - active enrichment rules and **Export finding…**. + index state and time, dialect, malformed counts. The dialect's *header* + / *no header* is a button: it reopens the file reading the first record + the other way. Below it every column with its detected type; a click + opens the column menu: *Count values · Timeline · Search in this column + · Enrich… · Redact on export…*. A file without a header shows *name + columns…*, which names its columns (kept by gridsift and recorded in + export manifests, never written to the file). Then the active enrichment + rules and **Export finding…**. - **Command bar** — one search field with *Regex*, *Exact* (whole-field match), *Aa* (case-insensitive), *Invert* and a column chip. Under it the **selection lineage** as chips, each with its match count: @@ -172,13 +182,17 @@ start; they exist for demos and screenshots. work is cancelled and exports are disabled until the file is reopened. - **Grid** — virtual rows over the index; typed headers; derived columns in green. *Go to row* jumps by ordinal. -- **Analysis dock** — *Values* (top-N of a column with share bars; click a - value to filter by it; the tab the dock opens on), *Dashboard* (below), - *Timeline* (drag a range on the chart → *Filter to range*), *Profile* - (the full column profile). +- **Analysis dock** — open from the start on *Values* (top-N of a column + with share bars; click a value to filter by it), then *Dashboard* + (below), *Timeline* (drag a range on the chart → *Filter to range*), + *Profile* (the full column profile). *Analysis* in the command bar hides + and shows it. - **Dashboard** — built from the file-wide column profile as soon as that is in and counted in the background, so it is ready when the tab is - opened: the timestamp column becomes a timeline card; columns with a + opened; whole-file counts, the timeline and the profile are kept in the + cache directory, so the next open of the same bytes shows the charts at + once (*cached* in the Values tab): the timestamp column becomes a + timeline card; columns with a handful of values (protocol, action, HTTP status, booleans) become pies; hosts, addresses, ports and paths become top-value bars; hashes, free text and plain numbers are skipped. Six panels at most by default —