diff --git a/.gitignore b/.gitignore index bc30a02..d9f1cdb 100644 --- a/.gitignore +++ b/.gitignore @@ -14,3 +14,6 @@ tmp/ # Python __pycache__/ *.pyc + +# Local agent notes +CLAUDE.local.md diff --git a/AGENTS.md b/AGENTS.md index 2c30636..63c8f8c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,5 +1,5 @@ -strict no backward compatibility no matter what, so feel free to make breaking changes as needed. Just make sure to -do `cargo clippy --all-targets -- -D warnings` after changes to make sure code style is correct and then `cargo test` as needed. +- strict no backward compatibility or legacy code paths no matter what, no change logs either. +- do `cargo clippy --all-targets -- -D warnings` after changes and then `cargo test` as needed. - Use `./tmp` as the temporary working directory for debug output, scratch files, etc. It is gitignored. - The project is a Cargo workspace: `crates/core` (`audio-pattern-detector-core`, the library) and `crates/cli` (`audio-pattern-detector`, the binary). Keep CLI-only dependencies (e.g. clap) out of the core crate. See `docs/development.md` for the code layout. @@ -21,9 +21,8 @@ do `cargo clippy --all-targets -- -D warnings` after changes to make sure code s ## Debugging opus/lossy audio -- Use `--debug` and `--debug-dir ` to save debug output per run. Use separate debug dirs for A/B comparisons. -- Debug graphs are not implemented yet (planned for v2, see `docs/roadmap.md`). Until then use the stderr diagnostics and the per-section peak dumps in `debug/cross_correlation_/`, which contain the MSE similarity and per-window Pearson r of every candidate. -- Debug audio sections in `audio_section/` can be listened to for ground truth verification. Don't assume old detection results are correct. +- There is no debug mode yet (planned for v2, see `docs/roadmap.md`). +- Listen to the audio around a candidate for ground truth verification. Don't assume old detection results are correct. - Pattern clips should be extracted from the same encoding as the target audio (e.g. from an Opus stream when matching Opus audio). Denoise as a fallback when source-matched clips aren't available. See `docs/denoise-strategy.md`. ## Version bumping diff --git a/Cargo.lock b/Cargo.lock index 6394e4d..0240cb5 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -54,7 +54,7 @@ dependencies = [ [[package]] name = "audio-pattern-detector" -version = "0.4.3" +version = "0.4.4" dependencies = [ "audio-pattern-detector-core", "clap", @@ -64,7 +64,7 @@ dependencies = [ [[package]] name = "audio-pattern-detector-core" -version = "0.4.3" +version = "0.4.4" dependencies = [ "base64", "fft-correlation", diff --git a/Cargo.toml b/Cargo.toml index 9be667a..1ff5c33 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -3,7 +3,7 @@ resolver = "2" members = ["crates/core", "crates/cli"] [workspace.package] -version = "0.4.3" +version = "0.4.4" edition = "2021" authors = ["Andrew Chen "] diff --git a/README.md b/README.md index e674748..907e6ca 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ Detects audio patterns specified by audio clips in target audio files. Designed Useful for AI workflows to efficiently segment audio files before processing (e.g., OpenAI Whisper transcription preprocessing). -Detection is a two-step process. **Step 1** always runs FFT cross-correlation against the audio to find and center potential match locations. **Step 2** verifies each candidate using one of three paths chosen by clip type: normal verification (partitioned MSE + multi-window Pearson correlation), short-clip verification (single-window variant for clips under 0.5s), or marker-tone verification (narrowband spectral check for `.apd.toml` patterns like station beeps). Robust against lossy-encoded audio (Opus, AAC). +Detection is a two-step process. **Step 1** always runs FFT cross-correlation against the audio to find and center potential match locations. **Step 2** verifies each candidate using one of three paths chosen by clip type: normal verification (partitioned MSE + center-window Pearson correlation), short-clip verification (single-window variant for clips under 0.5s), or marker-tone verification (narrowband spectral check for `.apd.toml` patterns like station beeps). Robust against lossy-encoded audio (Opus, AAC). Written in Rust: a single self-contained binary with no runtime dependencies (ffmpeg is only needed for non-WAV input files). Also available as a Python package with the same detector ([Python bindings](docs/python.md)). @@ -25,7 +25,7 @@ cargo install --git https://github.com/andrewtheguy/audio_pattern_detector.git - `ghcr.io/andrewtheguy/audio_pattern_detector:` is published with each release (`v`, plus `latest` for releases from `main`). It contains only the binary at `/usr/local/bin/audio-pattern-detector` (no ffmpeg), so the usual use is copying it into another image: ```dockerfile -COPY --from=ghcr.io/andrewtheguy/audio_pattern_detector:v0.4.3 /usr/local/bin/audio-pattern-detector /usr/local/bin/audio-pattern-detector +COPY --from=ghcr.io/andrewtheguy/audio_pattern_detector:v /usr/local/bin/audio-pattern-detector /usr/local/bin/audio-pattern-detector ``` The release workflow builds the image for amd64 and arm64 from the binaries it has already compiled (`runtime-prebuilt` target). To build it locally from source for the host architecture: `docker build --target runtime -t audio-pattern-detector .` @@ -93,8 +93,6 @@ audio-pattern-detector show-config ./clips/rthk_beep.apd.toml | `--chunk-seconds` | Seconds per chunk (default: 60, or "auto") | | `--timestamp-format` | JSONL timestamp fields: `both` (default), `ms`, or `formatted` | | `--height-min` | Minimum correlation peak height (default: 0.25; lower to find weak matches) | -| `--debug` | Enable debug mode (diagnostics, candidate audio, peak dumps; no charts yet) | -| `--debug-dir` | Base directory for debug output (default: ./tmp) | ## JSONL Output Format @@ -136,8 +134,8 @@ For custom audio sources, implement `SampleSource` and drive `AudioPatternDetect - **[Denoise Strategy](docs/denoise-strategy.md)** - How to denoise pattern clips for better matching with lossy-encoded or noisy audio - **[Python Bindings](docs/python.md)** - Python API, installation and building the wheel - **[Stdin Modes](docs/stdin-modes.md)** - WAV stdin and multiplexed stdin (IPC) with a Node.js example -- **[Development](docs/development.md)** - Building, linting, testing, code layout, debug output -- **[Roadmap](docs/roadmap.md)** - What v1 covers and the plan for debug charts in v2 +- **[Development](docs/development.md)** - Building, linting, testing, code layout +- **[Roadmap](docs/roadmap.md)** - The plan for the debug mode and its charts in v2 ## Development diff --git a/audio_pattern_detector.pyi b/audio_pattern_detector.pyi index 31f0ac0..1b6852d 100644 --- a/audio_pattern_detector.pyi +++ b/audio_pattern_detector.pyi @@ -41,8 +41,6 @@ class Detector: seconds_per_chunk: int | None = 60, target_sample_rate: int = 8000, height_min: float | None = None, - debug: bool = False, - debug_dir: StrPath = "./tmp", ) -> Detector: ... @property def clip_names(self) -> list[str]: ... diff --git a/crates/cli/src/main.rs b/crates/cli/src/main.rs index edfeba6..e6eafbf 100644 --- a/crates/cli/src/main.rs +++ b/crates/cli/src/main.rs @@ -71,14 +71,6 @@ struct MatchArgs { #[arg(long, value_name = "SECONDS", default_value = "60")] chunk_seconds: String, - /// Debug mode: diagnostics on stderr plus candidate audio and peak dumps under --debug-dir - #[arg(long)] - debug: bool, - - /// Base directory for debug output - #[arg(long, value_name = "DIR", default_value = "./tmp")] - debug_dir: PathBuf, - /// Override minimum correlation peak height (default: 0.25, lower to find weak matches) #[arg(long, value_name = "HEIGHT")] height_min: Option, @@ -158,10 +150,8 @@ fn cmd_match(args: MatchArgs) -> Result<()> { }; let options = MatchOptions { - debug_mode: args.debug, seconds_per_chunk, target_sample_rate: args.target_sample_rate, - debug_dir: args.debug_dir, height_min: args.height_min, }; diff --git a/crates/cli/tests/cli_integration.rs b/crates/cli/tests/cli_integration.rs index a50f65f..5a93e5b 100644 --- a/crates/cli/tests/cli_integration.rs +++ b/crates/cli/tests/cli_integration.rs @@ -8,7 +8,6 @@ //! Internal detection logic is tested in the other integration test files. use std::collections::BTreeSet; -use std::fs; use std::io::Write; use std::process::{Command, Output, Stdio}; @@ -30,9 +29,6 @@ const CBS_NEWS_CLIP_NAME: &str = "cbs_news"; const RAINBOW_INTRO_PATTERN: &str = "../../sample_audios/clips/天空下的彩虹intro.wav"; const PATTERN_FOLDER: &str = "../../sample_audios/clips"; const NONEXISTENT_FILE: &str = "nonexistent.wav"; -/// Pattern name that would escape the debug directory if used as a path. -const TRAVERSAL_CLIP_NAME: &str = "../../escape"; -const TRAVERSAL_SAFE_NAME: &str = ".._.._escape"; const DEFAULT_SAMPLE_RATE: u32 = 8000; struct CliResult { @@ -612,29 +608,3 @@ fn multiplexed_stdin_requires_no_pattern_file() { let result = run_cli_stdin(&["match", "--multiplexed-stdin"], payload); assert_eq!(result.code, 0); } - -#[test] -fn multiplexed_stdin_debug_output_stays_inside_debug_dir() { - let pattern_data = fs::read(CBS_NEWS_PATTERN).unwrap(); - let audio_data = fs::read(CBS_NEWS_AUDIO).unwrap(); - let payload = build_multiplexed_payload(&[(TRAVERSAL_CLIP_NAME, &pattern_data)], &audio_data); - - let root = tempfile::tempdir().unwrap(); - let debug_dir = root.path().join("nested").join("debug_out"); - let result = run_cli_stdin(&["match", "--multiplexed-stdin", "--debug", "--debug-dir", debug_dir.to_str().unwrap()], payload); - assert_eq!(detected_clip_names(&result.events()), BTreeSet::from([TRAVERSAL_CLIP_NAME.to_string()])); - - // Everything written under the temp root is inside debug_dir, in - // directories named after the sanitised clip name. - let mut top_level: Vec = - fs::read_dir(root.path()).unwrap().map(|e| e.unwrap().file_name().to_string_lossy().into_owned()).collect(); - top_level.sort(); - assert_eq!(top_level, vec!["nested".to_string()]); - let nested: Vec = fs::read_dir(root.path().join("nested")) - .unwrap() - .map(|e| e.unwrap().file_name().to_string_lossy().into_owned()) - .collect(); - assert_eq!(nested, vec!["debug_out".to_string()]); - assert!(debug_dir.join("audio_section").join(TRAVERSAL_SAFE_NAME).is_dir()); - assert!(debug_dir.join("debug").join(format!("cross_correlation_{TRAVERSAL_SAFE_NAME}")).is_dir()); -} diff --git a/crates/core/Cargo.toml b/crates/core/Cargo.toml index e33263f..0ba5403 100644 --- a/crates/core/Cargo.toml +++ b/crates/core/Cargo.toml @@ -22,8 +22,8 @@ pyo3 = { version = "0.28", optional = true, features = ["abi3-py39"] } realfft = "3.5" rustfft = "6.4" serde.workspace = true -serde_json.workspace = true toml = "1.1" [dev-dependencies] +serde_json.workspace = true tempfile.workspace = true diff --git a/crates/core/src/detector.rs b/crates/core/src/detector.rs index 111fec6..ff24c7b 100644 --- a/crates/core/src/detector.rs +++ b/crates/core/src/detector.rs @@ -1,10 +1,8 @@ use std::collections::{BTreeMap, HashSet}; -use std::path::PathBuf; use serde::ser::SerializeMap; use serde::{Serialize, Serializer}; use fft_correlation::{fft_correlate_1d, CorrelationTemplate, CorrelationWorkspace, Mode}; -use serde_json::json; use crate::audio_clip::{ validate_sample_rate, AudioClip, MarkerToneThresholds, Strategy, DEFAULT_TARGET_SAMPLE_RATE, @@ -19,7 +17,6 @@ use crate::time_format::seconds_to_time_whole; use crate::tone::{ analyze_pure_tone_candidate, extract_padded_segment, get_pure_tone_frequency, is_close, PureToneMetrics, }; -use crate::wav::write_wav_file; /// Default seconds per chunk for sliding window processing. pub const DEFAULT_SECONDS_PER_CHUNK: u32 = 60; @@ -49,16 +46,11 @@ pub type PeakTimes = BTreeMap>; #[derive(Debug, Clone)] pub struct DetectorOptions { - /// Enable debug output (diagnostics on stderr, candidate audio sections - /// and peak dumps under `debug_dir`). - pub debug_mode: bool, /// Seconds per chunk for sliding window processing. `None` or `0` /// auto-computes it as twice the longest clip (rounded up). pub seconds_per_chunk: Option, /// Sample rate of all clips and audio streams. pub target_sample_rate: u32, - /// Base directory for debug output files. - pub debug_dir: PathBuf, /// Override the minimum correlation peak height (default: 0.25). pub height_min: Option, } @@ -66,10 +58,8 @@ pub struct DetectorOptions { impl Default for DetectorOptions { fn default() -> Self { Self { - debug_mode: false, seconds_per_chunk: Some(DEFAULT_SECONDS_PER_CHUNK), target_sample_rate: DEFAULT_TARGET_SAMPLE_RATE, - debug_dir: PathBuf::from("./tmp"), height_min: None, } } @@ -128,22 +118,22 @@ struct ClipData { correlation_clip_absolute_max: f32, /// `Some` when candidates are verified as a marker tone. tone: Option, - /// Downsampled windows of `correlation_clip` used for Pearson r. - pearson_windows: Vec>, + /// Shorter than [`SHORT_CLIP_DURATION_THRESHOLD`]. + is_short_clip: bool, + /// Downsampled window of `correlation_clip` used for Pearson r. + pearson_window: Vec, } /// Window of the correlation envelope compared with Pearson r, as /// `(left partition, right partition, downsampled length)`. type PearsonWindow = (usize, usize, usize); -fn pearson_window_specs(is_short_clip: bool) -> (&'static [PearsonWindow], usize) { +fn pearson_window_spec(is_short_clip: bool) -> PearsonWindow { // Lengths are proportional to window width: round(101 * partitions / 2). - const SHORT: &[PearsonWindow] = &[(0, 10, 505)]; - const NORMAL: &[PearsonWindow] = &[(0, 5, 252), (4, 6, PEARSON_DOWNSAMPLE_BASE), (5, 10, 252)]; if is_short_clip { - (SHORT, 0) + (0, 10, 505) } else { - (NORMAL, 1) + (4, 6, PEARSON_DOWNSAMPLE_BASE) } } @@ -240,17 +230,8 @@ pub fn slicing_with_zero_padding(array: &[f32], width: usize, middle_index: usiz extract_padded_segment(array, begin, width) } -/// Per-candidate diagnostics dumped in debug mode. -#[derive(Default)] -struct ChunkDebug { - seconds: Vec, - similarities: Vec, -} - pub struct AudioPatternDetector { clips: Vec, - debug_mode: bool, - debug_dir: PathBuf, height_min: f32, target_sample_rate: u32, seconds_per_chunk: u32, @@ -264,7 +245,6 @@ impl AudioPatternDetector { pub fn new(audio_clips: Vec, options: DetectorOptions) -> Result { let sr = options.target_sample_rate; validate_sample_rate(sr)?; - let mut debug_mode = options.debug_mode; let mut names = HashSet::new(); let mut max_clip_length = 0; @@ -311,24 +291,14 @@ impl AudioPatternDetector { } } - if debug_mode && seconds_per_chunk != DEFAULT_SECONDS_PER_CHUNK { - eprintln!( - "seconds_per_chunk {seconds_per_chunk} is not 60 seconds, turning off debug mode \ - because it was made for 60 seconds only" - ); - debug_mode = false; - } - let clips: Vec = audio_clips .into_iter() - .map(|audio_clip| Self::prepare_clip(audio_clip, sr, debug_mode)) + .map(|audio_clip| Self::prepare_clip(audio_clip, sr)) .collect::>()?; let section_groups = section_groups(&clips); Ok(Self { clips, - debug_mode, - debug_dir: options.debug_dir, height_min: options.height_min.unwrap_or(DEFAULT_HEIGHT_MIN), target_sample_rate: sr, seconds_per_chunk, @@ -338,7 +308,7 @@ impl AudioPatternDetector { } /// Pre-compute everything about a clip that doesn't depend on the audio stream. - fn prepare_clip(audio_clip: AudioClip, sr: u32, debug_mode: bool) -> Result { + fn prepare_clip(audio_clip: AudioClip, sr: u32) -> Result { let clip_seconds = audio_clip.clip_length_seconds(); let sliding_window = clip_seconds.ceil() as u32; if sliding_window as f64 != clip_seconds { @@ -352,12 +322,6 @@ impl AudioPatternDetector { normalize_loudness(&mut clip, sr); let (correlation_clip, correlation_clip_absolute_max) = clip_correlation(&clip)?; - if debug_mode { - eprintln!("clip_length {} {}", audio_clip.name, clip.len()); - eprintln!("clip_length {} seconds {}", audio_clip.name, clip.len() as f64 / sr as f64); - eprintln!("correlation_clip_length {}", correlation_clip.len()); - } - let tone = match audio_clip.strategy { Some(Strategy::MarkerTone(params)) => params .dominant_frequency_hz @@ -372,11 +336,7 @@ impl AudioPatternDetector { }; let is_short_clip = clip_seconds < SHORT_CLIP_DURATION_THRESHOLD; - let pearson_windows = pearson_window_specs(is_short_clip) - .0 - .iter() - .map(|&window| downsample_window(&correlation_clip, window)) - .collect(); + let pearson_window = downsample_window(&correlation_clip, pearson_window_spec(is_short_clip)); Ok(ClipData { name: audio_clip.name, @@ -385,7 +345,8 @@ impl AudioPatternDetector { correlation_clip, correlation_clip_absolute_max, tone, - pearson_windows, + is_short_clip, + pearson_window, }) } @@ -398,11 +359,6 @@ impl AudioPatternDetector { self.target_sample_rate } - /// Whether debug output is active (it is turned off for non-default chunk sizes). - pub fn debug_mode(&self) -> bool { - self.debug_mode - } - /// Whether candidates for the named clip are verified as a marker tone. pub fn uses_marker_tone(&self, clip_name: &str) -> bool { self.clips.iter().any(|c| c.name == clip_name && c.tone.is_some()) @@ -573,7 +529,7 @@ impl AudioPatternDetector { let RunBuffers { workspace, templates, audio_section, correlation } = buffers; workspace.correlate(&mut templates[clip_index], Mode::Full, correlation)?; - let peaks = self.correlation_method(clip_data, audio_section, correlation, index)?; + let peaks = self.correlation_method(clip_data, audio_section, correlation, index); let section_start = index as i64 * self.seconds_per_chunk as i64 * sr as i64 - lookback_samples as i64; Ok(peaks @@ -596,104 +552,55 @@ impl AudioPatternDetector { audio_section: &[f32], correlation: &mut [f32], index: usize, - ) -> Result> { + ) -> Vec { let clip_name = &clip_data.name; let clip_length = clip_data.clip.len(); let correlation_clip_length = clip_data.correlation_clip.len(); - // Normalize by the larger of the two peaks so a much softer section - // cannot look like a full-strength match. + // Peak heights are relative to the larger of the two peaks so a much + // softer section cannot look like a full-strength match. The + // threshold is scaled instead of the correlation, which saves a pass + // over the section; verification normalizes each slice on its own. let absolute_max = absolute_in_place(correlation); let max_choose = clip_data.correlation_clip_absolute_max.max(absolute_max); - correlation.iter_mut().for_each(|v| *v /= max_choose); - - let section_ts = seconds_to_time_whole(index as f64 * self.seconds_per_chunk as f64); - if self.debug_mode { - eprintln!("---"); - eprintln!("section_ts: {section_ts}, index {index}"); - } + let height = self.height_min * max_choose; // No repetition within the duration of the clip. The height is kept // low so weak candidates are not missed; they are verified below. let peaks = find_peaks_1d( correlation, - &FindPeaksOptions { height: Some(self.height_min), distance: Some(clip_length), prominence: None }, + &FindPeaksOptions { height: Some(height), distance: Some(clip_length), prominence: None }, ); + let section_ts = || seconds_to_time_whole(index as f64 * self.seconds_per_chunk as f64); let mut peaks_final = Vec::new(); - let mut debug = ChunkDebug::default(); - for &peak in &peaks { + for peak in peaks { // Make sure the slice is not out of bounds at the beginning and end. let after = peak + correlation_clip_length / 2; let before = peak as isize - (correlation_clip_length / 2) as isize; if after > correlation.len() + 5 { eprintln!( - "{section_ts} {clip_name} peak {peak} after is {after} > len(correlation)+5 {}, skipping", + "{} {clip_name} peak {peak} after is {after} > len(correlation)+5 {}, skipping", + section_ts(), correlation.len() + 5 ); continue; } else if before < -5 { - eprintln!("{section_ts} {clip_name} peak {peak} before is {before} < -5, skipping"); + eprintln!("{} {clip_name} peak {peak} before is {before} < -5, skipping", section_ts()); continue; } let accepted = match &clip_data.tone { - Some(tone) => self.verify_marker_tone(tone, audio_section, peak, clip_length, §ion_ts), - None => self.verify_correlation_envelope(clip_data, correlation, peak, §ion_ts, &mut debug), + Some(tone) => self.verify_marker_tone(tone, audio_section, peak, clip_length), + None => self.verify_correlation_envelope(clip_data, correlation, peak), }; if accepted { peaks_final.push(peak); } - - if self.debug_mode { - self.write_debug_audio_section(clip_data, audio_section, peak, index, §ion_ts)?; - } - } - - if self.debug_mode && !peaks.is_empty() { - let peak_dir = self - .debug_dir - .join("debug") - .join(format!("cross_correlation_{}", safe_path_component(clip_name))); - std::fs::create_dir_all(&peak_dir)?; - let dump = json!({ - "peaks": peaks, - "seconds": debug.seconds, - "similarities": debug.similarities, - }); - let mut text = serde_json::to_string_pretty(&dump).expect("debug dump is valid JSON"); - text.push('\n'); - let file_name = format!("{index}_{}.txt", safe_path_component(§ion_ts)); - std::fs::write(peak_dir.join(file_name), text)?; - eprintln!("---"); } - Ok(peaks_final) - } - - /// Save the audio around a candidate so it can be listened to. - fn write_debug_audio_section( - &self, - clip_data: &ClipData, - audio_section: &[f32], - peak: usize, - index: usize, - section_ts: &str, - ) -> Result<()> { - let clip_name = safe_path_component(&clip_data.name); - let section_ts = safe_path_component(section_ts); - let audio_dir = self.debug_dir.join("audio_section").join(&clip_name); - std::fs::create_dir_all(&audio_dir)?; - - let clip_length = clip_data.clip.len(); - let start = peak.saturating_sub(clip_length).min(audio_section.len()); - let end = (peak + clip_length).min(audio_section.len()); - write_wav_file( - audio_dir.join(format!("{clip_name}_{index}_{section_ts}_{peak}.wav")), - &audio_section[start..end], - self.target_sample_rate, - ) + peaks_final } /// Verify a synthesized marker tone via short-time spectral analysis. @@ -709,7 +616,6 @@ impl AudioPatternDetector { audio_section: &[f32], peak: usize, clip_length: usize, - section_ts: &str, ) -> bool { let (metrics, left, right) = analyze_tone_candidate_context( audio_section, @@ -718,30 +624,7 @@ impl AudioPatternDetector { tone.dominant_frequency, self.target_sample_rate, ); - let accepted = marker_tone_accepts(tone.dominant_frequency, &tone.thresholds, &metrics, &left, &right); - - if self.debug_mode { - if !is_close(metrics.detected_frequency, tone.dominant_frequency, 0.05, 0.0) { - eprintln!( - "failed marker tone check for {section_ts}: dominant {:.1}Hz != expected {:.1}Hz", - metrics.detected_frequency, tone.dominant_frequency - ); - } else { - eprintln!( - "{} {section_ts}: band_purity={:.3} active_ratio={:.3} run={} active_purity={:.3} \ - freq={:.1}Hz flank_purity=({:.3}, {:.3})", - if accepted { "accepted marker tone" } else { "failed marker tone check for" }, - metrics.overall_band_purity, - metrics.active_frame_ratio, - metrics.longest_active_run, - metrics.active_frame_mean_purity, - metrics.detected_frequency, - left.overall_band_purity, - right.overall_band_purity, - ); - } - } - accepted + marker_tone_accepts(tone.dominant_frequency, &tone.thresholds, &metrics, &left, &right) } /// Verify a candidate by comparing the correlation envelope around the @@ -752,109 +635,41 @@ impl AudioPatternDetector { clip_data: &ClipData, correlation: &[f32], peak: usize, - section_ts: &str, - debug: &mut ChunkDebug, ) -> bool { let correlation_clip = &clip_data.correlation_clip; - let sr = self.target_sample_rate; let mut correlation_slice = slicing_with_zero_padding(correlation, correlation_clip.len(), peak); let slice_max = correlation_slice.iter().copied().fold(f32::NEG_INFINITY, f32::max); correlation_slice.iter_mut().for_each(|v| *v /= slice_max); let partition_size = correlation_clip.len() / PARTITION_COUNT; - let similarity_partitions: Vec = (0..PARTITION_COUNT) - .map(|i| { - let range = i * partition_size..(i + 1) * partition_size; - mean_squared_error(&correlation_clip[range.clone()], &correlation_slice[range]) - }) - .collect(); + let similarity_partitions: [f32; PARTITION_COUNT] = std::array::from_fn(|i| { + let range = i * partition_size..(i + 1) * partition_size; + mean_squared_error(&correlation_clip[range.clone()], &correlation_slice[range]) + }); let similarity_middle = mean(&similarity_partitions[MIDDLE_PARTITIONS]); let similarity_whole = mean(&similarity_partitions); - let is_short_clip = clip_data.clip.len() as f64 / (sr as f64) < SHORT_CLIP_DURATION_THRESHOLD; + let is_short_clip = clip_data.is_short_clip; let similarity = if is_short_clip { similarity_whole } else { // f32::min would skip a NaN operand; keep NaN so it is rejected below. if similarity_middle < similarity_whole { similarity_middle } else { similarity_whole } }; - let partition_summary = json!({"whole": similarity_whole as f64, "middle": similarity_middle as f64}); // A NaN similarity is rejected too. if similarity.is_nan() || similarity > SIMILARITY_HARD_LIMIT { - if self.debug_mode { - debug.seconds.push(peak as f64 / sr as f64); - debug.similarities.push(json!([similarity as f64, partition_summary, null])); - eprintln!( - "failed verification for {section_ts} due to similarity {similarity} > {SIMILARITY_HARD_LIMIT}" - ); - } return false; } - // Pearson r on the center window decides. The flanking windows are - // computed only for debug output. - let (windows, center_window_idx) = pearson_window_specs(is_short_clip); - let window_r = |window_idx: usize| { - let downsampled_slice = downsample_window(&correlation_slice, windows[window_idx]); - pearson_correlation_1d(&clip_data.pearson_windows[window_idx], &downsampled_slice) - }; - let pearson_r = window_r(center_window_idx); - - if self.debug_mode { - eprintln!("similarity {similarity} pearson_r {pearson_r}"); - let mut pearson_summary = serde_json::Map::new(); - let mut best = (center_window_idx, pearson_r); - for (window_idx, &(left, right, _)) in windows.iter().enumerate() { - let r = if window_idx == center_window_idx { pearson_r } else { window_r(window_idx) }; - if r > best.1 || (r == best.1 && window_idx < best.0) { - best = (window_idx, r); - } - pearson_summary.insert(format!("pearson_w{left}_{right}"), json!(r)); - } - let (best_left, best_right, _) = windows[best.0]; - let mut summary = serde_json::Map::new(); - summary.insert("pearson_r".into(), json!(pearson_r)); - summary.insert("best_window_left".into(), json!(best_left as f64)); - summary.insert("best_window_right".into(), json!(best_right as f64)); - summary.extend(pearson_summary); - - debug.seconds.push(peak as f64 / sr as f64); - debug.similarities.push(json!([similarity as f64, partition_summary, summary])); - } - - if pearson_r >= PEARSON_R_THRESHOLD { - true - } else { - if self.debug_mode { - eprintln!( - "failed verification for {section_ts} due to similarity {similarity} pearson_r {pearson_r}" - ); - } - false - } - } -} + // Pearson r on the center window decides. + let downsampled_slice = downsample_window(&correlation_slice, pearson_window_spec(is_short_clip)); + let pearson_r = pearson_correlation_1d(&clip_data.pearson_window, &downsampled_slice); -/// Turn a clip name or timestamp into a single portable file name component -/// for debug output. Clip names can come from untrusted input (multiplexed -/// stdin), so path separators and the characters Windows rejects (such as the -/// `:` of a timestamp) are replaced, and `.`/`..` cannot be produced. -pub fn safe_path_component(name: &str) -> String { - let mut component: String = name - .chars() - .map(|c| match c { - '/' | '\\' | ':' | '<' | '>' | '"' | '|' | '?' | '*' => '_', - c if c.is_control() => '_', - c => c, - }) - .collect(); - if component.is_empty() || component.chars().all(|c| c == '.') { - component.insert(0, '_'); + pearson_r >= PEARSON_R_THRESHOLD } - component } /// Analyze the candidate window plus the clip-length windows on either side. @@ -900,20 +715,6 @@ pub fn marker_tone_accepts( mod tests { use super::*; - #[test] - fn test_safe_path_component() { - assert_eq!(safe_path_component("rthk_beep"), "rthk_beep"); - assert_eq!(safe_path_component("天空下的彩虹intro"), "天空下的彩虹intro"); - assert_eq!(safe_path_component("00:39:00"), "00_39_00"); - assert_eq!(safe_path_component("../../escape"), ".._.._escape"); - assert_eq!(safe_path_component("..\\..\\escape"), ".._.._escape"); - assert_eq!(safe_path_component("/etc/passwd"), "_etc_passwd"); - assert_eq!(safe_path_component("ac\"d|e?f*g\0h"), "a_b_c_d_e_f_g_h"); - assert_eq!(safe_path_component(".."), "_.."); - assert_eq!(safe_path_component("."), "_."); - assert_eq!(safe_path_component(""), "_"); - } - #[test] fn test_zero_sample_rate_is_rejected() { let options = DetectorOptions { target_sample_rate: 0, ..DetectorOptions::default() }; diff --git a/crates/core/src/matching.rs b/crates/core/src/matching.rs index 7526c0a..19e941c 100644 --- a/crates/core/src/matching.rs +++ b/crates/core/src/matching.rs @@ -19,12 +19,10 @@ const MAX_PATTERN_DATA_BYTES: u32 = 100 * 1024 * 1024; #[derive(Debug, Clone)] pub struct MatchOptions { - pub debug_mode: bool, /// Seconds per chunk for the sliding window (`None` to auto-compute). pub seconds_per_chunk: Option, /// Sample rate used for processing; patterns and WAV files are resampled to it. pub target_sample_rate: u32, - pub debug_dir: PathBuf, /// Override the minimum correlation peak height (default: 0.25). pub height_min: Option, } @@ -32,10 +30,8 @@ pub struct MatchOptions { impl Default for MatchOptions { fn default() -> Self { Self { - debug_mode: false, seconds_per_chunk: Some(DEFAULT_SECONDS_PER_CHUNK), target_sample_rate: DEFAULT_TARGET_SAMPLE_RATE, - debug_dir: PathBuf::from("./tmp"), height_min: None, } } @@ -44,10 +40,8 @@ impl Default for MatchOptions { impl MatchOptions { fn detector_options(&self) -> DetectorOptions { DetectorOptions { - debug_mode: self.debug_mode, seconds_per_chunk: self.seconds_per_chunk, target_sample_rate: self.target_sample_rate, - debug_dir: self.debug_dir.clone(), height_min: self.height_min, } } diff --git a/crates/core/src/python.rs b/crates/core/src/python.rs index 71b3637..589af9d 100644 --- a/crates/core/src/python.rs +++ b/crates/core/src/python.rs @@ -140,8 +140,6 @@ impl PyDetector { seconds_per_chunk = Some(DEFAULT_SECONDS_PER_CHUNK), target_sample_rate = DEFAULT_TARGET_SAMPLE_RATE, height_min = None, - debug = false, - debug_dir = PathBuf::from("./tmp"), ))] fn new( py: Python<'_>, @@ -149,14 +147,10 @@ impl PyDetector { seconds_per_chunk: Option, target_sample_rate: u32, height_min: Option, - debug: bool, - debug_dir: PathBuf, ) -> PyResult { let options = DetectorOptions { - debug_mode: debug, seconds_per_chunk, target_sample_rate, - debug_dir, height_min, }; let detector = py.detach(|| { diff --git a/crates/core/tests/integration_matching.rs b/crates/core/tests/integration_matching.rs index c4d27a5..25384e3 100644 --- a/crates/core/tests/integration_matching.rs +++ b/crates/core/tests/integration_matching.rs @@ -49,7 +49,7 @@ fn assert_exists(path: &str) { assert!(Path::new(path).exists(), "File {path} not found"); } -/// `match_pattern(audio, patterns, debug_mode=False)` with accumulated results. +/// `match_pattern(audio, patterns)` with accumulated results. fn run_match(audio_file: &str, pattern_files: &[&str]) -> (PeakTimes, f64) { let (peak_times, total_time) = match_pattern(audio_file, pattern_files, &MatchOptions::default(), None, true).unwrap(); diff --git a/crates/core/tests/sliding_window.rs b/crates/core/tests/sliding_window.rs index ef4b2d0..f023a4e 100644 --- a/crates/core/tests/sliding_window.rs +++ b/crates/core/tests/sliding_window.rs @@ -24,7 +24,7 @@ const LONG_BEEP_DURATION: f64 = 2.5; const TONE_FREQUENCY: f64 = 1000.0; fn options(seconds_per_chunk: Option) -> DetectorOptions { - DetectorOptions { debug_mode: false, seconds_per_chunk, ..DetectorOptions::default() } + DetectorOptions { seconds_per_chunk, ..DetectorOptions::default() } } fn tone_clip(name: &str, frequency: f64, duration: f64) -> AudioClip { @@ -428,7 +428,7 @@ mod sliding_window_with_real_patterns { let (reference_results, _) = match_pattern( RTHK_BEEP_AUDIO, &[RTHK_BEEP_PATTERN], - &MatchOptions { debug_mode: false, ..MatchOptions::default() }, + &MatchOptions::default(), None, true, ) diff --git a/docs/development.md b/docs/development.md index 1e83ac2..03521c2 100644 --- a/docs/development.md +++ b/docs/development.md @@ -55,14 +55,7 @@ FFT cross-correlation comes from the [`fft-correlation`](https://github.com/andr ## Debug output -`--debug` writes diagnostics to stderr and files under `--debug-dir` (default `./tmp`, which is gitignored): - -- `audio_section//` — the audio around each candidate peak as WAV, for listening. -- `debug/cross_correlation_/` — per-section JSON dump of candidate peaks, their times, MSE similarity and Pearson r per window. - -Use separate `--debug-dir` values for A/B comparisons. Debug mode is only active with the default 60-second chunks. - -Graphs are not available yet; see [roadmap.md](roadmap.md) for the planned v2 charts. +There is no debug mode yet. See [roadmap.md](roadmap.md) for the plan. ## Detection Algorithm diff --git a/docs/pattern-matching.md b/docs/pattern-matching.md index c9d2728..aa43cd9 100644 --- a/docs/pattern-matching.md +++ b/docs/pattern-matching.md @@ -36,7 +36,7 @@ Raw audio stream (float32 PCM) │ │ │ NO → Correlation-envelope verification: │ │ • clip >= 0.5s → Normal: partitioned MSE │ - │ + 3-window Pearson r │ + │ + center-window Pearson r │ │ • clip < 0.5s → Short clip: whole-only MSE │ │ + single-window Pearson r │ └───────────────────────────────────────────────────────────┘ @@ -52,7 +52,7 @@ Before processing audio, each clip is prepared once: 2. **Self-correlation** - FFT cross-correlation of the clip with itself (`fft_correlate_1d(clip, clip, Mode::Full)` from the `fft-correlation` crate), producing a reference correlation curve. The absolute max is stored for normalization later. 3. **Tone-strategy setup** - for clips whose strategy is `marker_tone`, the dominant frequency is recorded as clip metadata. All other clips have no tone verifier. -This produces a `ClipData` struct per clip containing the normalized audio, clip name, sliding window, self-correlation curve, its absolute max, the downsampled Pearson windows of that curve, and the tone verifier (set only for tone-strategy clips). +This produces a `ClipData` struct per clip containing the normalized audio, clip name, sliding window, self-correlation curve, its absolute max, the downsampled Pearson window of that curve, and the tone verifier (set only for tone-strategy clips). ## Chunked Processing @@ -71,8 +71,8 @@ This step always runs first for every clip type. Its job is to locate and center For each chunk, each sliding-window group's `audio_section` is loaded into a `CorrelationWorkspace` (`load_signal`) from the `fft-correlation` crate. Then for each clip in the group: 1. Compute the full cross-correlation with `CorrelationWorkspace::correlate`, using the clip's `CorrelationTemplate` (its time-reversed spectrum, cached per FFT size for the run), and take the absolute value. The result is the same as `fft_correlate_1d(audio_section, clip, Mode::Full)`; the workspace only avoids repeating the section's forward FFT. -2. Normalize by `max(self_correlation_max, cross_correlation_max)` so the correlation curve is in [0, 1]. -3. Run peak detection with `height >= 0.25` and `distance >= clip_length` (prevents duplicate detections within one clip duration). +2. Take `max(self_correlation_max, cross_correlation_max)` as the reference peak, so a much softer section cannot look like a full-strength match. The correlation curve itself is left unscaled. +3. Run peak detection with `height >= 0.25 * reference peak` and `distance >= clip_length` (prevents duplicate detections within one clip duration). This is the same as a 0.25 threshold on the curve divided by the reference peak, without the extra pass over the section. Each peak is a candidate match location. Candidates are discarded only if the centered slice would extend more than 5 samples beyond the correlation array; otherwise zero-padding is used to keep the slice length consistent. @@ -80,7 +80,7 @@ Each peak is a candidate match location. Candidates are discarded only if the ce Every candidate peak from Step 1 is verified before being accepted as a match. The verifier branch is chosen by clip type, and the three branches below are alternatives — exactly one runs per candidate: -- **Normal Patterns** (`.wav` clips ≥ 0.5s) — partitioned MSE plus 3-window Pearson r on the centered correlation slice. +- **Normal Patterns** (`.wav` clips ≥ 0.5s) — partitioned MSE plus center-window Pearson r on the centered correlation slice. - **Short Clips** (`.wav` clips < 0.5s) — same correlation-envelope approach but with simplified single-window MSE and Pearson. - **Marker Tone** (`.apd.toml` clips with `strategy = "marker_tone"`) — narrowband spectral check at the declared dominant frequency, instead of the correlation-envelope shape check. @@ -97,14 +97,11 @@ Verification uses partitioned mean squared error (MSE) plus Pearson correlation The middle partitions are checked separately because real distortions tend to appear there. -2. **Multi-window Pearson correlation** - three overlapping regions of the curves are compared to find the best shape match: - - Window A: first half (partitions 0-4, 0-50%), downsampled to 252 points - - Window B: center (partitions 4-5, 40-60%), downsampled to 101 points - - Window C: second half (partitions 5-9, 50-100%), downsampled to 252 points +2. **Pearson correlation** - the center of the curves (partitions 4-5, 40-60%) is compared for shape. - Each window is downsampled using `resample_preserve_maxima` (sample count proportional to window width for consistent resolution), then the Pearson correlation coefficient is computed between the pattern's self-correlation window and the candidate's cross-correlation window. The best (highest) Pearson r across the three windows is used. + The window is downsampled to 101 points using `resample_preserve_maxima`, then the Pearson correlation coefficient is computed between the pattern's self-correlation window and the candidate's cross-correlation window. - Pearson r is scale-invariant — it measures shape similarity regardless of amplitude differences. This is important for lossy-encoded audio (e.g. Opus HLS streams) where codec artifacts inflate the correlation envelope but preserve the overall shape. The multi-window approach handles cases where the peak shape is slightly asymmetric or off-center. + Pearson r is scale-invariant — it measures shape similarity regardless of amplitude differences. This is important for lossy-encoded audio (e.g. Opus HLS streams) where codec artifacts inflate the correlation envelope but preserve the overall shape. 3. **Decision thresholds** (evaluated in order — MSE first, short-circuiting Pearson): - `similarity > 0.02` -> reject early (hard MSE ceiling; skip Pearson computation entirely) @@ -125,7 +122,7 @@ Short clips go through the normal correlation-envelope path but with simplified 1. **MSE** — only `similarity_whole` is used (no middle partition emphasis). The correlation envelope is too short for sub-region analysis to be meaningful. -2. **Pearson correlation** — a single 0-100% window (505 downsampled points) instead of the three partial windows. The full window captures the overall shape without splitting into regions that would be too small. +2. **Pearson correlation** — a single 0-100% window (505 downsampled points) instead of the center window. The full window captures the overall shape, since the center region alone would be too small. 3. **Same thresholds** — `similarity > 0.02` rejects early (no Pearson computation), `pearson_r >= 0.90` accepts. @@ -187,4 +184,4 @@ Accepted peaks (in sample indices) are converted to timestamps in fractional sec | Structure | Description | |-----------|-------------| | `AudioClip` | Input pattern: name, audio array, sample rate | -| `ClipData` | Pre-computed per clip: normalized audio, clip name, self-correlation curve, absolute max, sliding window, downsampled Pearson windows, tone verifier | +| `ClipData` | Pre-computed per clip: normalized audio, clip name, self-correlation curve, absolute max, sliding window, downsampled Pearson window, tone verifier | diff --git a/docs/python.md b/docs/python.md index 4351d65..8d85008 100644 --- a/docs/python.md +++ b/docs/python.md @@ -9,7 +9,7 @@ Wheels (Linux x86_64 and arm64, macOS Apple Silicon, Windows x86_64) are attache ```toml # pyproject.toml (uv) [project] -dependencies = ["audio-pattern-detector==0.4.3"] +dependencies = ["audio-pattern-detector=="] [[tool.uv.index]] name = "audio-pattern-detector" @@ -20,6 +20,8 @@ explicit = true audio-pattern-detector = { index = "audio-pattern-detector" } ``` +Replace `` with a released version, e.g. the latest from the releases page. + ```shell # pip pip install audio-pattern-detector --extra-index-url https://andrewtheguy.github.io/audio_pattern_detector/simple/ @@ -53,9 +55,9 @@ result = detector.match_wav_stream( ## API -### `Detector(pattern_files, *, seconds_per_chunk=60, target_sample_rate=8000, height_min=None, debug=False, debug_dir="./tmp")` +### `Detector(pattern_files, *, seconds_per_chunk=60, target_sample_rate=8000, height_min=None)` -Pattern files are `.wav` or `.apd.toml` paths (other formats are decoded through ffmpeg). The keyword arguments are the library equivalents of the CLI's `--chunk-seconds` (`None` is `auto`), `--target-sample-rate`, `--height-min`, `--debug` and `--debug-dir`. +Pattern files are `.wav` or `.apd.toml` paths (other formats are decoded through ffmpeg). The keyword arguments are the library equivalents of the CLI's `--chunk-seconds` (`None` is `auto`), `--target-sample-rate` and `--height-min`. | Member | Description | |--------|-------------| diff --git a/docs/roadmap.md b/docs/roadmap.md index 0a9c479..272e013 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1,42 +1,30 @@ # Roadmap -## v1 — Rust rewrite, no charts (current, released as 0.4.0) - -The detector is a single Rust crate (library + `audio-pattern-detector` binary), with optional Python bindings ([python.md](python.md)). Detection results match the previous Python implementation on all sample audio. - -`--debug` is text and audio only: - -| Output | Location under `--debug-dir` | -|--------|------------------------------| -| Per-candidate diagnostics (similarity, Pearson r, marker-tone metrics, rejection reason) | stderr | -| Audio around each candidate, for listening | `audio_section//___.wav` | -| Peak dump: candidate peaks, their times, MSE and per-window Pearson r | `debug/cross_correlation_/_.txt` (JSON) | - -Not in v1: any PNG graph output. The Python version drew these with matplotlib. - ## v2 — simple debug charts -Goal: bring back only the charts that are needed to tune detection, with only the chart elements needed to read them. Charts stay a debug aid: opt-in at build time, written to files, never displayed. +There is no debug mode yet; v2 adds one. + +Goal: only the charts that are needed to tune detection, with only the chart elements needed to read them. Charts are a debug aid: opt-in at build time, written to files, never displayed. ### Charts -| Chart | File under `--debug-dir` | What it answers | +| Chart | File under the debug directory | What it answers | |-------|--------------------------|-----------------| -| Pearson windows | `graph/pearson_downsampled//____w_.png` | The downsampled candidate window against the pattern's window — the curves verification actually compares. Check this first. | +| Pearson window | `graph/pearson_downsampled//___.png` | The downsampled candidate window against the pattern's window — the curves verification actually compares. Check this first. | | Correlation slice | `graph/cross_correlation_slice//___.png` | The full-resolution candidate envelope against the pattern's self-correlation: where does the shape diverge? | | Section correlation | `graph/cross_correlation//__.png` | The whole section's correlation curve: were there candidate peaks at all, and how high? | -The first two apply to the correlation-envelope paths (normal and short clips) and, as before, are only drawn for candidates with similarity ≤ 0.1. The third applies to every clip, including marker tones. +The first two apply to the correlation-envelope paths (normal and short clips) and are only drawn for candidates with similarity ≤ 0.1. The third applies to every clip, including marker tones. -Dropped from the Python version: +Not planned: -- `clip_correlation` and `cross_correlation_slice_original` — the pattern's self-correlation on its own; the same curve is already overlaid in the correlation slice chart. -- `mean_squared_error_similarity` scatter — the values are in the peak dump. +- A chart of the pattern's self-correlation on its own — the same curve is already overlaid in the correlation slice chart. +- A scatter of the MSE similarities — plain numbers are enough for those. ### Chart elements - Line series only: candidate in one colour, pattern overlaid in a second, semi-transparent colour. -- Title (carrying the Pearson r and the `*best*` window marker where relevant). +- Title (carrying the Pearson r where relevant). - X and Y axes with tick labels and an axis label each. - Fixed 1000×400 px PNG. @@ -46,11 +34,11 @@ Not included: legends, grid lines, themes, annotations, interactivity, configura - Library: [`plotters`](https://crates.io/crates/plotters) with the bitmap backend only (`default-features = false`). - Text needs a font. Use the `ab_glyph` feature with a font embedded via `include_bytes!` so the binary has no fontconfig/freetype dependency and renders identically everywhere. -- Gate everything behind an optional `charts` cargo feature so default builds do not carry the dependency. Without the feature `--debug` behaves as in v1. -- The detector already has the data at the right places: `verify_correlation_envelope` holds the slice, the pattern curve and the downsampled windows; `correlation_method` holds the section correlation. -- Debug file names write `` with `_` instead of `:` (`00_39_00`) so they are valid on Windows, and clip names are sanitised to a single path component. Chart file names should go through the same `safe_path_component` helper. +- Gate the charts behind an optional `charts` cargo feature so default builds do not carry the dependency. +- The detector has the data at the right places: `verify_correlation_envelope` holds the slice, the pattern curve and the downsampled center window; `correlation_method` holds the section correlation. That curve is unscaled (the peak threshold is scaled by the reference peak instead), so the section chart has to divide it by the reference peak to show heights against the 0.25 threshold. +- Debug file names write `` with `_` instead of `:` (`00_39_00`) so they are valid on Windows. Clip names can come from untrusted input (multiplexed stdin), so they must be sanitised to a single path component before being used in a file name. ### Open questions -- Debug mode is still switched off unless `--chunk-seconds` is 60 (inherited from the Python version). Nothing in the v1 output depends on that; decide whether to lift it together with the charts. +- Which text and audio output the debug mode has besides the charts (per-candidate diagnostics, the audio around each candidate, peak dumps). - Whether marker-tone candidates deserve a chart of their own (e.g. per-frame band purity). Not planned until a tuning session shows the stderr metrics are not enough.