Skip to content

feat: optimize memory usage and add --auto-bf option - #939

Open
cyw-cc wants to merge 1 commit into
chhylp123:masterfrom
cyw-cc:master
Open

feat: optimize memory usage and add --auto-bf option#939
cyw-cc wants to merge 1 commit into
chhylp123:masterfrom
cyw-cc:master

Conversation

@cyw-cc

@cyw-cc cyw-cc commented Jul 26, 2026

Copy link
Copy Markdown

Apply three sets of optimizations:

  1. Memory/threading optimizations (kalloc, kthread, Hash_Table, POA):

    • Improve memory allocation in kalloc.cpp
    • Optimize threading in kthread.cpp
    • Refactor Hash_Table.cpp for better memory handling
    • Minor POA.cpp adjustments
  2. Levenshtein distance optimizations (Correct, Levenshtein_distance.h):

    • Add SIMD-accelerated banded BPM functions (AVX2/SSE)
    • Add Reserve_Banded_BPM_8_AVX2_only and Reserve_Banded_BPM_4_SSE_only
    • Expand Levenshtein_distance.h with optimized edit distance implementation
  3. Automatic bloom filter detection (--auto-bf, id=366):

    • sys.cpp: yak_available_mem_gb() reads MemAvailable from /proc/meminfo
    • Process_Read.cpp: ha_scan_read_stats() samples read lengths (avg/N50/min/max)
    • htab.cpp: ha_auto_bf_shift() two-level decision logic
      • Level 1: disable bloom filter if available memory insufficient
      • Level 2: disable bloom filter for HiFi-like data (avg>8kb, N50>8kb)
    • Assembly.cpp: call ha_auto_bf_shift() before ha_ft_gen()

Test results (SRR11606870, 4M HiFi reads, 16 threads):

  • Peak RSS: 24.92 -> 16.92 GB (-32%)
  • Real time: 1467 -> 1171 sec (-20.2%)
  • Output: identical to baseline across all 5 GFA files

Apply three sets of optimizations:

1. Memory/threading optimizations (kalloc, kthread, Hash_Table, POA):
   - Improve memory allocation in kalloc.cpp
   - Optimize threading in kthread.cpp
   - Refactor Hash_Table.cpp for better memory handling
   - Minor POA.cpp adjustments

2. Levenshtein distance optimizations (Correct, Levenshtein_distance.h):
   - Add SIMD-accelerated banded BPM functions (AVX2/SSE)
   - Add Reserve_Banded_BPM_8_AVX2_only and Reserve_Banded_BPM_4_SSE_only
   - Expand Levenshtein_distance.h with optimized edit distance implementation

3. Automatic bloom filter detection (--auto-bf, id=366):
   - sys.cpp: yak_available_mem_gb() reads MemAvailable from /proc/meminfo
   - Process_Read.cpp: ha_scan_read_stats() samples read lengths (avg/N50/min/max)
   - htab.cpp: ha_auto_bf_shift() two-level decision logic
     * Level 1: disable bloom filter if available memory insufficient
     * Level 2: disable bloom filter for HiFi-like data (avg>8kb, N50>8kb)
   - Assembly.cpp: call ha_auto_bf_shift() before ha_ft_gen()

Test results (SRR11606870, 4M HiFi reads, 16 threads):
- Peak RSS: 24.92 -> 16.92 GB (-32%)
- Real time: 1467 -> 1171 sec (-20.2%)
- Output: identical to baseline across all 5 GFA files
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant