Summary
BatchIterator cuts audio into hard, non-overlapping 30s windows, so a word spanning a 30s boundary is split across two windows. Each window is decoded independently, so the word tends to be dropped, duplicated, or garbled at the seam.
Where
pipeline/main.py:
# BatchIterator
sample = self.array[self._idx : self._idx + self.sample_len]
self._idx += self.sample_len # advances a full window — no overlap
...
@property
def sample_len(self) -> int:
return self.sampling_rate * 30Windows are [0:30], [30:60], [60:90]… with zero overlap, and process_batch stitches them with a fixed offset = base_offset + 30 * idx. A word at, say, 29.5–30.5s has its first 0.5s in window 0 and its back 0.5s in window 1; Whisper sees two truncated fragments and typically emits the word twice, drops it, or produces a garbled token.
Suggested fix
Advance windows by 30 - overlap (e.g. a 5s overlap) and de-duplicate the shared region when stitching — keep window N's segments up to the seam midpoint and window N+1's from the midpoint on, preferring whichever window decoded the word more interior. Windows stay independent, so GPU batching is preserved; only ~overlap/30 extra compute.
Context
We hit this porting the engine into tigerflow-ml's transcription task and implemented the overlap+merge fix there (princeton-ddss/tigerflow-ml#118). Filing here so the service can match.
Summary
BatchIteratorcuts audio into hard, non-overlapping 30s windows, so a word spanning a 30s boundary is split across two windows. Each window is decoded independently, so the word tends to be dropped, duplicated, or garbled at the seam.Where
pipeline/main.py:Windows are
[0:30],[30:60],[60:90]… with zero overlap, andprocess_batchstitches them with a fixedoffset = base_offset + 30 * idx. A word at, say, 29.5–30.5s has its first 0.5s in window 0 and its back 0.5s in window 1; Whisper sees two truncated fragments and typically emits the word twice, drops it, or produces a garbled token.Suggested fix
Advance windows by
30 - overlap(e.g. a 5s overlap) and de-duplicate the shared region when stitching — keep window N's segments up to the seam midpoint and window N+1's from the midpoint on, preferring whichever window decoded the word more interior. Windows stay independent, so GPU batching is preserved; only ~overlap/30extra compute.Context
We hit this porting the engine into tigerflow-ml's transcription task and implemented the overlap+merge fix there (princeton-ddss/tigerflow-ml#118). Filing here so the service can match.