Conformer output depends on padding, silently hurting accuracy at inference (convolution module ignores lengths)

[proshian]

🐛 Describe the bug

torchaudio.models.Conformer gives different outputs for the same sequence depending on how many padding frames are appended to it (even in eval() mode with correct lengths). (In practice the padding amount varies because sequences are padded to the longest one in their batch, so a sequence's output silently depends on its batch mates.) Attention respects the padding mask derived from lengths, but _ConvolutionModule ignores it, in two ways:

1. Padding bleeds into valid frames through the depthwise convolution. The depthwise conv reads each frame's neighbors, so a sequence's last valid frames pull in whatever follows them. Run alone, that's the conv's own zero-padding; batched with a longer sequence, it's the padding frames - which carry non-zeros (even if they start at zero). So non-zero padding leaks into valid frames, and attention then carries it across the whole sequence.
2. Normalization statistics include padded steps. Both BatchNorm1d (default) and GroupNorm (use_group_norm=True) compute their mean/var over the time axis, so padded frames enter the statistics and skew the normalization applied to the valid ones.

Minimal reproduction:

import torch
from torchaudio.models import Conformer

torch.manual_seed(0)
INPUT_DIM, SHORT_LEN, LONG_LEN = 80, 50, 400

model = Conformer(
    input_dim=INPUT_DIM, num_heads=4, ffn_dim=128, num_layers=4, depthwise_conv_kernel_size=31
).eval()

short = torch.rand(1, SHORT_LEN, INPUT_DIM)
long = torch.rand(1, LONG_LEN, INPUT_DIM)

# (a) the short sequence alone in its batch: no padding at all
out_alone, _ = model(short, torch.tensor([SHORT_LEN]))

# (b) the same sequence zero-padded to share a batch with the long sequence
padded_short = torch.nn.functional.pad(short, (0, 0, 0, LONG_LEN - SHORT_LEN))
out_batch, _ = model(torch.cat([long, padded_short]), torch.tensor([LONG_LEN, SHORT_LEN]))

max_diff = (out_alone[0] - out_batch[1, :SHORT_LEN]).abs().max()
scale = out_alone.abs().mean()
print(f"max |out_alone - out_batched| = {max_diff:.6f} (mean |out| = {scale:.4f})")

Observed:

max |out_alone - out_batched| = 0.282953 (mean |out| = 0.8238)

Expected: a difference at float-precision level (~1e-6), since both calls process the same sequence and lengths marks the padding. With use_group_norm=True the same script gives max diff = 1.226370 — larger than the mean output magnitude.

Real-world impact. I hit this in practice (proshian/neural-swipe-typing, swipe-gesture decoding, where sequence lengths vary a lot within a batch). A Conformer encoder trained with large mixed-length batches had strong batched-validation metrics, but when switched to deployment-style inference (one sequence at a time, no padding) the metrics dropped dramatically. The model had learned to rely on padding-dependent behavior; nothing in the API suggests that the output of Conformer(input, lengths) depends on batch composition.

How other implementations handle this. Masking before the depthwise convolution and offering a padding-invariant normalization:

• NVIDIA NeMo zeroes padded frames right before the depthwise conv (x = x.masked_fill(pad_mask.unsqueeze(1), 0.0)) and exposes a norm_type option including layer_norm: nemo/collections/asr/parts/submodules/conformer_modules.py
• WeNet masks padded frames both before the depthwise conv and at the module output, and supports batch_norm/layer_norm: wenet/models/transformer/convolution.py

I have a backward-compatible fix (opt-in conv_mask_padding and conv_norm_type arguments; default behavior and state_dict layout unchanged) and will open a PR referencing this issue.

Versions

Collecting environment information...
PyTorch version: 2.13.0+cpu
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Pro (10.0.19045 64-bit)
GCC version: (rubenvb-4.6.3) 4.6.3
Clang version: 17.0.1
CMake version: version 4.2.1
Libc version: N/A

Python version: 3.12.9 (main, Feb 12 2025, 14:52:31) [MSC v.1942 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19045-SP0
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
Is XPU available: False
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: False
Caching allocator config: N/A

CPU:
Name: Intel(R) Core(TM) i3-6100U CPU @ 2.30GHz
Manufacturer: GenuineIntel
Family: 206
Architecture: 9
ProcessorType: 3
DeviceID: CPU0
CurrentClockSpeed: 2301
MaxClockSpeed: 2301
L2CacheSize: 512
L2CacheSpeed: None
Revision: 19971

Versions of relevant libraries:
[pip3] numpy==2.5.1
[pip3] torch==2.13.0+cpu
[pip3] torchaudio==2.11.0+cpu
[conda] No relevant packages