VMD causes excessive memory usage for long signals due to storing all iteration results

[liwenchao36]

Description

When using PySDKit.VMD on a relatively long time series, the VMD implementation attempts to allocate a very large amount of memory and raises a numpy.core._exceptions._ArrayMemoryError.

The main source of the memory usage seems to be this allocation in pysdkit/_vmd/vmd_c.py:

u_hat_plus = np.zeros(
    [self.max_iter, len(freqs), self.K],
    dtype=complex
)

This stores the frequency-domain modes for all iterations.
For long signals, the memory requirement becomes very large because its complexity is approximately:
O(max_iter * signal_length * K)

Example

For my data:
Original signal length: ~265,140
Mirrored signal length / len(freqs): 530,280
max_iter: 500
K: 8
dtype: complex128

PySDKit tries to allocate:
shape = (500, 530280, 8)

which requires about:
31.6 GiB

Full traceback:
File ".../pysdkit/_vmd/vmd_c.py", line 186, in fit_transform
    u_hat_plus = np.zeros([self.max_iter, len(freqs), self.K], dtype=complex)

numpy.core._exceptions._ArrayMemoryError:
Unable to allocate 31.6 GiB for an array with shape
(500, 530280, 8) and data type complex128

[wwhenxuan]

Hi @liwenchao36 👋, thanks for the clear report and the exact memory breakdown — that was very helpful.

Why this error occurs

The crash comes from a large temporary allocation in classic VMD (pysdkit/_vmd/vmd_c.py), not from NumPy itself.

Before ADMM iterations, the implementation mirror-extends the input (working length ≈ 2× the original length), then allocates frequency-domain buffers that keep every iteration:

u_hat_plus  ~ (max_iter, T_mirrored, K)   complex128
lambda_hat  ~ (max_iter, T_mirrored)      complex128

With your numbers:

• original length ≈ 265,140
• mirrored length / len(freqs) ≈ 530,280
• max_iter = 500, K = 8
• u_hat_plus ≈ (500, 530280, 8) → about 31.6 GiB

So peak memory scales roughly as (O(\texttt{max_iter} \times T_{\text{mirrored}} \times K)), and long records can exhaust RAM even when the algorithm is otherwise fine.

Analysis

Two points are easy to mix up:

1. Implementation / memory
   Storing the full spectral history is convenient, but ADMM only needs the previous / current iterate. That full-history buffer is what made long signals impractical.

2. Decomposition quality vs. length
   Classic VMD assumes approximately stationary narrowband modes over the processed segment. Length alone does not necessarily hurt quality; the immediate failure here is memory. Segmentation remains a possible workaround, but it can introduce boundary artifacts and mode-alignment issues across chunks.

What we changed (released in v0.4.45)

We added a controllable parameter:

from pysdkit import VMD

# default: keep full spectral history (previous behavior)
vmd = VMD(alpha=2000, K=8, tau=0.0, store_history=True)

# recommended for long signals: rolling buffers only
vmd = VMD(alpha=2000, K=8, tau=0.0, store_history=False)
IMFs = vmd.fit_transform(signal)

• store_history=True (default): previous behavior, full (max_iter, T, K) history
• store_history=False: keep only previous/current spectral buffers → memory about (O(T \times K))

This is a memory optimization of the same ADMM updates, not a different decomposition model. On the same demo signal, reconstruction quality stays essentially the same; the main win is peak memory (and sometimes earlier stopping with a clearer residual check).

The functional API supports the same flag:

from pysdkit import vmd
u, u_hat, omega = vmd(signal, alpha=2000, K=8, tau=0.0, store_history=False)

A comparison notebook is available here:
examples/vmd/vmd_store_history.ipynb

How to get the fix

Please upgrade to PySDKit 0.4.45 (now on PyPI):

pip install -U PySDKit

Then rerun with store_history=False for long records.

[liwenchao36]

感谢您的详细分析和修复方案，我会升级后测试store_history=False的效果。 在 2026-08-11 15:52:23，"WhenXuan" ***@***.***> 写道： wwhenxuan left a comment (wwhenxuan/PySDKit#53) Hi @liwenchao36 👋, thanks for the clear report and the exact memory breakdown — that was very helpful. Why this error occurs The crash comes from a large temporary allocation in classic VMD (pysdkit/_vmd/vmd_c.py), not from NumPy itself. Before ADMM iterations, the implementation mirror-extends the input (working length ≈ 2× the original length), then allocates frequency-domain buffers that keep every iteration: u_hat_plus ~ (max_iter, T_mirrored, K) complex128 lambda_hat ~ (max_iter, T_mirrored) complex128 With your numbers: original length ≈ 265,140 mirrored length / len(freqs) ≈ 530,280 max_iter = 500, K = 8 u_hat_plus ≈ (500, 530280, 8) → about 31.6 GiB So peak memory scales roughly as (O(\texttt{max_iter} \times T_{\text{mirrored}} \times K)), and long records can exhaust RAM even when the algorithm is otherwise fine. Analysis Two points are easy to mix up: Implementation / memory Storing the full spectral history is convenient, but ADMM only needs the previous / current iterate. That full-history buffer is what made long signals impractical. Decomposition quality vs. length Classic VMD assumes approximately stationary narrowband modes over the processed segment. Length alone does not necessarily hurt quality; the immediate failure here is memory. Segmentation remains a possible workaround, but it can introduce boundary artifacts and mode-alignment issues across chunks. What we changed (released in v0.4.45) We added a controllable parameter: frompysdkitimportVMD# default: keep full spectral history (previous behavior)vmd=VMD(alpha=2000, K=8, tau=0.0, store_history=True) # recommended for long signals: rolling buffers onlyvmd=VMD(alpha=2000, K=8, tau=0.0, store_history=False) IMFs=vmd.fit_transform(signal) store_history=True (default): previous behavior, full (max_iter, T, K) history store_history=False: keep only previous/current spectral buffers → memory about (O(T \times K)) This is a memory optimization of the same ADMM updates, not a different decomposition model. On the same demo signal, reconstruction quality stays essentially the same; the main win is peak memory (and sometimes earlier stopping with a clearer residual check). The functional API supports the same flag: frompysdkitimportvmdu, u_hat, omega=vmd(signal, alpha=2000, K=8, tau=0.0, store_history=False) A comparison notebook is available here: examples/vmd/vmd_store_history.ipynb How to get the fix Please upgrade to PySDKit 0.4.45 (now on PyPI): pip install -U PySDKit Then rerun with store_history=False for long records. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

[liwenchao36]

现在是大数据的时代，要增加实用性，必须接收更大的数据，不是简单的限制 在 2026-08-11 22:04:52，"立聊里" ***@***.***> 写道： 感谢您的详细分析和修复方案，我会升级后测试store_history=False的效果。 在 2026-08-11 15:52:23，"WhenXuan" ***@***.***> 写道： wwhenxuan left a comment (wwhenxuan/PySDKit#53) Hi @liwenchao36 👋, thanks for the clear report and the exact memory breakdown — that was very helpful. Why this error occurs The crash comes from a large temporary allocation in classic VMD (pysdkit/_vmd/vmd_c.py), not from NumPy itself. Before ADMM iterations, the implementation mirror-extends the input (working length ≈ 2× the original length), then allocates frequency-domain buffers that keep every iteration: u_hat_plus ~ (max_iter, T_mirrored, K) complex128 lambda_hat ~ (max_iter, T_mirrored) complex128 With your numbers: original length ≈ 265,140 mirrored length / len(freqs) ≈ 530,280 max_iter = 500, K = 8 u_hat_plus ≈ (500, 530280, 8) → about 31.6 GiB So peak memory scales roughly as (O(\texttt{max_iter} \times T_{\text{mirrored}} \times K)), and long records can exhaust RAM even when the algorithm is otherwise fine. Analysis Two points are easy to mix up: Implementation / memory Storing the full spectral history is convenient, but ADMM only needs the previous / current iterate. That full-history buffer is what made long signals impractical. Decomposition quality vs. length Classic VMD assumes approximately stationary narrowband modes over the processed segment. Length alone does not necessarily hurt quality; the immediate failure here is memory. Segmentation remains a possible workaround, but it can introduce boundary artifacts and mode-alignment issues across chunks. What we changed (released in v0.4.45) We added a controllable parameter: frompysdkitimportVMD# default: keep full spectral history (previous behavior)vmd=VMD(alpha=2000, K=8, tau=0.0, store_history=True) # recommended for long signals: rolling buffers onlyvmd=VMD(alpha=2000, K=8, tau=0.0, store_history=False) IMFs=vmd.fit_transform(signal) store_history=True (default): previous behavior, full (max_iter, T, K) history store_history=False: keep only previous/current spectral buffers → memory about (O(T \times K)) This is a memory optimization of the same ADMM updates, not a different decomposition model. On the same demo signal, reconstruction quality stays essentially the same; the main win is peak memory (and sometimes earlier stopping with a clearer residual check). The functional API supports the same flag: frompysdkitimportvmdu, u_hat, omega=vmd(signal, alpha=2000, K=8, tau=0.0, store_history=False) A comparison notebook is available here: examples/vmd/vmd_store_history.ipynb How to get the fix Please upgrade to PySDKit 0.4.45 (now on PyPI): pip install -U PySDKit Then rerun with store_history=False for long records. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

[liwenchao36]

能否增加一些自适应的分解方法 在 2026-08-12 08:38:12，"立聊里" ***@***.***> 写道： 现在是大数据的时代，要增加实用性，必须接收更大的数据，不是简单的限制 在 2026-08-11 22:04:52，"立聊里" ***@***.***> 写道： 感谢您的详细分析和修复方案，我会升级后测试store_history=False的效果。 在 2026-08-11 15:52:23，"WhenXuan" ***@***.***> 写道： wwhenxuan left a comment (wwhenxuan/PySDKit#53) Hi @liwenchao36 👋, thanks for the clear report and the exact memory breakdown — that was very helpful. Why this error occurs The crash comes from a large temporary allocation in classic VMD (pysdkit/_vmd/vmd_c.py), not from NumPy itself. Before ADMM iterations, the implementation mirror-extends the input (working length ≈ 2× the original length), then allocates frequency-domain buffers that keep every iteration: u_hat_plus ~ (max_iter, T_mirrored, K) complex128 lambda_hat ~ (max_iter, T_mirrored) complex128 With your numbers: original length ≈ 265,140 mirrored length / len(freqs) ≈ 530,280 max_iter = 500, K = 8 u_hat_plus ≈ (500, 530280, 8) → about 31.6 GiB So peak memory scales roughly as (O(\texttt{max_iter} \times T_{\text{mirrored}} \times K)), and long records can exhaust RAM even when the algorithm is otherwise fine. Analysis Two points are easy to mix up: Implementation / memory Storing the full spectral history is convenient, but ADMM only needs the previous / current iterate. That full-history buffer is what made long signals impractical. Decomposition quality vs. length Classic VMD assumes approximately stationary narrowband modes over the processed segment. Length alone does not necessarily hurt quality; the immediate failure here is memory. Segmentation remains a possible workaround, but it can introduce boundary artifacts and mode-alignment issues across chunks. What we changed (released in v0.4.45) We added a controllable parameter: frompysdkitimportVMD# default: keep full spectral history (previous behavior)vmd=VMD(alpha=2000, K=8, tau=0.0, store_history=True) # recommended for long signals: rolling buffers onlyvmd=VMD(alpha=2000, K=8, tau=0.0, store_history=False) IMFs=vmd.fit_transform(signal) store_history=True (default): previous behavior, full (max_iter, T, K) history store_history=False: keep only previous/current spectral buffers → memory about (O(T \times K)) This is a memory optimization of the same ADMM updates, not a different decomposition model. On the same demo signal, reconstruction quality stays essentially the same; the main win is peak memory (and sometimes earlier stopping with a clearer residual check). The functional API supports the same flag: frompysdkitimportvmdu, u_hat, omega=vmd(signal, alpha=2000, K=8, tau=0.0, store_history=False) A comparison notebook is available here: examples/vmd/vmd_store_history.ipynb How to get the fix Please upgrade to PySDKit 0.4.45 (now on PyPI): pip install -U PySDKit Then rerun with store_history=False for long records. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

[wwhenxuan]

Hi @liwenchao36 ，感谢您的反馈。您这两条评论其实对应两个不同的问题，我分开进行介绍。

1. 自适应分解：库里已经有一批，不只是预设 $K$ 的 VMD

经典 VMD 需要事先指定模态数 $K$ 和带宽参数 $\alpha$。如果您希望算法自己根据数据决定模态数、带宽或瞬时频率，PySDKit 里已经有多条路线：

需求	方法	说明
不必预先指定 $K$	EMD / CEEMDAN / FAEMD / ALIF	按极值/滤波自适应筛分
变分框架、逐个提取、无需 $K$	SVMD、SJMD	用停止准则决定何时停
频谱自适应划分	EWT / EFD	按 Fourier 边界建滤波器组
自适应带宽 / 瞬时频率（chirp）	AVNCMD、ACMD、BA_ACMD、DD_ACMD、APMD	跟踪时变 IF，不必手调 (\alpha)
非平稳、短时处理	STVMD（dynamic=True）、STNBMD	按帧分解，更适合长记录

和您之前那条 VMD 记录最接近、又不必指定 $K$ 的，是 SVMD：

from pysdkit import SVMD

svmd = SVMD(max_alpha=20000, tau=0.0, stopc=4, max_modes=10)
IMFs = svmd.fit_transform(signal)   # 模态数由停止准则决定

如果信号是调频/非线性 chirp，而不是近似平稳窄带模态，可以用 AVNCMD（自适应更新瞬时频率，不再依赖 VNCMD 里那个敏感的 $\alpha$：

from pysdkit import AVNCMD
import numpy as np

# iniIF: 每行一个模态的初始瞬时频率，形状 (K, N)
# 不知道精确 IF 时，可用常数频率种子，算法会在迭代中自适应修正
iniIF = np.array([
    np.full(len(signal), 20.0),
    np.full(len(signal), 80.0),
])
avncmd = AVNCMD(iniIF=iniIF, fs=fs, beta=1e-6)
estIF, estMode, estIA = avncmd.fit_transform(signal)

示例笔记本：

• examples/vmd/svmd.ipynb
• examples/vncmd/avncmd.ipynb
• examples/acmd/acmd.ipynb

2. 大数据 / 长记录：store_history=False 不是在“限制算法”，而是去掉不必要的缓存

完全同意：实用性意味着要能吃下更长的记录，而不是靠截断信号来躲内存。v0.4.45 的改动只是：ADMM 本来就只需要上一轮迭代，不必把 (max_iter, T, K) 的频谱历史全存下来。同一套更新规则，峰值内存从 $O(\text{maxiter} \times T \times K)$ 降到 $O(T \times K)$，不是换了一种分解模型，也不是人为限制输入长度。

对您那种约 $2.65 \times 10^5$ 点的记录，更实用的方法是：

# (A) 仍用经典 VMD，只保留滚动缓冲区
from pysdkit import VMD
vmd = VMD(alpha=2000, K=8, tau=0.0, store_history=False)
IMFs = vmd.fit_transform(signal)

# (B) 短时 VMD：按帧分解，内存大致随窗长而不是整段长度增长
from pysdkit import STVMD
stvmd = STVMD(K=8, alpha=50.0, n_fft=256, dynamic=True)
IMFs = stvmd.fit_transform(signal)

# (C) 流式 / 超长记录：滑动窗 EMD，不必一次性吃完整段
from pysdkit import OnlineEMD
oemd = OnlineEMD(n_extrema=10, max_imfs=8)
IMFs = oemd.fit_transform(signal)

AVNCMD / ACMD 这类时域变分方法并不适合您现在的长度。 它们会建 $O(N)$ 甚至 $O(N^2)$ 的系统（AVNCMD 的字典大约是 $N \times 2KN$ ），26 万点会比原来的 VMD 历史缓存更吃内存。自适应 ≠ 可扩展；长记录请优先 store_history=False 的 VMD、STVMD 或 OnlineEMD。

最后需要强调：每种模态分解算法都有其特殊的归纳偏置，如果对于严肃使用，请尽可能的验证多种不同方法的效果进行合理选择！

[liwenchao36]

有个小建议，可以把分解方法，封装skill，发布，会更好点。 在 2026-08-21 18:43:37，"WhenXuan" ***@***.***> 写道： wwhenxuan left a comment (wwhenxuan/PySDKit#53) Hi @liwenchao36 ，感谢您的反馈。您这两条评论其实对应两个不同的问题，我分开进行介绍。 1. 自适应分解：库里已经有一批，不只是预设 $K$ 的 VMD 经典 VMD 需要事先指定模态数 $K$ 和带宽参数 $\alpha$。如果您希望算法自己根据数据决定模态数、带宽或瞬时频率，PySDKit 里已经有多条路线： | 需求 | 方法 | 说明 | | 不必预先指定 $K$ | EMD / CEEMDAN / FAEMD / ALIF | 按极值/滤波自适应筛分 | | 变分框架、逐个提取、无需 $K$ | SVMD、SJMD | 用停止准则决定何时停 | | 频谱自适应划分 | EWT / EFD | 按 Fourier 边界建滤波器组 | | 自适应带宽 / 瞬时频率（chirp） | AVNCMD、ACMD、BA_ACMD、DD_ACMD、APMD | 跟踪时变 IF，不必手调 (\alpha) | | 非平稳、短时处理 | STVMD（dynamic=True）、STNBMD | 按帧分解，更适合长记录 | 和您之前那条 VMD 记录最接近、又不必指定 $K$ 的，是 SVMD： frompysdkitimportSVMDsvmd=SVMD(max_alpha=20000, tau=0.0, stopc=4, max_modes=10) IMFs=svmd.fit_transform(signal) # 模态数由停止准则决定 如果信号是调频/非线性 chirp，而不是近似平稳窄带模态，可以用 AVNCMD（自适应更新瞬时频率，不再依赖 VNCMD 里那个敏感的 $\alpha$： frompysdkitimportAVNCMDimportnumpyasnp# iniIF: 每行一个模态的初始瞬时频率，形状 (K, N)# 不知道精确 IF 时，可用常数频率种子，算法会在迭代中自适应修正iniIF=np.array([ np.full(len(signal), 20.0), np.full(len(signal), 80.0), ]) avncmd=AVNCMD(iniIF=iniIF, fs=fs, beta=1e-6) estIF, estMode, estIA=avncmd.fit_transform(signal) 示例笔记本： examples/vmd/svmd.ipynb examples/vncmd/avncmd.ipynb examples/acmd/acmd.ipynb 2. 大数据 / 长记录：store_history=False 不是在“限制算法”，而是去掉不必要的缓存 完全同意：实用性意味着要能吃下更长的记录，而不是靠截断信号来躲内存。v0.4.45 的改动只是：ADMM 本来就只需要上一轮迭代，不必把 (max_iter, T, K) 的频谱历史全存下来。同一套更新规则，峰值内存从 $O(\text{maxiter} \times T \times K)$ 降到 $O(T \times K)$，不是换了一种分解模型，也不是人为限制输入长度。 对您那种约 $2.65 \times 10^5$ 点的记录，更实用的方法是： # (A) 仍用经典 VMD，只保留滚动缓冲区frompysdkitimportVMDvmd=VMD(alpha=2000, K=8, tau=0.0, store_history=False) IMFs=vmd.fit_transform(signal) # (B) 短时 VMD：按帧分解，内存大致随窗长而不是整段长度增长frompysdkitimportSTVMDstvmd=STVMD(K=8, alpha=50.0, n_fft=256, dynamic=True) IMFs=stvmd.fit_transform(signal) # (C) 流式 / 超长记录：滑动窗 EMD，不必一次性吃完整段frompysdkitimportOnlineEMDoemd=OnlineEMD(n_extrema=10, max_imfs=8) IMFs=oemd.fit_transform(signal) AVNCMD / ACMD 这类时域变分方法并不适合您现在的长度。 它们会建 $O(N)$ 甚至 $O(N^2)$ 的系统（AVNCMD 的字典大约是 $N \times 2KN$ ），26 万点会比原来的 VMD 历史缓存更吃内存。自适应 ≠ 可扩展；长记录请优先 store_history=False 的 VMD、STVMD 或 OnlineEMD。 最后需要强调：每种模态分解算法都有其特殊的归纳偏置，如果对于严肃使用，请尽可能的验证多种不同方法的效果进行合理选择！ — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>