[BUG] Trajectory filter for long trajectories.

[ChiCheng45]

Description of the error
Currently the trajectory filter loads the entire trajectory before applying the filtering. This means that for some machine with lower amounts of RAM or very long trajectories this job might cause some out of memory issues.

I think there might be way around this since what we want to do is

$$\mathrm{FT}^{-1}[r(\omega) g(\omega)] = r(x) * g(x) = r'(x)$$

where $r'(x)$ is the filtered positions, $r(\omega)$ is the FT of the positions, $g(\omega)$ is the frequency filter, and $g(x)$ is the inverse FT of the frequency filter.

If we use the convolution of $r(x)$ and $g(x)$ to determine $r'(x)$ then I think we can use one of the following methods

https://en.wikipedia.org/wiki/Overlap%E2%80%93add_method
https://en.wikipedia.org/wiki/Overlap%E2%80%93save_method

so we should be able to calculate the convolution of $r(x)$ and $g(x)$ and therefore $r'(x)$ in parts maybe something like 200 frames at a time or something.

[RobBuchananCompPhys]

I think this is basically true, but I am not quite sure if I understand how we turn this into an algorithm that is more efficient than what Python already does when it applies the filter. It's probably already something like the shift-and-sum method.

I wonder if the most efficient way to do this would be to take the framework we use for PositionPowerSpectrum and for each atomic contribution we use the part of the convolution theorem in the square brackets above, and pointwise multiply the spectra with the filter frequencies, then IFT each back to the time domain. I think this may throw away phase information, but otherwise could be a way of doing the filtering in the (preferable) way other MDANSE jobs handle large trajectories.

[ChiCheng45]

I think this is basically true, but I am not quite sure if I understand how we turn this into an algorithm that is more efficient than what Python already does when it applies the filter. It's probably already something like the shift-and-sum method.

I wonder if the most efficient way to do this would be to take the framework we use for PositionPowerSpectrum and for each atomic contribution we use the part of the convolution theorem in the square brackets above, and pointwise multiply the spectra with the filter frequencies, then IFT each back to the time domain. I think this may throw away phase information, but otherwise could be a way of doing the filtering in the (preferable) way other MDANSE jobs handle large trajectories.

The proposal above is to reduced the memory usage by the trajectory filter since it currently loads the entire MD trajectory into RAM to then do the filter. With the above methods your could probably load up 200 frames at a time or something. I'm not suggesting that it would be faster just that it wouldn't take all your system memory up since you'd do the convolution in chunks.

[ChiCheng45]

#1087 (comment)