ZIPMAT BUG: "zmat" used up all system memory

[xingchenli1996]

% test code:

C = zeros(100000,1);

for i = 1:100000

    if isPrimeNumber(i) ~= 1

        tmp = randi(256, 1, 16384); tmp = tmp - 1; tmp = uint8(tmp);

    elseif isPrimeNumber(i) == 1

        tmp = repmat(128, 1, 16384); tmp = tmp - 1; tmp = uint8(tmp);

    end
    
    C(i) = size(zipmat((uint8(tmp)), -9, 'lzma', 'nthread', 1), 2);

    disp([num2str(C(i)), ';         ', num2str(i), ' / ', num2str(100000)])

end

The usage is to test whether a big and complex data contained some lines that can be compressed to a smaller size than its original size.

And, after the subfunction run succesfully, the temporary variable cleared, but the system memory occupied by ZMAT mex does not free automatically, and to my surprise, continue growing. except a quit and restart of MATLAB Application (which is absolutely impossible during script running)

When this code is running, system will use up at once.

[xingchenli1996]

function isPrime = isPrimeNumber(n)
    if n <= 1
        isPrime = false;
    elseif n <= 3
        isPrime = true;
    elseif mod(n,2) == 0 || mod(n,3) == 0
        isPrime = false;
    else
        i = 5;
        while i * i <= n
            if mod(n,i) == 0 || mod(n, i + 2) == 0
                isPrime = false;
                return;
            end
            i = i + 6;
        end
        isPrime = true;
    end
end

[fangq]

@xingchenli1996, are you using the precompiled version (directly download and addpath) or the github CI build?

https://mcx.space/nightly/github/zmat-allinone-github-latest.zip

if you did not use the CI build, can you give it a try and see if this memory leakage problem still appear?

[xingchenli1996]

@xingchenli1996, are you using the precompiled version (directly download and addpath) or the github CI build?

https://mcx.space/nightly/github/zmat-allinone-github-latest.zip

if you did not use the CI build, can you give it a try and see if this memory leakage problem still appear?

Yes，I am currently using one of the precompiled version inside Git master's [Private] folder, but memory overload failure occurs. However, because of a Github limit, I could not upload the video directly to Github. I could only upload the video to netdisk in my country, China

链接: https://pan.baidu.com/s/1Tt0sInQkK4XZ2Owj5jvfOA?pwd=1234 提取码: 1234
--来自百度网盘超级会员v7的分享

[fangq]

@xingchenli1996, thanks for reporting this issue. I was able to reproduce it.

the problem was caused by known memory leakage for eazylzma -- a library I used for the lzma compression/decompression. This memory leakage was reported in lloyd/easylzma#4 and lloyd/easylzma#7, but the upstream code was no longer maintained.

After merging the changes from the above pull requests, I was able to see that the memory leakage has disappeared.

by the way, matlab has a built-in function isprime, there is no need to write your own.

let me know if this issue is also fixed on your side.

[xingchenli1996]

@xingchenli1996, thanks for reporting this issue. I was able to reproduce it.

the problem was caused by known memory leakage for eazylzma -- a library I used for the lzma compression/decompression. This memory leakage was reported in lloyd/easylzma#4 and lloyd/easylzma#7, but the upstream code was no longer maintained.

After merging the changes from the above pull requests, I was able to see that the memory leakage has disappeared.

by the way, matlab has a built-in function isprime, there is no need to write your own.

let me know if this issue is also fixed on your side.

The Memory failure disappeared now. Thanks very much for your work. Now my project that automatically examine whether a complex array contains a block that can be compressed, could run as expected. Thank you again and you are a genius.