Skip to content

npy_inflate fails with "Error inflating stream" on deflate streams larger than ~2 MB #46

Description

@olokobayusuf

npzstringreader / npzreader throw std::runtime_error("Error inflating stream") when reading NPZ entries written by numpy.savez_compressed() once the compressed entry exceeds roughly 2 MB (i.e. spans more than one of npy_inflate's 1 MiB input chunks). Small entries read fine. The root cause seems to be the inner loop condition in npy_inflate in src/zip.cpp, which drops unconsumed input.

Repro

Generate a small and a large archive with numpy:

import numpy as np

rng = np.random.default_rng(0)
np.savez_compressed("small.npz", rng.standard_normal(219_040).astype("float32"))  # ~800 KB, reads OK
np.savez_compressed("large.npz", rng.standard_normal(876_160).astype("float32"))  # ~3.2 MB, fails

Read them back with libnpy:

#include <npy/npy.h>
#include <fstream>
#include <iostream>

int main() {
    std::ifstream input("large.npz", std::ios::binary);
    std::string bytes{std::istreambuf_iterator<char>(input), {}};
    npy::npzstringreader reader(std::move(bytes));
    auto tensor = reader.read<npy::tensor<float>>(reader.keys().front());  // throws
    std::cout << tensor.size() << '\n';
}

small.npz reads correctly butlarge.npz throws Error inflating stream.


I'm happy to open a PR with a fix. Just say the word.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions