You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Stream OOXML decryption to avoid buffering the whole payload - #102
OOXMLFile.decrypt() materialized the entire decrypted OOXML package in memory and copied it several times. Both ECMA376Agile.decrypt() and ECMA376Standard.decrypt() accumulated output in a BytesIO and returned a second full copy, and ooxml.py copied the result again into a BytesIO just to run zipfile.is_zipfile(). Peak RSS was ~4–5× the input size.
Changes
Agile (msoffcrypto/method/ecma376_agile.py): extracted the existing 4096-byte segment loop into a streaming generator ECMA376Agile.decrypt_stream(...). decrypt(...) is now a thin wrapper returning b"".join(...), preserving the old API.
Standard (msoffcrypto/method/ecma376_standard.py): added ECMA376Standard.decrypt_stream(...), which feeds AES-ECB incrementally in chunks and yields exactly totalSize plaintext bytes. decrypt(...) remains as a wrapper.
Validation (msoffcrypto/format/ooxml.py): replaced zipfile.is_zipfile(io.BytesIO(obuf)) with a tail-only end-of-central-directory check (_is_zipfile_tail) that keeps just the last 65,557 bytes while streaming, so the payload is never copied for validation. The same InvalidKeyError is raised on failure.
OLE stream read (msoffcrypto/format/ooxml.py): olefile.openstream() eagerly loads the whole EncryptedPackage stream into a BytesIO (see olefile's own FIXME), which dominated the remaining memory. Added _OleStreamReader to follow the FAT chain and read one sector at a time, with a fallback to openstream() for small/mini-FAT streams.
Backward compatibility
OOXMLFile.decrypt(outfile) and the top-level OfficeFile.decrypt(outfile) keep the same signatures.
ECMA376Agile.decrypt(...) and ECMA376Standard.decrypt(...) still return bytes.
Tests
Added tests/test_ooxml_streaming.py: verifies the streaming output is byte-identical to the buffered decrypt(), works with a non-seekable output destination, and that a wrong password still raises InvalidKeyError. No large binary fixtures are committed.
Full suite passes (pytest).
Memory
Measured with /usr/bin/time -l on a 1.79 GB agile-encrypted .ppsx:
before: ~4.95 GiB peak RSS
after: ~0.88 GiB peak RSS
The remaining ~0.88 GiB comes from olefile.OleFileIO.__init__ parsing the OLE structure (its loadfat builds the FAT with array + array in a loop), which runs before any decryption and is outside this patch. The streaming decrypt path itself now stays at a small constant buffer plus one 4096-byte segment.
Out of scope
verify_integrity=True (the agile HMAC path) still reads the full stream once, because it needs the entire payload to compute the HMAC. It is opt-in and left unchanged.
Thank you for your valuable contribution!
I'd like to merge this, but since there is some overlap with the related PR in olefile, I'd prefer to wait a little while to see how that is handled there.
If it looks like that may take some time, I'd be happy to move forward with addressing it here instead.
I'm also considering keeping the existing implementation and adding the streaming implementation separately, rather than replacing the current one.
Thank you for your valuable contribution! I'd like to merge this, but since there is some overlap with the related PR in olefile, I'd prefer to wait a little while to see how that is handled there. If it looks like that may take some time, I'd be happy to move forward with addressing it here instead. I'm also considering keeping the existing implementation and adding the streaming implementation separately, rather than replacing the current one.
Thank you for the thoughtful review!
I've opened #103, which addresses the overlap concern directly: it's the same streaming implementation, but without the _OleStreamReader workaround. It relies on the lazy OleStream from the related olefile PR (decalage2/olefile#177) and reads EncryptedPackage directly via openstream(), so there's no duplicated OLE-stream handling on the msoffcrypto side anymore.
I'm happy to wait and see how the olefile maintainer responds — if that stalls, we can always switch back to #102.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #101
Problem
OOXMLFile.decrypt()materialized the entire decrypted OOXML package in memory and copied it several times. BothECMA376Agile.decrypt()andECMA376Standard.decrypt()accumulated output in aBytesIOand returned a second full copy, andooxml.pycopied the result again into aBytesIOjust to runzipfile.is_zipfile(). Peak RSS was ~4–5× the input size.Changes
msoffcrypto/method/ecma376_agile.py): extracted the existing 4096-byte segment loop into a streaming generatorECMA376Agile.decrypt_stream(...).decrypt(...)is now a thin wrapper returningb"".join(...), preserving the old API.msoffcrypto/method/ecma376_standard.py): addedECMA376Standard.decrypt_stream(...), which feeds AES-ECB incrementally in chunks and yields exactlytotalSizeplaintext bytes.decrypt(...)remains as a wrapper.msoffcrypto/format/ooxml.py): replacedzipfile.is_zipfile(io.BytesIO(obuf))with a tail-only end-of-central-directory check (_is_zipfile_tail) that keeps just the last 65,557 bytes while streaming, so the payload is never copied for validation. The sameInvalidKeyErroris raised on failure.msoffcrypto/format/ooxml.py):olefile.openstream()eagerly loads the wholeEncryptedPackagestream into aBytesIO(see olefile's ownFIXME), which dominated the remaining memory. Added_OleStreamReaderto follow the FAT chain and read one sector at a time, with a fallback toopenstream()for small/mini-FAT streams.Backward compatibility
OOXMLFile.decrypt(outfile)and the top-levelOfficeFile.decrypt(outfile)keep the same signatures.ECMA376Agile.decrypt(...)andECMA376Standard.decrypt(...)still returnbytes.Tests
tests/test_ooxml_streaming.py: verifies the streaming output is byte-identical to the buffereddecrypt(), works with a non-seekable output destination, and that a wrong password still raisesInvalidKeyError. No large binary fixtures are committed.pytest).Memory
Measured with
/usr/bin/time -lon a 1.79 GB agile-encrypted.ppsx:The remaining ~0.88 GiB comes from
olefile.OleFileIO.__init__parsing the OLE structure (itsloadfatbuilds the FAT witharray + arrayin a loop), which runs before any decryption and is outside this patch. The streaming decrypt path itself now stays at a small constant buffer plus one 4096-byte segment.Out of scope
verify_integrity=True(the agile HMAC path) still reads the full stream once, because it needs the entire payload to compute the HMAC. It is opt-in and left unchanged.