latin1 - #7
Merged
Merged
Conversation
Both stream-consuming paths decoded each Buffer chunk independently
(chunk.toString('utf8')), so a multi-byte code point straddling a chunk
boundary decoded to U+FFFD on both sides. Route Buffer chunks through a
persistent StringDecoder: Reader holds one on the instance (flushed via
decoder.end() in _finalize so a genuinely truncated trailing sequence
still surfaces), and read() uses one across its data events. String
chunks bypass the decoder as before.
Existing fixtures never caught this because the conformance corpora are
pure ASCII. Add a boundary matrix splitting 2-, 3-, and 4-byte code
points at every byte position through both Reader and read(), plus a
combined case with one boundary inside an escape sequence and another
inside a code point.
https://claude.ai/code/session_01Mr6xn5zNx83zs5Nmknz5ad
cross-test-rust.js defaulted to nsv 0.0.9 and cross-test-python.js's banner claimed PyPI 0.2.2; current releases are 0.0.12 and 0.2.3 (verified against both registries, and both cross-tests pass against them). Update the CI pins to match so the labels stay truthful. https://claude.ai/code/session_01Mr6xn5zNx83zs5Nmknz5ad
The StringDecoder fix made the utf8 assumption correct at chunk boundaries, but baked a text encoding into a format that is deliberately encoding-agnostic. NSV structure is byte-level (only 0x0A, 0x5C, 0x6E are significant), so mirror the Rust implementation's byte-level core (decode_bytes, the bytes-only streaming Reader): assume no encoding and transport Buffer chunks byte-identically via latin1, which is safe at any chunk boundary by construction. String chunks are parsed as given; decoding text is the caller's job (stream.setEncoding, or re-encode cells from latin1). Tests now assert the byte contract: output independent of chunk boundaries (2-, 3-, 4-byte UTF-8 code points split at every byte position, through both Reader and read()), byte-faithful recovery of the original cells, and a combined escape-plus-code-point split. https://claude.ai/code/session_01Mr6xn5zNx83zs5Nmknz5ad
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.