Repository navigation
-STDIN skips a UTF-8 BOM before the first path (4.2.46) - #802
Merged
Merged
Conversation
This was referenced Oct 26, 2026
A .NET program that starts nlp with stdin redirected (C#, or PowerShell through System.Diagnostics.Process) writes Console.InputEncoding's UTF-8 BOM into the pipe as soon as the process starts, before the caller writes anything. -STDIN read the first line as "C:\...\a.txt", reported [stream: no such input file: a.txt] and went on with the rest, so the first input was silently not analyzed. Found while streaming 2,509 pieces of a book through -STDIN from PowerShell: every run was one piece short. analyzeStream now drops a UTF-8 BOM at the start of the first line only. CI: the -STDIN step on Windows and Linux (macOS has no -STDIN step) now also sends the three paths with a BOM in front and expects all three analyzed. On master it analyzes two. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ddehilster
force-pushed
the
fix/stdin-bom
branch
from
October 26, 2026 19:55
3acbbdc to
b06fe83
Compare
ddehilster
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a .NET program starts
nlp -STDINwith stdin redirected,nlpsilently skips the first input file. That includes C# programs and PowerShell scripts that go throughSystem.Diagnostics.Process. .NET writes the UTF-8 byte-order mark (EF BB BF) ofConsole.InputEncodinginto the pipe as soon as the process starts, before the caller writes anything.-STDINthen read the first line as\uFEFFC:\...\a.txtand reportedIt carried on with the rest, so the run ended one input short.
I found this while streaming 2,509 pieces of a book through
-STDINfrom PowerShell for the zoning experiment: every run was one piece short.Fix
analyzeStreamdrops a UTF-8 BOM at the start of the first line only. A BOM can only appear at the start of a stream.Tests
-STDINCI step on Windows and Linux now also sendsa.txt b.txt c.txtwith a BOM in front, and expects all three analyzed with the right first and last flags. macOS has no-STDINstep.a.txtis reported missing and only two inputs are analyzed.Version 4.2.46 (rebased onto 4.2.45, zone mode). #803 follows as 4.2.47; I plan to release both under one tag, because of the PyPI size limit.
🤖 Generated with Claude Code