Console: stop mangling Turkish and other non-ASCII output - #86
Conversation
Closes #83. The report was that the console "sometimes" shows garbled characters. There were two separate causes, and one of them is not intermittent at all. **The JVM was not writing UTF-8.** Without being told otherwise, System.out uses the platform console code page. Measured here on Turkish Windows with Temurin 21, "Çağan" left the JVM as the cp1254 bytes C7 61 F0 61 6E and the checkmark was replaced by a literal '?' before it ever reached us. MSMS then decodes those bytes as UTF-8, so the letters arrive as mojibake - and the '?' is simply gone, no decoder can recover it. Launch args now set stdout.encoding and stderr.encoding, plus the sun.* names JDK 18 and earlier read; both pairs are honoured on 21, and an unrecognised -D is just a property nobody looks at. Deliberately not -Dfile.encoding: that changes the default charset, so on JDK 17 it would also change how plugins read their own config files, and fixing the console must not quietly re-encode someone's data. The flags are prepended, including for the custom preset, so a user who sets their own encoding still wins - the JVM takes the last definition. **Chunk boundaries split characters.** consumeStream did chunk.toString('utf-8') on whatever the pipe handed over. A chunk can end part-way through a multi-byte sequence; toString on a truncated one emits U+FFFD and then mis-decodes the continuation bytes at the head of the next chunk, so a single split character damages two. Every Turkish letter outside ASCII is two bytes, a section sign is two, an emoji is four - it needs a character to straddle a pipe boundary, which is exactly why this half looked intermittent. Line assembly moves into a LineSplitter that owns a StringDecoder, which holds an incomplete tail back until the bytes finishing it arrive. One splitter per stream, never one shared: stdout and stderr are independent byte streams, and a shared decoder would splice one's half-finished character onto the other's next chunk and corrupt both. While it was being extracted, the splitter also grew a flush(), called on exit. A server that dies mid-line never writes the newline that would release its last line - which is usually the line saying why it died. Verified with MSMS_SMOKE (which starts a real server): the fixture line is split at every single byte offset and must survive all of them, with a counter-assertion that a naive decode of the same split does produce U+FFFD, so the loop cannot pass by testing nothing. Also: unterminated line held then released, flush not repeatable, CRLF and multi-line chunks, every preset carrying the encoding flags ahead of -jar, and a user override surviving as the last definition. BRIDGE and EVENTS smoke re-run, since bridge protocol lines share the new path.
'exit' means the process is gone; Node only guarantees the stdio pipes are drained by 'close'. Flushing solely on 'exit' can therefore still drop the final unterminated line - the exact line this change exists to rescue. Probing it here, no data ever arrived between the two events across payloads from 1 KB to 8 MB, so this is not something the machine reproduces. It is a documented hazard all the same, and relying on one platform's pipe timing for the last words of a crashing server is not a trade worth making. Both events now flush. 'exit' first, so the line lands above the "stopped" notice where it reads correctly; 'close' second, catching anything that slipped through - which today would be lost outright, making half a line strictly better than none. Verified that a StringDecoder stays usable after end(), and the smoke now asserts a splitter still decodes correctly after a flush, since that is what makes the double call safe.
Self reviewOne real finding, fixed in 98d6b3d. Flushing only on
|
Closes #83.
The report was that the console sometimes shows garbled characters for
Turkish text. There turned out to be two independent causes, and the larger one
is not intermittent at all — it was hiding behind the one that is.
Cause 1: the JVM was never writing UTF-8
Unless told otherwise, Java's
System.outuses the platform console code page.Measured on this machine (Turkish Windows, Temurin 21):
C7 61 F0 61 6EisÇağanin cp1254. MSMS decodes it as UTF-8, so it arrivesas mojibake. Worse:
✅became a literal?inside the JVM — that characterwas destroyed before it reached us, and no decoder on our side could ever have
brought it back.
Fixed in the launch args:
Four properties for two settings because
stdout.encodingis the name JDK 19+documents and
sun.stdout.encodingis what JDK 18 and earlier read. Verifiedthat both pairs are honoured on 21, and an unrecognised
-Dis just a systemproperty nobody looks at — so all four is safe on any JDK a server might run on.
Deliberately not
-Dfile.encoding=UTF-8. That changes the default charset,which on JDK 17 would also change how plugins read their own config files.
Fixing the console must not quietly re-encode somebody's data.
The flags are prepended, including for the
custompreset, so a user whosets their own
-Dstdout.encoding=...still wins — the JVM takes the lastdefinition of a property on the command line (verified, not assumed).
Cause 2: chunk boundaries split characters
A pipe hands over bytes, not characters. A chunk can end part-way through a
multi-byte sequence, and
toString()on a truncated one emits U+FFFD andmis-decodes the continuation bytes at the head of the next chunk — one split
character damages two.
Every Turkish letter outside ASCII is two bytes, a section sign (colour codes)
is two, an emoji is four. It needs a character to land exactly on a pipe
boundary, which is precisely why this half of the bug looked intermittent while
cause 1 was constant.
Line assembly moves into
src/main/core/lineSplitter.ts, which owns aStringDecoder— that holds an incomplete trailing sequence back until thebytes finishing it arrive. One splitter per stream, never one shared: stdout
and stderr are independent byte streams, and a shared decoder would splice one's
half-finished character onto the other's next chunk and corrupt both.
Bonus: last words of a crashing server
Extracting the splitter made an existing gap obvious, so it also grew a
flush(), called on process exit. A server that dies mid-line never writes thenewline that would have released that line — which is usually the line saying
why it died. It used to be dropped on the floor.
Verification
MSMS_SMOKE(exit 0 — and this gate starts a real Minecraft server, so thelaunch-args change is exercised end to end), plus
MSMS_SMOKE_BRIDGEandMSMS_SMOKE_EVENTSre-run because bridge protocol lines share the new path.The decoding test splits the fixture line at every single byte offset and
requires all of them to survive. It carries a counter-assertion that a naive
decode of the same split does produce U+FFFD — otherwise the loop could pass
by testing a fixture that no longer contains a multi-byte character.
Also asserted: an unterminated line is held and then released by
flush(),flush()is not repeatable, CRLF and multi-line chunks split correctly, everypreset carries the encoding flags ahead of
-jar(after it they would beprogram arguments, not JVM options), and a user-set encoding survives as the
last definition.