Skip to content

Console: stop mangling Turkish and other non-ASCII output - #86

Merged
CaYatur merged 2 commits into
mainfrom
fix/console-encoding
Jul 28, 2026
Merged

Console: stop mangling Turkish and other non-ASCII output#86
CaYatur merged 2 commits into
mainfrom
fix/console-encoding

Conversation

@CaYatur

@CaYatur CaYatur commented Jul 28, 2026

Copy link
Copy Markdown
Owner

Closes #83.

The report was that the console sometimes shows garbled characters for
Turkish text. There turned out to be two independent causes, and the larger one
is not intermittent at all — it was hiding behind the one that is.

Cause 1: the JVM was never writing UTF-8

Unless told otherwise, Java's System.out uses the platform console code page.
Measured on this machine (Turkish Windows, Temurin 21):

$ java Enc                       # prints  TR: Çağan ğüşiöç §a ✅
0000000   T   R   :     307   a 360   a   n     360 374 376   i 366 347
0000020     247   a       ?  \r  \n

$ java -Dstdout.encoding=UTF-8 Enc
0000000   T   R   :     303 207   a 304 237   a   n     304 237 303 274
0000020 305 237   i 303 266 303 247     302 247   a     342 234 205  \r

C7 61 F0 61 6E is Çağan in cp1254. MSMS decodes it as UTF-8, so it arrives
as mojibake. Worse: became a literal ? inside the JVM — that character
was destroyed before it reached us, and no decoder on our side could ever have
brought it back.

Fixed in the launch args:

-Dstdout.encoding=UTF-8  -Dstderr.encoding=UTF-8
-Dsun.stdout.encoding=UTF-8  -Dsun.stderr.encoding=UTF-8

Four properties for two settings because stdout.encoding is the name JDK 19+
documents and sun.stdout.encoding is what JDK 18 and earlier read. Verified
that both pairs are honoured on 21, and an unrecognised -D is just a system
property nobody looks at — so all four is safe on any JDK a server might run on.

Deliberately not -Dfile.encoding=UTF-8. That changes the default charset,
which on JDK 17 would also change how plugins read their own config files.
Fixing the console must not quietly re-encode somebody's data.

The flags are prepended, including for the custom preset, so a user who
sets their own -Dstdout.encoding=... still wins — the JVM takes the last
definition of a property on the command line (verified, not assumed).

Cause 2: chunk boundaries split characters

mp[key] += chunk.toString('utf-8')

A pipe hands over bytes, not characters. A chunk can end part-way through a
multi-byte sequence, and toString() on a truncated one emits U+FFFD and
mis-decodes the continuation bytes at the head of the next chunk — one split
character damages two.

Every Turkish letter outside ASCII is two bytes, a section sign (colour codes)
is two, an emoji is four. It needs a character to land exactly on a pipe
boundary, which is precisely why this half of the bug looked intermittent while
cause 1 was constant.

Line assembly moves into src/main/core/lineSplitter.ts, which owns a
StringDecoder — that holds an incomplete trailing sequence back until the
bytes finishing it arrive. One splitter per stream, never one shared: stdout
and stderr are independent byte streams, and a shared decoder would splice one's
half-finished character onto the other's next chunk and corrupt both.

Bonus: last words of a crashing server

Extracting the splitter made an existing gap obvious, so it also grew a
flush(), called on process exit. A server that dies mid-line never writes the
newline that would have released that line — which is usually the line saying
why it died. It used to be dropped on the floor.

Verification

MSMS_SMOKE (exit 0 — and this gate starts a real Minecraft server, so the
launch-args change is exercised end to end), plus MSMS_SMOKE_BRIDGE and
MSMS_SMOKE_EVENTS re-run because bridge protocol lines share the new path.

The decoding test splits the fixture line at every single byte offset and
requires all of them to survive. It carries a counter-assertion that a naive
decode of the same split does produce U+FFFD — otherwise the loop could pass
by testing a fixture that no longer contains a multi-byte character.

Also asserted: an unterminated line is held and then released by flush(),
flush() is not repeatable, CRLF and multi-line chunks split correctly, every
preset carries the encoding flags ahead of -jar (after it they would be
program arguments, not JVM options), and a user-set encoding survives as the
last definition.

Closes #83.

The report was that the console "sometimes" shows garbled characters. There
were two separate causes, and one of them is not intermittent at all.

**The JVM was not writing UTF-8.** Without being told otherwise, System.out
uses the platform console code page. Measured here on Turkish Windows with
Temurin 21, "Çağan" left the JVM as the cp1254 bytes C7 61 F0 61 6E and the
checkmark was replaced by a literal '?' before it ever reached us. MSMS then
decodes those bytes as UTF-8, so the letters arrive as mojibake - and the '?'
is simply gone, no decoder can recover it. Launch args now set
stdout.encoding and stderr.encoding, plus the sun.* names JDK 18 and earlier
read; both pairs are honoured on 21, and an unrecognised -D is just a
property nobody looks at. Deliberately not -Dfile.encoding: that changes the
default charset, so on JDK 17 it would also change how plugins read their own
config files, and fixing the console must not quietly re-encode someone's
data. The flags are prepended, including for the custom preset, so a user who
sets their own encoding still wins - the JVM takes the last definition.

**Chunk boundaries split characters.** consumeStream did
chunk.toString('utf-8') on whatever the pipe handed over. A chunk can end
part-way through a multi-byte sequence; toString on a truncated one emits
U+FFFD and then mis-decodes the continuation bytes at the head of the next
chunk, so a single split character damages two. Every Turkish letter outside
ASCII is two bytes, a section sign is two, an emoji is four - it needs a
character to straddle a pipe boundary, which is exactly why this half looked
intermittent.

Line assembly moves into a LineSplitter that owns a StringDecoder, which
holds an incomplete tail back until the bytes finishing it arrive. One
splitter per stream, never one shared: stdout and stderr are independent byte
streams, and a shared decoder would splice one's half-finished character onto
the other's next chunk and corrupt both.

While it was being extracted, the splitter also grew a flush(), called on
exit. A server that dies mid-line never writes the newline that would release
its last line - which is usually the line saying why it died.

Verified with MSMS_SMOKE (which starts a real server): the fixture line is
split at every single byte offset and must survive all of them, with a
counter-assertion that a naive decode of the same split does produce U+FFFD,
so the loop cannot pass by testing nothing. Also: unterminated line held then
released, flush not repeatable, CRLF and multi-line chunks, every preset
carrying the encoding flags ahead of -jar, and a user override surviving as
the last definition. BRIDGE and EVENTS smoke re-run, since bridge protocol
lines share the new path.
Copilot AI review requested due to automatic review settings July 28, 2026 02:01
@CaYatur CaYatur added bug Something isn't working area:console Server console output labels Jul 28, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

'exit' means the process is gone; Node only guarantees the stdio pipes are
drained by 'close'. Flushing solely on 'exit' can therefore still drop the
final unterminated line - the exact line this change exists to rescue.

Probing it here, no data ever arrived between the two events across payloads
from 1 KB to 8 MB, so this is not something the machine reproduces. It is a
documented hazard all the same, and relying on one platform's pipe timing for
the last words of a crashing server is not a trade worth making.

Both events now flush. 'exit' first, so the line lands above the "stopped"
notice where it reads correctly; 'close' second, catching anything that
slipped through - which today would be lost outright, making half a line
strictly better than none. Verified that a StringDecoder stays usable after
end(), and the smoke now asserts a splitter still decodes correctly after a
flush, since that is what makes the double call safe.
@CaYatur

CaYatur commented Jul 28, 2026

Copy link
Copy Markdown
Owner Author

Self review

One real finding, fixed in 98d6b3d.

Flushing only on 'exit' can still drop the line it exists to rescue

The new flushStreams() was hooked to child.on('exit'). But 'exit' means
the process is gone; Node only guarantees the stdio pipes are drained by
'close'. Anything still in flight between the two events would be flushed
before it arrived and then dropped — which is precisely the last unterminated
line this change was added to save.

Probed it rather than guessing:

size 1000     bytes after exit: 0
size 100000   bytes after exit: 0
size 500000   bytes after exit: 0
size 2000000  bytes after exit: 0
size 8000000  bytes after exit: 0

So it does not reproduce here at any payload size — Node's reader drains the
pipe eagerly on this platform. It is a documented hazard all the same, and
leaning on one platform's pipe timing for the last words of a crashing server
is not a trade worth making.

Both events now flush. 'exit' first, so the line lands above the "stopped"
notice where it reads correctly. 'close' second, catching anything that
slipped through — which today would be lost outright, so half a line is strictly
better than none.

That makes flushStreams() a double call, which only works if a StringDecoder
survives its own end(). Verified:

write half: ""        end: "�"
write after end (split again): "Çağan"     end again: ""

The smoke now asserts a splitter still decodes a split character correctly after
a flush, so nothing can quietly break that property.

Also reviewed, deliberately left alone

  • 'error' does not flush. That path is a spawn failure — there is no
    output to flush.
  • -Dfile.encoding not set. Tempting, and it would also fix the console, but
    it changes the default charset, so on JDK 17 it would change how plugins read
    their own config files. Explicitly out of scope: fixing the console must not
    re-encode somebody's data.
  • Four -D properties for two settings. stdout.encoding is the JDK 19+
    name, sun.stdout.encoding the JDK 18-and-earlier one. Verified both pairs are
    honoured on Temurin 21, and an unrecognised -D is inert.
  • custom preset now gets the flags too, contradicting "used verbatim". They
    are prepended, and the JVM takes the last definition of a property — verified
    by passing -Dstdout.encoding=UTF-8 -Dstdout.encoding=windows-1254 and getting
    cp1254 bytes back. So a user's own setting still wins, and a custom command line
    is not the one place the console silently mangles Turkish.
  • The stopped placeholder ManagedProcess allocates two splitters it never
    uses. Two tiny objects; not worth making the field optional and adding null
    checks on the hot path.

Verification after the fix

MSMS_SMOKE exit 0 (this gate starts a real server), MSMS_SMOKE_BRIDGE and
MSMS_SMOKE_EVENTS exit 0.

@CaYatur
CaYatur merged commit d041668 into main Jul 28, 2026
1 check passed
@CaYatur
CaYatur deleted the fix/console-encoding branch July 28, 2026 02:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:console Server console output bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Console shows replacement characters for Turkish and other non-ASCII output

2 participants