Evaluate Latin-1 fallback for command-output decode paths (shell/powershell/winrm) instead of surrogateescape

[NavidSassan]

Context

PR #255 fixed a decode crash in url.fetch() by falling back to Latin-1 when a remote host sends non-UTF-8 content without declaring a charset. During that work we established, empirically, that errors='surrogateescape' is the wrong tool whenever the decoded text is later re-encoded for output (which every monitoring plugin does when it prints its result to stdout).

The concern

Three command-output decode paths still decode subprocess output as UTF-8 with surrogateescape:

• shell.py (Unix branch): txt.to_text(stdout, encoding='utf-8', errors='surrogateescape')
• powershell.py: txt.to_text(result.stdout) (defaults to errors=None -> surrogateescape)
• winrm.py: txt.to_text(stdout) (defaults to surrogateescape)

surrogateescape never fails at decode, but it maps invalid bytes to lone surrogates (U+DC80-U+DCFF). Those raise UnicodeEncodeError the moment the plugin re-encodes the message for stdout. So a non-UTF-8 byte in command output does not crash at decode time; it crashes later at output, which is harder to diagnose.

Reproduction through the real call path

lib.txt.to_text() for the decode, lib.base.oao() for the output (run from the directory that holds the lib package so import lib.txt resolves):

python3 -c "import lib.txt, lib.base; lib.base.oao(lib.txt.to_text(b'M\xfcller'))"
# UnicodeEncodeError: 'utf-8' codec can't encode character '\udcfc' in position 1: surrogates not allowed
# traceback lands on the print() inside oao()

0xfc (u-umlaut in Latin-1 / CP1252) decodes to the lone surrogate \udcfc without error, then blows up at the stdout re-encode inside oao(). Latin-1 handles the same byte cleanly:

python3 -c "print(b'M\xfcller'.decode('latin-1').encode('utf-8'))"
# b'M\xc3\xbcller'   (valid UTF-8 for "Mueller" with u-umlaut)

Comparison across the byte values seen in the monitoring-plugins decode issues (0xb0, 0x81, 0xc2, 0xfc), decode followed by a UTF-8 output re-encode:

bytes	latin-1 -> output	surrogateescape -> output
>m\x81ller	OK	UnicodeEncodeError
M\xfcller	valid UTF-8, OK	UnicodeEncodeError
\xc2 x	OK	UnicodeEncodeError

Latin-1 maps every byte 1:1 to a real Unicode scalar, so it survives both the JSON round-trip and the stdout re-encode.

Note: txt.to_bytes is not a valid stand-in for the output step in this repro. It also maps errors=None -> surrogateescape and would round-trip 0xfc back to bytes cleanly, hiding the bug. The crash only surfaces through a strict encoder, which is what print/stdout is (default strict under a UTF-8 locale; PYTHONIOENCODING=utf-8:surrogateescape would mask it).

Proposal

Evaluate switching the Unix shell.py branch and the powershell.py / winrm.py decode calls from surrogateescape to a Latin-1 fallback (decode with the expected codec, fall back to Latin-1 on UnicodeDecodeError), consistent with how url.py now handles undeclared-charset web content.

Out of scope / already correct

The Windows shell.py branch is already correct and must stay as is: it detects the real OEM/console code page via _windows_output_encoding() and decodes with that (errors='replace' as a net). That is the right pattern (correct codec first) and fixed Linuxfabrik/monitoring-plugins#681 and friends.

Notes

• This changes command-output decoding behavior, so each module needs its own test coverage before merging.
• Kept out of PR fix(url): fall back to Latin-1 when a response has no declared charset #255 deliberately to keep that fix scoped to the cometsystem regression.