fix #1190 support to display packet content as UTF-8 - #1382
Conversation
992ccb1 to
6ee3b82
Compare
lotabout
changed the title
2ecc6b2 to
354fa81
Compare
lotabout
changed the title
|
I tried this on Windows using MSVC and clang-cl. But there is no print-ascii.c(118,11): error: call to undeclared function 'wcwidth'; ISO C99 and later do not support implicit function declarations
[-Wimplicit-function-declaration]
118 | int w = wcwidth(wc);
| ^ |
|
How did it build in Appveyor then? |
Since |
9c0622f to
a5fa5a2
Compare
|
@gvanem Could you please help me give it another try on Windows? I’ve added Markus Kuhn’s implementation as a replacement on Windows. |
6629672 to
a13a471
Compare
|
@gvanem Please help to try again with latest code, it turns out that locale should be set correctly for （left: tcpdump, right: cat. Compiled on Windows 11 with VS 2022, Shell: PowerShell with 
|
a448b15 to
d991fd2
Compare
|
tcpdump CI is failing because of my recent changes in libpcap. Please wait until this is fixed. |
@infrastation Got it, Thanks~ |
a5fa5a2 to
a13a471
Compare
|
Work very well now! |
870f465 to
b5f77ca
Compare
|
@fxlb Hi! Sorry for the ping (I saw you're very active here). This is my first PR to this project — is there anything else I should do to help get it merged, apart from passing the CI checks? |
|
Why is nobody besides me and @lotabout interested in merging this? |
|
|
||
|
|
||
| /* | ||
| * The blow is_utf8_printable is taken from ngrep |
There was a problem hiding this comment.
Presumably that should be "The below is_utf8_printable is taken from ngrep".
| } | ||
|
|
||
| /* Check if the wide character is printable */ | ||
| #if defined(_WIN32) || defined(_WIN64) |
There was a problem hiding this comment.
Are there any compilers for Windows that define _WIN64, but don't define _WIN32, on "modern" Windows? ("modern" here means "not 16-bit Windows".)
| utf8_len = ndo->ndo_utf8 ? is_utf8_printable(cp, length, NULL) : 0; | ||
|
|
||
| if (utf8_len > 0) { | ||
| /* Valid printable UTF-8 character */ |
There was a problem hiding this comment.
What if utf8_len is greater than length - i.e., you have a partial UTF-8 character?
|
The title of the pull request and commit message should probably be just "Add support to display packet content as UTF-8", with "Fix #1190" in the body of the commit message. |
|
|
I agree that for this new feature it is more important to implement it safely than quickly. In particular, it should be impossible to weaponise it as is sometimes done using ANSI escape codes, see CVE-2025-46394 for one example. So this requires a bit more attention before it is declared safe and ready. |
nd_utf8_glyph() reads one displayable "glyph" from the packet buffer: a code point encoded as one of the well-formed byte sequences of Table 3-7 of the Unicode Standard (no overlong forms, surrogates or values above U+10FFFF), that is a letter, a spacing mark, a digit, a punctuation character or a symbol, optionally followed by up to two combining marks. Everything else (malformed or truncated sequences, controls, format characters such as bidirectional controls and zero width characters, separators other than U+0020, private use, unassigned and Default_Ignorable code points) is rejected and left to the caller to print as ".". A glyph is always one or two terminal columns wide, so the output can carry neither terminal control sequences nor invisible text. The classification comes from tables generated by gen-unicode-tables.py from a pinned Unicode version (currently 16.0.0) rather than from the C library, so the result does not depend on the locale, on the platform or on wchar_t, and the same expected output can be used on every platform in the test suite. Every byte is read with GET_U_1(). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
--charset=utf-8 makes ascii_print() and hex_and_ascii_print() print the glyphs accepted by nd_utf8_glyph() as is; the default, --charset=ascii, keeps the current byte-for-byte output. In the -X text column each packet byte still corresponds to exactly one column: a glyph is printed at the position of its first byte, followed by one filler space per additional byte. Fillers of a glyph that spans two lines continue on the next line. A two-column glyph that starts at the last position of a line is printed at the start of the next line instead; a two-column code point is at least three bytes long in UTF-8, so it always fits there. Since a filler always separates two multi-byte glyphs, terminals cannot merge adjacent code points (ZWJ sequences, emoji modifiers, flags) in the -X column, and the rendered width equals the computed width. The hex column is unchanged; the text column of a line is built in a buffer whose size follows from the per-position bound of one glyph of ND_UTF8_GLYPH_MAX_BYTES bytes plus one glyph carried over from the previous line. No setlocale() is needed; the terminal has to be set to UTF-8. Fix the-tcpdump-group#1190 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
utf8.pcap holds UDP payloads covering: 2-, 3- and 4-byte characters; combining marks (NFD, Thai, Devanagari, a third stacked mark and a mark without a base); ZWJ, emoji modifier, flag, keycap and tag sequences; content that must never be emitted (C1 CSI, ESC, RLO, LRI, U+2028, NBSP, ideographic space, ZWSP, BOM, soft hyphen, private use, noncharacter, unassigned); malformed UTF-8 (overlong, surrogate, above U+10FFFF, F5 lead byte, lone continuation byte, sequences truncated at the end of the packet); every -X line boundary case; and a capture truncated in the middle of a character. The -ascii variants check that the default output is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
b5f77ca to
383c996
Compare
5 participants
This PR adds a new --utf8 option to enable UTF-8 character support.
When UTF-8 support is enabled, tcpdump will detect and display UTF-8 characters in the payload as-is when using the
-Aoption.Note that in
-Xmode, if a multi-byte character spans across two lines, it will appear on the first line, and a spaces will be padded on the next line.Tests
-Aand-x, ensuring their MD5 checksums remain identical without --utf8.utf8.pcap.zip