don't use tb_wcswidth to determine grapheme width.

[Old-Farmer]

I have found that this library use inner tb_wcswidth to determine grapheme width. This is not valid. For example, 👩‍👩‍👧 is mostly only 2 col width in modern terminal， but tb_wcswidth generate 6.

May be you can disable the width checking for grapheme? or just provide a mostly right one?

[txgk]

There are 2 ways to use tb_wcswidth function (or any other wide character function in termbox2): 1) built-in and 2) libc-provided. By default built-in functions are used and I also have found them to be not 100% accurate in some cases. To use libc-provided wide character functions, you must define TB_OPT_LIBC_WCHAR and CJK/emojis/etc will have properly calculated column width.

[txgk]

@adsr since built-in wide character facilities can be incorrect in some cases and cause troubles like this, perhaps it's worth to make libc-provided wide character functions the default approach to handling Unicode?

In most header-only libraries, disabling standard library is usually opt-in (not enabled by default) since most developers go with libc and only a few require no-libc compilation.

[Old-Farmer]

It‘ not about wcwidth, It's the way to calculate a grapheme width. It's ambiguous and don't have any specification. Current implmentation of width calc of a grepheme is to add them up, it's not that correct for some situation, like 👩‍👩‍👧，or 👩‍🚀。

[Old-Farmer]

See https://mitchellh.com/writing/grapheme-clusters-in-terminals

[Old-Farmer]

Maybe I think this library can provide a function to let users register a width calc function?

[Old-Farmer]

Check my currenct tmp solution if you like. master...Old-Farmer:termbox2:dev

[adsr]

Thank you for calling this out @Old-Farmer. Adding up the wcwidth of individual codepoints in a grapheme cluster is indeed incorrect. As a default implementation it's probably more correct to use the first codepoint's width -- though I'd bet that's not always correct.

As an alternative to a function pointer, we might consider adding a width field to the tb_cell struct and a setter function. Not saying I prefer that -- just mapping solution space. I can't think of anything else at the moment.

Do you agree with using the first codepoint's width as a default implementation?

@adsr since built-in wide character facilities can be incorrect in some cases and cause troubles like this, perhaps it's worth to make libc-provided wide character functions the default approach to handling Unicode?

libc doesn't have any concept of grapheme clusters (at least glibc and musl) so I don't think that would help here.

@txgk If you've noticed other issues with the built-in Unicode methods I'd be very curious to hear about those.

[Old-Farmer]

@adsr using first codepoint width to determine the egc width is a great default, but need to take care of VS15 and VS16. I have a found a blog of how kitty process text https://sw.kovidgoyal.net/kitty/text-sizing-protocol/?utm_source=chatgpt.com#the-algorithm-for-splitting-text-into-cells , really a good reference.

By the way, not all terminals and applications(like tmux), use the same algorithms to process text. Even modern terminals, like Alacritty, doesn't recognize a lot of egc. So a user-assiged width per cell maybe helpful for users to tweak the rendering behavior by different environment. user-assiged width per cell also makes rendering more predictable from users' perspective. So it will also be better to expose the egc width calc funtion if users uses the default.

[txgk]

It‘ not about wcwidth, It's the way to calculate a grapheme width. It's ambiguous and don't have any specification. Current implmentation of width calc of a grepheme is to add them up, it's not that correct for some situation, like 👩‍👩‍👧，or 👩‍🚀。
See https://mitchellh.com/writing/grapheme-clusters-in-terminals

@Old-Farmer oh, I didn't get the issue initially. The article you provided has nice explanatioon and now I understand the problem, thanks :)

@txgk If you've noticed other issues with the built-in Unicode methods I'd be very curious to hear about those.

@adsr I've tried to disable TB_OPT_LIBC_WCHAR now and to my surprise couldn't catch any strange behaviors at the moment, so consider my earlier complaint as noise. I'll keep using termbox2 with TB_OPT_LIBC_WCHAR disabled and shout if anything spooky does show up. Sorry for the false alarm on my part :^)

Best regards, Grigory

[adsr]

No problem, thank you @txgk.

I posted #126 for a better default cluster width routine. I think it's at least better than the existing one, and maybe good enough for real world usage. Please have a look.

It's hard to write a test for this unless we make tb_cluster_width part of the public API. Maybe it would be handy to expose alongside tb_wcwidth and tb_iswprint.

We can make a separate issue to track the ability to override the default behavior with a function pointer or another struct field.

[Old-Farmer]

LGTM

[NikitaIvanovV]

I have had pretty good experience time with libgrapheme while making a program with termbox2. Maybe it could be possible to make it an optional dependency for those who want termbox2 to understand correct grapheme cluster width?

static int tb_cluster_width(uint32_t *ch, size_t nch) {
#ifdef TB_OPT_LIBGRAPHEME
    return grapheme_next_character_break(ch, nch);
#else
    int wmax = -1;
    int vs15 = 0, vs16 = 0, ri = 0, zwj = 0;
    size_t i = 0;
    for (i = 0; i < nch; i++) {
        uint32_t c = ch[i];
        switch (c) {
            case 0xfe0e: ++vs15; break;
            case 0xfe0f: ++vs16; break;
            case 0x200d: ++zwj; break;
            default: if (c >= 0x1f1e6 && c <= 0x1f1ff) ++ri;
        }
        int w = tb_wcwidth(c);
        if (w > wmax) wmax = w;
    }
    if (wmax >= 1) {
        if (vs15) return 1;
        else if (vs16 || zwj || ri >= 2) return 2;
    }
    return wmax;
#endif
}