Whole UNICODE debacle

[WaldemarH]

Hi guys,

I've been reading and reading and reading,..... on what to do with this whole UNICODE thingy and for us to properly support UNICODE it's not just enough to use TCHAR. Unfortunately even UTF-16 is a multi-code-unit encoding.

Encoding form    Code unit size   Code unit
UTF-32           32-bit           Code point is encoded with one 32-bit code unit
                                  - U+000000 – U+10FFFF (1 unit)

UTF-16           16-bit           Code point is encoded with one or two 16-bit code units
                                  - U+000000 – U+00FFFF (1 unit)
                                  - U+010000 – U+10FFFF (2 units, surrogate pair)

UTF-8            8-bit            Code point is encoded with one to four 8-bit code units
                                  - U+000000 – U+00007F (1 unit)
                                  - U+000080 – U+0007FF (2 units)
                                  - U+000800 – U+00FFFF (3 units)
                                  - U+010000 – U+10FFFF (4 units)

Currently Salamander uses locale which uses whole unsinged 8 bit as a representation of the character, but if we want to make it a real 'full blown UNICODE' there is no other way then to make it multi-code-unit encoding aware.

And as soon as you go down that rabbit hole we'll have to start using string class that processes code-units and not chars.
A simple At(0) would return a View that could be 1 to 2/4 code-units long (depending on the char type used).

Now with W10 some UTF-8 support started to be present, but I'll assume that this is really just a User level layer that converts UTF-8 into UTF-16 and not a real UTF-8 Kernel level support (i.e. slower calls)... there is simply toooo much history involved to make Wins UTF-8 OS.

Also additional though is that if we limit the APP to UTF-8 most of the old Wins versions wont support UNICODE and that would be a bummer if we'll invest so much time to make the app UNICODE aware.

So my solution would be:

1. we'll have a 'string class' which will ALWAYS assume that the string is encoded (i.e. can be made out of multiple code-units... locale (1 code unit), UTF8 (1..4) or UTF16 (1..2))

2. always use TEXT( .. )_str to create a 'view of a text' which 'string class' will know how to use

3. if some really special string manipulation would be performed we can:

   1. create a much more complex algorithm that is multi-code-units aware
      (higher chance of new bugs)
   2. decode the stored string to char32_t string and use the original algo. on a direct UINICODE
      code points
      This solution would made Salamander a little slower as it would be processing DWORDs instead of
      BYTES, but the algos would stay the same and the results can always be converted back to the
      "encoded" string class to lower the space needed.

4. string class will have:
   Size_CodePoints() -> return number of "real UNICODE characters" (even if encoded as 4 bytes it will still represent 1 real UNICODE character)
   Size_CodeUnits() -> return number of code-units
   Size_Bytes() -> number of bytes used to encode the string

I've intentionally used Size instead of Length just to make sure that nobody would think of "string length" anymore.. it's "size" i.e number of requested units (whatever that unit is).

So.. my thoughts:

1. every defined string needs to be a view (e.g. TEXT( "abc" )_str ).. i.e. no more raw text
2. use 'string class' as much as you can
3. for direct manipulation of a text use char32_t "characters"

With this rules we'll have the flexibility to use locale or UTF-8 or UTF-16 encoding... i.e. we can then adapt to any OS version we need.

Anyway I'm already writing the string class... it's now the 3rd version already... that's why this whole text came to be as I've encountered issues when dealing with characters.

Any thoughts?

Best regards
Waldemar

[melloware]

@WaldemarH this may be an ignorant C++ question but I feel like there has to be some String library out there that already handles all this? We can't be the first app to run into this issue right?

[WaldemarH]

I've been thinking about it and had a plan to search for it.

Now the thing is that Sal. doesn't do much string manipulation like toUpper,... whatever, but it deals more with paths (removing, appending, substring searching, splitting,...) and the idea was that this specialized Sal. string class would do most of them, so we would have as little "out of order string manipulations where a direct access to string text would be needed" as possible (currently these are more or less the norm.. always working directly on the strings).
With this "out of string class code" would look the same for all the char types that we need to support and as we are in the process of conversion the class needs to handle char (locale), char8_t (UTF8) and wchar_t (UTF16) with the same function behavior (and probably also all the conversions between them at this stage).

So there is probably very little chance that something like this exists.. then again we are not the first that we need to do that, so there is still some hope left.

[janrysavy]

The key point is that this class will be only a few hundred lines; in the overall Unicode migration, it’s the smallest issue. Ninety-nine percent of the work lies in refactoring all the scattered path operations in the code to use this class.

[janrysavy]

MAX_PATH appears in the code about 5,000 times, which means manually updating 5,000 spots when migrating to the mentioned class. Several thousand more places are related even though they don’t explicitly reference MAX_PATH. We’re easily talking about ~10K changes, where you have to understand what the existing code does, refactor it to use one of the new class’s methods (or add such a method), and ideally step through in the debugger to verify the code still works after the change.

[shakeyourbunny]

What is then the issue with all of that? I have been following the whole UTF discussion with Salamander for over 10 years and in the meantime, other similiar applications (that are even older, like Total Commander) don't have this issue.

Instead, what I have gathered around all of the iterations of Salamander, there always have been bandaids around the issues, written some custom support slapped onto parts of Salamander.

Why doesn't Salamander does replace the old ANSI encoding core (I'll just call it in lack of better definition) with a proper, portable (for the future...) one that sits on top of standard Windows functions, encapsulates them and provide for the rest of the application an uniform interface?

I won't even start at the portability of the application for other operating systems, where I sorely miss Salamander, but have arranged myself begrudgingly using KDE Dolphin (and midnight commander).

Come to it, it smells like at times, like the Python 2 and 3 situation, where the latter evolution of Python fundamentally tackled that and solved it mostly?

[WaldemarH]

The problem here is that there is A LOT of code and I mean A LOT. And that code also assumes the simple 1 char is 1 code point, where (since W2000) wide character support in Windows uses UTF16 which can have 2 code units to define 1 code point. Which means that a most of the path processing in Salamander would sooner or later fail to do the proper thing.

Now I've started converting a Find dialog to TCHAR support... I just wanted to do some small change.. that was a mistake.. Find is huge and it uses all kinds of stuff. So I've rewritten registry functions and updated all kinds of strings stuff.. have also written a few different versions of the string class, but at the end nothing worked, so I'm now in a situation where there are char and wide string classes where one or the other is used as a TChar string representation... now the class has also knowledge of code unit/code point difference which std classes don't have. Now it's still a simple class which can be replaced if needed.
I've also integrated the string view classes so now the code will use that instead of const char*.
But as I've said there is A LOT of code to update and it will take time... and when Salamander will be done, there are still plugins to be done. So...

[janrysavy]

Thanks to @WaldemarH for the thorough exploration of the issue and for starting work on the UTF-16 conversion. He came to the same conclusion as Petr and I did: we need to encapsulate all operations on path buffers into a dedicated class.

The old Salamander code is very unfortunate in this regard - path operations are scattered throughout the codebase, implemented differently in each place depending on what was convenient at the time.

If we want to switch to UTF-16 (or UTF-8), we must handle the situation where a single character does not always map back to exactly 2 (or 1) bytes in the buffer, exactly as WaldemarH points out. Without encapsulating path operations, we can never be sure that somewhere in the code there isn’t pointer arithmetic accessing [pos - 1] in the buffer.

If we wanted to ignore this risk (which we believe is dangerous and certainly not what Salamander users would expect in terms of robustness and code quality), one option would be to simply triple the size of all MAX_PATH buffers, let the system return UTF-8 instead of ANSI strings, and… pray.

However, the UTF-8 solution comes with a problem that UTF-16 paths do not suffer from. Internally, all names on file system are just sequences of WORDs. They are not necessarily valid UTF-16 strings. On disk, it is easy to encounter WORD sequences that are invalid in sense of UTF-16 encoding.

When we request UTF-8, Windows performs a UTF-16 -> UTF-8 conversion. The behavior on invalid UTF-16 sequences depends on the conversion settings (we can check in the debugger what parameters Windows 11 currently uses, but this provides no guarantees for the future).

Once the conversion to UTF-8 has happened "somehow", Salamander may perform an operation on the path - for example, appending a backslash and a filename. The operating system will then convert the string back UTF-8 -> UTF-16. At this point, the original directory name may differ from the new one.

In short, the UTF-8 approach carries all the problems ( https://github.com/KRtkovo-eu-AI/salamander/issues?q=is%3Aissue%20state%3Aopen%20UTF8 ) we need to solve for UTF-16, and adds this extra one on top.

[janrysavy]

To demonstrate the described disadvantage of the UTF-8 approach compared to UTF-16, I prepared a sample that lets you test file and directory names that are invalid from the perspective of UTF-16 encoding. It’s available at tools\utfnames\utfnames.vcxproj. Example usage:

utfnames.exe createdir . U+0046 U+D800
  code points: U+0046 U+D800
  utf-16 units: 0x0046 0xD800
  preview: F?

utfnames.exe list
[UTF-8]
  [DIR] F�
    code points: U+0046 U+FFFD
    utf-8 encoding: 0x46 0xEF 0xBF 0xBD
    utf-16 encoding (from utf-8): 0x0046 0xFFFD
    utf-16 preview: F�
    utf-8 access: failed (2: The system cannot find the file specified.)

[UTF-16]
  [DIR] F?
    code points: U+0046 U+D800
    utf-16 encoding: 0x0046 0xD800
    utf-16 access: directory, attributes DIRECTORY|NOT_CONTENT_INDEXED (0x00002010)

This creates a directory name that can be opened by UTF-16 applications (for example, Windows Explorer, Notepad, …), but will not be accessible to UTF-8 applications.

[shakeyourbunny]

Just asking a dumb question: Why do other file managers on Windows don't have any issues on that and that since decades?

Also you will be cornering yourself all the years that is being discussed, but no proper action taken.

My recommendation (I'm not really a developer, I let Claude look at your source and asked it how to solve it):

Go Direct to UTF-16 - Skip TCHAR Altogether
Don't bother with TCHAR compatibility or maintaining ANSI versions. Here's why:

• Windows has been UTF-16 internally since Windows NT (1993)
• Every successful Windows file manager today uses UTF-16
• The performance overhead is negligible on modern systems
• TCHAR adds complexity without meaningful benefit in 2025

Implement a SalString wrapper for gradual migration
Something like that:

class SalString {
    std::wstring m_str;
public:
    // Implicit conversions for legacy compatibility
    operator const char*() const; // Returns temp UTF-8
    operator const wchar_t*() const { return m_str.c_str(); }
    
    // Modern interface
    const std::wstring& wide() const { return m_str; }
    std::string utf8() const; // Convert to UTF-8
};

Convert file operations to UTF-16 systematically (one module at a time)

• File system operations (FindFirstFile, CreateFile, etc.) - use the W versions
• Path handling - implement proper long path support (?\ prefix)
• Directory enumeration - core file manager functionality
• UI display - can be done later with conversion layers

Add UTF-8 manifest support for modern Windows versions

• Use std::wstring for new string handling
• Keep existing buffer management but wrap it
• Add RAII only where it significantly improves safety (file handles, etc.)

Create plugin compatibility layer that converts between UTF-8/UTF-16 as needed
The experimental UTF-8 approach actually can work well if done right:

• Use SetProcessDefaultCPAuto() or manifest-based UTF-8 support
• Keep internal strings as UTF-8
• Convert only at Win32 API boundaries

This is actually simpler than UTF-16 for text processing

Ship incremental updates rather than waiting for complete rewrite
The Real Issue: Analysis Paralysis, the OpenSalamander team is overthinking this because:

• Emotional attachment to the legacy codebase
• Fear of breaking existing functionality
• Perfectionist approach instead of pragmatic incremental fixes
• Plugin compatibility concerns that could be solved with adapter layers

[WaldemarH]

Ping me when it will be finished.

[melloware]

appreciate all your efforts @WaldemarH and @janrysavy i think once everything is stable and OS is released i think you are going to see a flurry of new plugins and activity!

[WaldemarH]

@melloware .. it will take some time as OSal is big with a lot of custom/local code.
My idea is also to replace some custom classes like arrays with STL vector, but all this will happen after discussion with Jan and why specific things were used in the first place.. like Registry thread (still waiting for explanation :) ).

For now I've added some things that bothered me or was missing it like 'to other panel' menu item (very useful.. I'm using it all the time.. still thinking how to find even faster solution), 'no color under item icon in file list', scroll to open history list,..
With all this there was also a lot of code cleaning, redesigning,...

[WaldemarH]

@janrysavy hi jan may I ask on which files are you currently working? .. maybe I can merge someting to files that are already translated?

[janrysavy]

it's in a very work-in-progress state, we're changing hundreds of files...

[Luddite-Geekazoid]

I had assumed that the request for UNICODE support was so that the Internal Viewer would function better. I hadn't considered anything besides that.

Whenever I needed to view something that Salamander didn't support, I'd open it in Emacs.

[0xeb]

@Luddite-Geekazoid , please try this fork: https://github.com/0xeb/salamander/ . Please report bugs.