Handling filesystems with non-Unicode filenames

[Aurel300]

Many filesystems allow filenames in the format:

Any byte except NUL, / (ext4)

or even:

Any byte except : (MFS)

Both NTFS, ReFS (Windows) and HFS+ (macOS) impose a fairly reasonable limitation:

Any Unicode except NUL, / (ReFS)

With an only-Unicode paths limitation, we can express paths as Unicode Strings in Haxe and have nice FileSystem interfaces. Since we support Linux, however, there is a potential problem when running on e.g. ext4 and needing to read or write into a file with a name that is not a valid Unicode string.

(Tested on RC.2, Ubuntu 14.04 on an ext4 filesystem.) The behaviour atm is very inconsistent across targets:

$ mkdir invalid
$ cd invalid
$ touch `printf "\xc3\x28"` # (this creates a file with a Unicode-invalid filename)

class Test {
  public static function main():Void {
    trace(sys.FileSystem.readDirectory("invalid")[0].charCodeAt(0));
  }
}

On eval, filename.charCodeAt(0) == 232
On neko, filename.charCodeAt(0) == 195
On PHP, filename.charCodeAt(0) == 104
On Python, filename.charCodeAt(0) == 56515 (AND it thinks the String is valid Unicode, i.e. Python sanitised it before Haxe got it)
On JS (node), filename.charCodeAt(0) == 65533

The proposed solution by @ncannasse was to have a scheme for "escaping" and "unescaping" paths between the filesystem and Haxe. Any invalid Unicode sequence would be replaced by a special Unicode character followed by the original byte turned into a Unicode point, i.e. "<invalid byte 7E>" -> "<escape character><unicode character U+007E>" (note that U+007E is not the same as the byte 7E).

[RealyUniqueName]

I thought, it's specified that our strings are unicode now, please, let's not make them "some times it's bytes in an ugly representation" :)

[RealyUniqueName]

Our strings are now unicode so we don't allow anymore arbitrary byte injection in it.

#8119 (comment)

[Aurel300]

I agree, I would also prefer strings to always be valid Unicode. We do need to deal with the case of a non-Unicode filename somehow, though. Some alternative options:

• simply throw an error when a non-Unicode filename is found
• return nulls instead
• change the API to return Either<String, Bytes>

None are particularly good solutions.

I think having a completely separate alternative API to deal with Bytes-encoded paths would be good, but it is a large task and does not solve the original problem either.

[ncannasse]

The best way to deal with it is to do the following:

• have filepathOfBytes : haxe.io.Bytes -> String and bytesOfFilepath : String -> haxe.io.Bytes that does some escaping/unescaping of invalid unicode chars in the form \uSPECIAL\u00<byte> where \uSPECIAL is an unassigned unicode code point.
• make sure (on filesystem that matters) that all not utf8-encoded strings are passed through filepathOfBytes for proper encoding
• make sure that all file paths go through bytesOfFilePath before being sent to the low level API
• user that want to deal with bytes filepath can call the two filepathOfBytes and bytesOfFilepath explicitely

[Aurel300]

After some more thought I realised that having the filepathOfBytes and bytesOfFilepath functions in Haxe code assumes we can read file paths as byte sequences to begin with.

user that want to deal with bytes filepath can call the two filepathOfBytes and bytesOfFilepath explicitely

Likewise here, we need to provide a Bytes API for the users to be able to do that.

[ncannasse]

@Aurel300 yes, we already have haxe.io.Bytes API for manipulating bytes, and we could add public functions fromFilePath/toFilePath to it for doing the escaping/unescaping proposed in my other comment.

[Aurel300]

@Aurel300 yes, we already have haxe.io.Bytes API for manipulating bytes, and we could add public functions fromFilePath/toFilePath to it for doing the escaping/unescaping proposed in my other comment.

I meant a Bytes API to produce the filepaths as Bytes. For example the equivalent of sys.FileSystem.readDirectory(dir) would be something returning Array<Bytes> instead of Array<String>, each entry being a filename in the directory.

[ncannasse]

@Aurel300 we don't need this, because that would require all APIs taking/returning file paths to have two versions, which is bad design.

Instead, FileSystem.readDirectory would return (potentially escaped) String. The user that want to get the exact corresponding bytes can use Bytes.fromFilePath(fileName) . However, he can directly read the file using sys.io.File.getContent(fileName) for example.