Cache directory listings to fix Linux case-folding I/O slowdown (#327)

[WillScarlettOhara]

Problem

Follow-up to #327. On Linux, WeiDU emulates a case-insensitive filesystem by
routing every file operation through fcase() (fcase/fcase.c), whose
casepath() resolved each path component with an opendir() + full readdir()
scan of the parent directory. With no caching a single lookup is O(entries),
and load_resource() (src/load.ml) repeats it across every override directory
for every resource — so a large modded override costs O(resources × files)
directory scans and dominates install time. #327 measured COPY_EXISTING_REGEXP
over BG2EE CRE going from ~1.5s to 2m46s (sys 1.2s → 2m43s) between v249
and v251rc4, and an EET megamod install slowing by 7–8×.

That issue was closed after adding the --no-case-fold / WEIDU_NO_CASE_FOLD
escape hatch, with the note that a real fix needs a different approach. This is
that approach.

Change

Add a per-directory listing cache (lowercased base name → real base name) in
fcase.c: each directory is scanned once, then component lookups are O(1).

Invalidation is driven from OCaml, where the semantic operation is known — which
is what makes correct invalidation feasible (doing it blindly inside C is not):

• case_ins_linux.ml mutation wrappers add / remove / rename entries on create,
  unlink, mkdir, rmdir and rename.
• exec_command (util.ml) clears the cache after running an external program.
• A few stray Unix.unlink calls in tpaction.ml (biff/backup handling) are
  routed through Case_ins.unix_unlink so those deletions are reflected.

It is transparent and on by default whenever case-folding is on; the existing
--no-case-fold / WEIDU_NO_CASE_FOLD toggle is unaffected. The cache is
Linux-only — case_ins_{mac,win}.ml gain only a no-op fcase_cache_clear so
shared code can call it unconditionally.

Correctness

• Match semantics are unchanged: same strcasecmp, same append-as-is for
  names that do not exist yet. Behavior on case-insensitive filesystems (NTFS,
  CIOPFS, …) is therefore preserved — unlike the earlier micro-optimizations in
  I/O performance on Linux should be improved #327 that broke NTFS.
• A stale missing entry is harmless (append-as-is is already correct on a CI
  FS); only a stale present entry is risky, which is why non-Case_ins
  mutations are routed through the wrappers and the cache is cleared after
  external commands.
• WeiDU never chdir()s, so relative cache keys (./Override) stay valid for
  the whole run.

Performance

Standalone C harness exercising the real fcase.c, old casepath vs new, over a
synthetic modded-scale override (single pass, distinct files):

override files	resolutions	old (no cache)	new (cached)	speedup
20 000	4 000	5.642 s	0.006 s	~950×
50 000	10 000	42.264 s	0.018 s	~2300×

Testing

• fcase.c logic verified with a standalone C harness (17 checks over a real temp
  directory tree): basic folding, create-then-read hot path without rescan,
  staleness + clear, delete, nested + absolute paths, descend-into-file fallback,
  rename. All pass, warning-free under -Wall.
• All three case_ins_*.ml variants typecheck clean.

Please verify before merge in a normal build environment (the change was
developed on a box that could not build unsafe-string OCaml):

1. make on Linux; confirm fcase.o links.
2. Perf on a real large install — expect sys back to seconds.
3. NTFS3 / NTFS-3G / CIOPFS regression, given the NTFS history in I/O performance on Linux should be improved #327.
4. --no-case-fold / WEIDU_NO_CASE_FOLD still bypass entirely.

🤖 Generated with Claude Code

[WillScarlettOhara]

Pushed a follow-up commit (c79688c) fixing a correctness bug I found while testing this branch against a large real-world mod install (IWD2EE, 3073 files into override/).

The bug. The install died with resource [IEex_TRA.lua] not found for 'COPY' — on a file the same run had written into override/ moments earlier.

casepath() derives its cache keys from the prefix it has resolved so far, and it pushed every component onto that prefix, including ".". WeiDU composes paths as game_path ^ "/override/..." with game_path = ".", so lookups resolved override under the prefix "./.", while the OCaml invalidation side keys on Filename.dirname, i.e. ".". That is two cache entries for one directory: the mkdir of override/ and every file created in it landed under ".", and the "./." entry — scanned before override/ existed — stayed stale for the rest of the run. Resolution then stopped at the unmatched override component, fcase returned the path unchanged, and the case-sensitive stat() of the lowercase name failed.

The fix, in two layers:

1. One directory, one key: casepath() now drops "." and empty components (the latter from // and trailing slashes) instead of appending them to the resolved prefix.
2. A missed invalidation now degrades into a re-scan rather than a wrong answer: each cached directory is stamped with its mtime, and a MISS is revalidated against it before it is believed. Any mutation the OCaml wrappers do not report (an unwrapped code path, another process) bumps the directory mtime and gets picked up. Hits stay syscall-free; fcase_cache_add/_remove re-stamp the entry so our own mutations don't force a re-scan.

Numbers — installing IWD2EE Core (3073 files) on Linux, identical resulting install (same file count, same dialog.tlk string count):

	real	user	sys
without the cache (parent commit)	63.4s	41.1s	21.7s
with the cache + this fix	54.9s	40.1s	14.2s

Before the fix, this same install failed outright at 0.4s, so the branch as it stood could not have completed a mod of this size.

[FredrikLindgren]

Nice work. Sorry I've been slow to respond. I hope to be able to more thoroughly review this in the next short while.

[WillScarlettOhara]

Nice work. Sorry I've been slow to respond. I hope to be able to more thoroughly review this in the next short while.

Thank you. No rush on reviewing all this. I haven't fully tested it on my end yet because I'm currently busy with other projects, and running Linux/Windows comparisons forces me to switch back and forth via dual boot since I unfortunately only have one PC.

Still, performance does tank via Wine, but these commits should allow native linux WeiDU to get closer to Windows speeds.

[FredrikLindgren]

Okay. The merge window for this will probably be v253, as I intend for v252 to be a bit of a bug-fix release, after v251 wound up being a bit useless after an ill-considered feature.