wasi:filesystem alternative - force guests to do all path resolution

[alexcrichton]

This issue is an extension of my comment here with a more precise design in a location better suiting something that's actually entirely orthogonal to the proposed direction of #742. To recap slightly, there are a number of motivations that this issue is designed to resolve:

1. Most hosts (to my knowledge) do not attempt to sandbox wasi:filesystem APIs. For example Node's wasip1 shim explicitly says it doesn't sandbox. My personal understanding of the situation is that it's just extremely difficult to write a sandboxed implementation with the current APIs that wasi:filesystem requires hosts to provide. To me, subjectively again, this is not a great property of wasi:filesystem if the goal is to provide a sandbox and most hosts just don't do that.
2. Embedders generally expect wasi:filesystem to be sandboxed, however. Additionally embedders expect the ability to weave together multiple host directories into the guest at various paths. Embedders also generally expect the ability to make host directories either read-only or entirely read/write. Embedders expect all modifications to be neatly contained within these directories, however.
3. No guest, to my knowledge, handles path resolution correctly with multiple preopens. For example with /a and /b preopens the guest path /a/../b/foo.txt does not resolve on any guest -- again, as far as I know.
4. No host nor guest, to my knowledge handles overlapping preopens in a standard or compatible way. If a guest has preopens /foo and /foo/bar, then behavior is probably quite host specific, embedder-specific, and guest-specific.
5. To my knowledge there is no existing portable OS API that can be used to exposed a woven-together filesystem to a guest as a single directory. Basically I do not think that these problems all exist for lack of a "oh we can just use that" solution. These, I believe, are all genuinely very hard problems to solve and with no preexisting solution suitable for hosts. (someone please correct me if I'm wrong!)

In my comment I also outlined why I don't think that #742 is a viable solution to at least a subset of these concerns. Notably implement #742 would involve putting even more complexity into hosts for path resolution and such, and if many hosts already aren't sandboxing anything today they'll almost certainly not provide another VFS-style layer on top of all of that.

---

One thing I've articulated to myself at least is that to have reasonable semantics for a filesystem there needs to be a single "thing" that does path resolution. Right now this is spread across the guest (picking a preopen) and the host (operation on a sub-path from a preopen). The basic idea behind this issue is to do all path resolution in the guest. What I mean by this is that wasi:filesystem interface surface area would never perform path resolution. Instead it would only have extremely primitive operations like "open this directory in this other directory" or "open this file in a directory" or "remove this file within this directory" -- things like that. Notably the name of the path being operated on would still be passed to APIs, but notably the path could not contain ., .., or /. This means that names passed to APIs are required to effectively be file names themselves.

Another aspect of this hypothetical new wasi:filesystem is that while symlinks would remain they would be followed in-guest as opposed to the host. Hosts would implement everything with the O_NOFOLLOW equivalent, for example. If you tried to open a file which is a symlink, that'd just return an error saying "nope, you gotta readlink that".

As-is, this sort of design I believe would neatly solve (1) and (2). Hosts would be required to validate paths, but the validation is trivial. Hosts would be required to pass O_NOFOLLOW everywhere and not operate on symlinks, but it should be quite easy to write conformance tests for this. If directories have no "parent" operation, then everything is trivially sandboxed because by definition guests can only recurse within the directories they are given. Hosts wanting to provide read-only views can also quite easily do so by rejecting any write-operations on such preopens. Overall, the theory at least, is that an API like this is significantly easier for hosts to implement and also a significantly easier-to-sandbox implementation as well.

As-is, this would additionally solve (3) and most (4). If guests do path resolution then they get to handle everything and correctly operating on multiple preopens is expected to fall out naturally. For example a guest-relative parent of /a exists (it's /, and it's just that if you try to open that it'll be a guest-level error). Additionally guests would be able to handle overlapping directories in at least a way that's consistent across hosts, albeit probably still a bit odd between guest languages.

API-wise what I'm thinking would more-or-less look the same as today's wasi:filesystem. Minor differences could include returning a descriptor from create-directory-at, returning resources for child directories in the stream returned by read-directory, removing all path-flags usage (nofollow is now the only option), adjusting relative APIs like link-at to perhaps say "put this descriptor at this path" instead of "put this path at this other path", etc. Largely things would look the same, but semantically they'd be totally different because the guest would have to do a lot more work.

Guests, by and large, expect a POSIX-like API to operate on files. Things like open, read, write, mkdir, etc. This proposal would place a significant burden on guests to implement this POSIX-like API on top of this hypothetical idea. Guests would have to do all the path resolution themselves, they'd have to maintain in-guest parent pointers for directories (if the host doesn't do that), and they'd have to handle symlinks by using readlink-at and then interpreting the result.

---

I do not want to paint this approach as a silver bullet, as it's far from it. Open questions in my mind include:

• How difficult would it be to actually build an API like this on existing hosts today? This wasi:filesystem is a significant departure from the POSIX-like interfaces most hosts support today, for example. One counter-argument to this concern, though, is that a fully sandboxed host is already a significant implementation and my hunch is that this design of wasi:filesystem would be much easier to implement than that.
• What would the performance of this API look like? My expectation is that naively this would be significantly slower than native hosts for traversing files. Opening /a/b/c/d.txt, for example, would mean opening b within /a, then using that to open c, then using that to open d.txt, and if any of those is a symlink guests would have to readlink-at-it and then handle the result. This is a significant departure from "probably one syscall on the host in fast paths today". For example Wasmtime will use O_RESOLVE_BENEATH with openat today, but that's fundamentally incompatible with this wasi:filesystem design.
• Naively I expect that the implemetation in at least wasi-libc to be quite complicated of how to do all this. I don't know if it would be reasonable to expect that implementation complexity to be reflected into other guest languages not based on wasi-libc.
• I've no idea how we would transition from today's wasi:filesystem to this hypothetical design. Best I can think of is "just deprecate everything" and add a bunch of parallel APIs. Not great, though.

---

Anyway, lots of possible upsides for this (every host can sandbox now!) and also lots of downsides (complicated for guests, still leaves lots of open questions for wasi:filesystem like "are paths case sensitive"). I wanted to write all of this down in an official issue at least as having this buried in #742 is probably not a good spot for this.

[alexcrichton]

To elaborate a bit on differences from today's WITs, here's how I would envision behavior/API changes for today's functions on descriptor and such:

• All path or *-path arguments are required to be a "segment". notably cannot contain / or cannot contain ../. as path elements. For example foo.txt is allowed, but ./foo.txt is not. Nor is /foo.txt. The one exception to this is the old-path argument to the symlink-at API. As the content of the symlink this is not validated at all.
• All path-flags arguments are removed and hardcoded as nofollow-always-enabled.

If it's possible to get away with it, I wouldn't propose adding a sort of descriptor.parent() -> option<descriptor> API to find the parent directory (returning none for preopens). I say this because I don't know how it would be implemented on the host, but the lack of this API forces the guest to maintain a stack of descriptors for all open directories so they can resolve to .. appropriately.

For preopens I think something needs to change about the structure of the names. If a guest gets preopens for . and / that's sort of nonsensical from a guest trying to view preopens as "just a single filesystem" as there's no relation to . and /. Without having thought about this too-too-hard, what I'd propose is something like:

interface preopens {
    use types.{descriptor, error-code};

    root: func() -> option<preopen>;

    resource preopen {
        descriptor: func() -> option<descriptor>;

        // here `path` must be a "path segment"
        child: func(path: string) -> result<option<preopen>, error-code>;
        children: func() -> list<tuple<string, preopen>>;
    }
}

Notably this (a) imposes a tree structure, (b) forces guests to think of "one root fs", (c) enables overlap as is allowed today, and (d) keeps path conventions purely in the guest by only operating on segments. Hosts would have to do a little work to modify the "list of unstructured name plus dir" that they manage today, but modeling it as a something like this I would hope wouldn't be too bad.

[lukewagner]

Moving path-resolution logic into the guest makes a bunch of sense, especially for these cases of assembling a VFS from multiple native directories. For this, what you sketched above sgtm!

To your performance-concern bullet though: I am also somewhat worried that, for cases where the host does just want to expose the whole filesystem or just one chroot-style directory (which both seem like valid and not-uncommon use cases) unconditionally doing all the path resolution in the guest with host-calls (and, I assume, syscalls) for each path segment might be a pretty significant performance penalty that in these cases doesn't buy the host or guest much value in exchange for the overhead.

One assumption I believe we all have is that most guest source code only wants to think about a single rooted filesystem; I think it's only in rather advanced/specialized virtualization scenarios where the guest code wants to "see below the hood" and think about preopens, etc. Thus, even if we want the path resolution to happen in the guest for the above reasons, it seems like the host's choice of how it wants to assemble this single rooted (possibly-virtual) filesystem.

Based on this, I wonder if we can reconcile these goals by defining two interfaces in wasi:filesystem:

• preopens as you've sketched above
• root (or some TBD better name), which also contains use types.{descriptor, error-code} but has a open: async func(path: string) -> result<descriptor, error-code> that does do path resolution

Then we say that the wasi:cli/command world imports only root, not preopens, so that, by default our producer tools are spitting out components that assume path resolution is handled "on the other side" of the import call. But we also define a new world wasi:cli/preopen-vfs (or some TBD better name) that exports root and imports preopens (and the other usual CLI stuff). And then WASI or wasi-sdk would publish a canonical component .wasm that targets wasi:cli/preopen-vfs, implementing root's path resolution in terms of preopens in guest code.

So then if the host wants to expose the OS filesystem in a basic way, it provides a native wasi:filesystem/root impl, or if the host wants to do something more advanced, it natively implements wasi:filesystem/preopens and automatically links in the canonical preopen-vfs implementation to virtualization wasi:filesystem/root. If we're doing host-/sys-call-per-segment anyways, the additional overhead of the preopen-vfs component would hopefully be amortized.

A few nice consequences fall out of this hybrid approach:

• It helps us isolate the impact of VFS sandboxing in perf testing (and in any external benchmarks folks do)
• In non-VFS scenarios, it reduces the baseline size and overhead of wasi-libc
• It opens the door to more advanced VFS scenarios that go beyond preopens, such as, e.g., "mounting" various future WASI stores as directories (analogous to what Emscripten's VFS does in the browser). I expect trying to address all these use cases in wasi-libc would feel like a bridge too far (b/c it adds overhead to all wasi-libc users), whereas if it's in a separate opt-in component (of which there can be several, targeting different worlds), it seems fair game.

(I mentioned this here b/c it could help address the perf downside mentioned in the OP, but if it's too much of a tangent, happy to move discussion to a separate issue.)

[badeend]

I'm slowly starting to like this idea. Though I have two main concerns:

Correctness

Path resolution logic is non-trivial. It needs to:

• handle . & .. segments (logically, not syntactically),
• maintain a stack of parent directories in order to know what .. actually refers to,
• recurse into symlinks,
• detect loops in symlinks,
• handle symlinks in the middle of paths differently than symlinks at the end of the path.
• be resistant to TOCTOU vulnerabilities,
• and probably more.. For reference, see wasmtime's implementation here: https://github.com/bytecodealliance/cap-std/blob/main/cap-primitives/src/fs/manually/open.rs

On the one hand, you could argue that this complexity is exactly why moving it into the guest is a good idea.
On the other hand, all guests need to implement this logic near-identically in order to stay interoperable with each other. Otherwise we can end up in a situation where symlinks created by guest A are unresolvable by guest B or vice-versa, despite running on the same host.

Whichever route we take, we need to make sure that the path resolution logic is fully specified as part of the WASI itself.

Performance

Naively resolving paths in user space takes at least (N-1)*2 + 1 syscalls, where N is the total number of path segments:

• An open+close pair for each intermediate segment, and
• a final open for the basename.

Example: resolving a relatively modest path /a/b/c/d/e/f takes 11 syscalls. If symlinks are involved, that number could easily multiply.

---

Requiring the resolution logic to live inside the guest strips away all possibilities for the host to accelerate this logic. It forces the worst-possible performance characteristics onto the wasm application, even when the host doesn't care about the additional security.
A valid use case (IMHO) for not needing the additional sandboxing is: running the WASI workload inside a Docker container, where the filesystem virtualization is already handled "one level up".

Split interface

I think we can solve these problems by splitting the interface into two aspects:

• the core interface that exposes the descriptor-based primitives.
• a "utility" interface, which is semantically fully specified in terms of the lower level core interface. This can be implemented in the guest as e.g. a reusable wasm component. Or it can be implemented directly by the host for extra performance.

Abbreviated example:

interface core {
    get-root: func() -> descriptor;

    resource descriptor {
        /// Opens exactly *one* segment. Never follows symbolic links. Never traverses `..` etc.
        open: async func(name: string, ..) -> result<descriptor, error-code>;

        /// Performs the action on the open file descriptor itself.
        stat: async func() -> result<descriptor-stat, error-code>;

        // (...)
    }
}

interface paths {
    /// Opens an arbitrary path, starting at a base directory. Also traversing `..` and symlinks (if requested)
    /// This is fully implementable in terms of the `core` interface, but will be more efficient when implemented natively.
    open-at: async func(base-dir: descriptor, path: string, ..) -> result<descriptor, error-code>;
    
    /// Semantically equivalent to `open-at(base-dir, path, ..)?.stat()`, but potentially more efficient.
    stat-at: async func(base-dir: descriptor, path: string, ..) -> result<descriptor-stat, error-code>;

    // (...)
}

Notice the layering:

• paths::open-at is defined in terms of core::descriptor::open.
  • Reducing the number of syscalls from N to 1.
• paths::stat-at is in turn defined in terms of paths::open-at and core::descriptor::stat.
  • Reducing the number of syscalls from 3 (open+fstat+close) to 1 (fstatat).

The POSIX stat family boils down to various incantations of the single WASI stat method:

• fstat(fd, ..) is fd.stat()
• fstatat(fd, path, ..) is paths::stat-at(fd, path) (and in turn paths::open-at(fd, path)?.stat())
• stat(path, ..) is paths::stat-at(core::get-root(), path)

Single root preopen

The example above also exposes the root preopen as a regular descriptor. Now that the descriptor API has been drastically simplified and they are truly are "just" simple nodes in a tree, I don't think it adds significant implementation complexity for hosts to implement it this way. Realistically speaking, the guest just wants a single tree and doesn't care how it came to be there. This also resolves the ambiguity of having overlapping preopens.