-
-
Notifications
You must be signed in to change notification settings - Fork 145
feat(fsops): add Backend interface, local implementation, and pool resolver #1914
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
nitrobass24
wants to merge
39
commits into
develop
Choose a base branch
from
feat/fsops-backend-interface
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
39 commits
Select commit
Hold shift + click to select a range
29bf02e
feat(fsops): add Backend interface, local implementation, and pool re…
nitrobass24 85117b5
fix(fsops): add nil guards to NewPool and fix package doc
nitrobass24 f493a61
fix(fsops): guard against nil DirEntry in WalkDir callback
nitrobass24 84203bc
fix(fsops): check os.Stat error in HardlinkTree test
nitrobass24 56baa6b
fix(fsops): reject recursive Remove when IgnorePaths is set
nitrobass24 01eec8a
fix(fsops): propagate WalkDir errors to channel
nitrobass24 0180680
fix(fsops): return context error from noopBackend when cancelled
nitrobass24 be36492
Merge branch 'develop' into feat/fsops-backend-interface
nitrobass24 8b9d1bc
Merge remote-tracking branch 'origin/develop' into feat/fsops-backend…
nitrobass24 d49a1ff
fix(fsops): adopt handle-based rollback from new hardlinktree API
nitrobass24 0500c3c
Merge remote-tracking branch 'origin/feat/fsops-backend-interface' in…
nitrobass24 a78d210
fix(fsops): surface ReadDir entry metadata errors and correct package…
nitrobass24 cfecf93
fix(fsops): report GetFileID failures on walk entries; portable test …
nitrobass24 36ede40
fix(fsops): match IgnoreDirNames case-insensitively; drop misleading …
nitrobass24 3b9d69b
refactor(fsops): drop backend surface with no callers
nitrobass24 8d962b5
fix(fsops): degrade FileID resolution failure instead of failing the …
nitrobass24 fdca03b
fix(fsops): don't lstat every ReadDir entry for an unread Mode field
nitrobass24 ad15417
fix(fsops): make noop RemoveTree honor the nil-handle contract
nitrobass24 9c46f8d
test(fsops): make cancellation test able to fail; cover SkippedExists
nitrobass24 0cc2264
docs(fsops): stop citing out-of-repo design docs; document caller obl…
nitrobass24 a06a174
docs: add fsops to the architecture module map
nitrobass24 478ebce
docs(design): add SFTP-native remote backend design
nitrobass24 31004ca
docs(design): scope Windows remotes — probe-degraded SFTP tier, no ex…
nitrobass24 a113cac
refactor(fsops): cut unused flexibility from the backend surface
nitrobass24 03187ab
docs(design): decide file identity wire form — opaque, at remote-back…
nitrobass24 30f15e0
docs: fix migration tense; specify TOFU confirmation and remote path/…
nitrobass24 61cc6a1
test(fsops): cover FileIDErr degradation on identity failure
nitrobass24 e5c0d3d
Merge branch 'develop' into feat/fsops-backend-interface
nitrobass24 948e5cb
docs(fsops): describe both operation tiers in the Backend godoc
nitrobass24 46be690
Merge remote-tracking branch 'origin/feat/fsops-backend-interface' in…
nitrobass24 0a5dff0
fix(fsops): skip stat-failed walk entries; give Stat followed-target …
nitrobass24 c933f52
docs(design): adopt security review suggestions — tagged FileID, syml…
nitrobass24 0f1b905
fix(fsops): pin portable error semantics; document path domains and S…
nitrobass24 c4a763d
test(sharedextents): adapt windows test to reflinktree.Create handle …
nitrobass24 27bf61f
test(fsops): use require for error assertions per testifylint
nitrobass24 91b8a16
docs(design): align Schema section with the pinned-key security design
nitrobass24 c5a5a06
fix(fsops): match unix Statfs semantics for file paths on Windows
nitrobass24 b8936aa
docs(fsops): correct the WalkDir final-entry comment; cover the per-e…
nitrobass24 a0006a7
docs(design): per-extension capability tracking; decide SameFilesyste…
nitrobass24 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,95 @@ | ||
| // Copyright (c) 2025-2026, s0up and the autobrr contributors. | ||
| // SPDX-License-Identifier: GPL-2.0-or-later | ||
|
|
||
| package fsops | ||
|
|
||
| import ( | ||
| "context" | ||
| "io/fs" | ||
|
|
||
| "github.com/autobrr/qui/pkg/hardlinktree" | ||
| ) | ||
|
|
||
| // Backend abstracts filesystem operations so services work identically against | ||
| // a local filesystem or a future SSH-backed remote. It covers exactly the | ||
| // operations qui's services need: syscall-level primitives (stat, walk, | ||
| // mkdir, remove) plus the high-level tree operations (HardlinkTree, | ||
| // ReflinkTree, RemoveTree) that create-and-rollback as a unit. Path | ||
| // manipulation (filepath.Clean, filepath.Rel, etc.) is not part of this | ||
| // interface — it stays as direct calls in service code. | ||
| // | ||
| // Every method accepts a context.Context and must respect cancellation. | ||
| // | ||
| // Error semantics are portable across implementations: a missing path is | ||
| // reported compatibly with errors.Is(err, fs.ErrNotExist) and a denied one | ||
| // with errors.Is(err, fs.ErrPermission) — from Stat, Lstat, ReadDir, | ||
| // WalkDir (on entries), Remove, and Statfs alike. Remote backends must map | ||
| // their transport's errors onto the same sentinels. | ||
| type Backend interface { | ||
| // --- Read --- | ||
|
|
||
| // Stat returns metadata for a single path, following symlinks — FileID | ||
| // and Nlinks describe the target, so symlinked torrent data keeps its | ||
| // identity. Returns a non-nil error wrapping fs.ErrNotExist if the path | ||
| // does not exist. A failure to resolve identity is reported in | ||
| // LstatInfo.FileIDErr, not as a Stat error, so one identity-opaque file | ||
| // cannot fail callers that only need metadata. | ||
| Stat(ctx context.Context, path string) (*LstatInfo, error) | ||
|
|
||
| // Lstat is like Stat but does not follow symlinks. | ||
| Lstat(ctx context.Context, path string) (*LstatInfo, error) | ||
|
|
||
| // ReadDir returns directory entries. | ||
| ReadDir(ctx context.Context, path string) ([]DirEntry, error) | ||
|
|
||
| // WalkDir walks a directory tree and streams entries on the returned channel. | ||
| // The channel is closed when the walk completes, is cancelled via ctx, or | ||
| // hits an unrecoverable error. Callers must drain the channel or cancel | ||
| // ctx; abandoning it leaks the walk goroutine. Entries whose metadata | ||
| // cannot be read are skipped, not emitted — WalkEntry.Err carries only | ||
| // enumeration-level walk failures. | ||
| WalkDir(ctx context.Context, root string, opts WalkOptions) (<-chan WalkEntry, error) | ||
|
|
||
| // Statfs returns free/total bytes for the filesystem containing path. | ||
| Statfs(ctx context.Context, path string) (*StatfsResult, error) | ||
|
|
||
| // SameFilesystem returns true if both paths reside on the same filesystem | ||
| // (same device ID on Unix, same volume serial on Windows). | ||
| SameFilesystem(ctx context.Context, p1, p2 string) (bool, error) | ||
|
|
||
| // --- Write (mutating) --- | ||
|
|
||
| // MkdirAll creates a directory and all parents. Equivalent to os.MkdirAll. | ||
| // For torrent content and link-tree dirs, pass fsutil.ContentDirMode / | ||
| // fsutil.LinkTreeBaseDirMode rather than a hand-typed mode (#1704, #2086). | ||
| MkdirAll(ctx context.Context, path string, perm fs.FileMode) error | ||
|
nitrobass24 marked this conversation as resolved.
|
||
|
|
||
| // Remove removes a file or directory. If opts.Recursive is true, removes | ||
| // the entire tree (like os.RemoveAll). | ||
| Remove(ctx context.Context, path string, opts RemoveOptions) error | ||
|
|
||
| // --- High-level (atomic, server-orchestrated) --- | ||
|
|
||
| // HardlinkTree creates a hardlink tree from plan. Rolls back what it | ||
| // created on partial failure. The result records the files and dirs this | ||
| // call made, for a later RemoveTree. | ||
| HardlinkTree(ctx context.Context, plan *hardlinktree.TreePlan) (*TreeCreateResult, error) | ||
|
|
||
| // ReflinkTree creates a reflink (CoW) tree from plan. Rolls back what it | ||
| // created on partial failure. The result records the files and dirs this | ||
| // call made, for a later RemoveTree. | ||
| ReflinkTree(ctx context.Context, plan *hardlinktree.TreePlan) (*TreeCreateResult, error) | ||
|
|
||
| // RemoveTree removes exactly the files and dirs recorded in created — | ||
| // never the whole plan, which could delete links shared with sibling | ||
| // torrents (discussion #2282). A plan root that already existed before | ||
| // the create is therefore NOT removed; callers own pruning it. Safe to | ||
| // call with a nil result. | ||
| RemoveTree(ctx context.Context, created *TreeCreateResult) error | ||
|
|
||
| // --- Capabilities --- | ||
|
|
||
| // SupportsReflink returns whether the filesystem at path supports CoW | ||
| // reflinks. The string return is a human-readable reason when unsupported. | ||
| SupportsReflink(ctx context.Context, path string) (bool, string, error) | ||
| } | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| // Copyright (c) 2025-2026, s0up and the autobrr contributors. | ||
| // SPDX-License-Identifier: GPL-2.0-or-later | ||
|
|
||
| package fsops | ||
|
|
||
| import "errors" | ||
|
|
||
| // Sentinel errors returned by Backend implementations. | ||
| var ( | ||
| // ErrNoFilesystemAccess is returned by the NoopBackend for instances that | ||
| // have no filesystem access configured (neither local nor remote). | ||
| ErrNoFilesystemAccess = errors.New("filesystem access is not configured for this instance") | ||
| ) |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.