Repository navigation
feat(vfs): add read intents and per-mount read profiles - #22
Merged
Merged
Conversation
- FakeDriver read behavior for strict-range fallback, gated bodies, and call/concurrency stats - scenario tests for parallel ranges, per-file cap, slot release, and prefetch cancellation - VFS delete, FUSE adapter release, mobile close/cancel, and syncer worker isolation
- cancellation registries drain together across invalidate/session/seek/close - scheduler capacity invariants hold under 32x100 concurrent acquire/release - findWindow costs ~31/80/450 ns/op at 1/10/100 in-flight windows - VFS and direct read agree on parallel/single/fallback policy - record the minimum-viable scope decision in the OpenSpec change
- add ReadPlan and PlanResolver with concrete defaults (no 0/-1 sentinels) - move plan resolution after the FUSE seek heuristic and apply the concrete window - adaptive sequential upgrade returns a new plan while keeping the FUSE window - add resolver table tests and assert the adaptive window stays unchanged
- add ScopeRegistry with (cacheKey, sessionID) keys, refcounts, and cancel reasons - remove prefetch.cancels/reasons/inFlight; dedupe via the window registry - move the foreground read counter into its own foregroundState - remove canceled scopes so later prefetches start with a fresh lifetime - update cancellation tests and OpenSpec phase 2 tasks
- add CachePolicy as the single hot/durable admission point - collapse direct and wait-window cache lookups into lookupCachedChunk - preserve existing phase/source diagnostics for both lookup paths - add admission matrix and parallel hot-only regression tests - record phase 3 in the OpenSpec change
- architecture.md: list plan, scope, and cache-policy responsibilities - changelog: record the internal read-pipeline convergence - mark the phase 4 documentation and gate tasks complete
- keep the first per-file channel when a partial window asks for a different limit - pass the stable profile BulkStreams as perFileLimit; window streams only control the split - add a regression test that fails before the fix - record the Scheduler next phase and mark tasks 5.1/5.2 complete
- Scheduler owns global normal/high slots, per-file budgets, prefetch permits, and the foreground count - replace slotState, parallelState, prefetchState, and foregroundState in the VFS read path - keep the two-phase acquire order and priority behavior unchanged - migrate VFS foreground, parallel, prefetch, and stream read-ahead; direct sharing remains task 5.5
- move the scheduler implementation to pkg/readbudget so drive and vfs can share it - keep VFS aliases and behavior unchanged - give direct reads an independent 16-slot scheduler instance - update the architecture policy, docs, and task 5.5
- ReadPlan.MaxStreams is a hint; Scheduler.GrantStreams clamps it by the live per-file budget - WithGrantedStreams carries the actual split through window, prefetch, and diagnostics - ReadStream and prefetch record one stream; adaptive bulk recomputes the grant - update tests, docs, and OpenSpec tasks 5.6/5.7
Quark stores content hashes in lowercase; an uppercase SHA-1 makes /file/update/hash reject content the provider already knows with "sha1 非法", which breaks deduplicated uploads. Send lowercase md5/sha1. Quark's CDN answers an out-of-range Range request with the whole object, so clamp reads to the entry and return an empty stream at EOF. The crypt wrapper reads ciphertext through the raw driver but passed the plaintext size, so drivers that clamp to entry.Size truncated the final block and decryption failed. Pass the encrypted size instead.
The delete scheduler drops a key before running its callback, so an already-fired remote delete looked idle. Probes that wait for idle returned before the provider delete finished, and an immediate unmount left test directories behind on slow providers such as Koofr. Expose active deletes in the debug snapshot and include them in the probe idle wait. Harden the real-cloud probes at the same time: accept a workspace root id, keep the control socket path short, use plain-text fixture names for providers that block .bin downloads, and fall back to cache counters when the bounded read-event ring evicts traversal evidence.
Add --path to run the read benchmark against a file that is already on the mount. This keeps large-file baselines read-only: no fixture upload, no cleanup, and no remote writes. Sequential samples cross-check their content hash, and seek probes skip the fixture byte pattern when the content is unknown.
The encrypted wrapper dropped parallel_range and internal_parallel_read, so an encrypted mount always looked single-stream to the read planner and never forwarded the raw driver's ReadProfile. Forward both so encrypted mounts keep the backend's read scheduling.
…bject A strict sub-range request that the CDN answers with 200 was returned as if it were the requested range. Return drive.ErrUnsupported so the read layer falls back to a single stream instead of assembling the whole object into one sub-range. The non-strict path keeps the 200 fallback.
The migration worked functionally (streams=3, no fallback) but measured 25-30% lower cold sequential throughput than a single stream on the same file. Record the control experiment and keep Quark on internal_parallel_read.
Add --streams and --window-chunks so read benchmarks can compare configurations on one mounted VFS instead of rebuilding and remounting for every profile. The override is measurement-only: it applies to the read planner for the duration of the probe call, is cleared on return, and is not exposed through config or production read paths.
Live measurements showed concurrent sub-ranges slower than one 4 MiB range on all three providers: fixed per-request latency dominates and does not scale with concurrency. Set ReadProfile BulkStreams=1 and keep parallel_range declared so a measured profile can re-enable it later.
- classify not-found upload failures as permanent, so a deleted target does not accumulate retries forever - stop requeueing after maxUploadRetries consecutive transient failures - classify sftp staging errors so a missing target reports not found
…e watchdog Replace the fixed API timeouts for Koofr, WebDAV, and 139 uploads with a driverutil upload budget derived from the source size plus an idle watchdog that cancels a request only when the transport stops consuming the body. Multi-GB uploads could not finish within the shared 15-minute, 30-second, and 60-second timeouts and restarted from zero on every attempt.
- add [mounts.read_profile] with streams/window_chunks, inheriting the driver profile for unset fields - validate ranges and reject parallel streams for drivers without parallel_range or with internal_parallel_read - resolve the effective profile once in core and pass it to VFS reads and direct copy/share-fetch paths - expose the effective profile and its source in the mount debug snapshot - keep Koofr/Drime defaults single-stream; document the opt-in and record the cross-environment benchmark in OpenSpec
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
关联 Issue
Closes #21
改了什么 / 为什么
并行读不是普适加速：真实云盘实测下 Koofr、Drime、123 单流更快（#21 的对比数据也确认默认 1 MiB 子块会退化），所以驱动默认仍保持单流，同时把并行读做成每个 mount 可显式开启的读策略。
bulk/interactive/probe贯穿 VFS、读调度和驱动；只有bulk与自适应顺序流可以并行，交互读与探测保持单流。ReadProfile；新增[mounts.read_profile]的streams/window_chunks，优先级为 mount 覆盖 > 驱动 profile > 内置默认，驱动 limiter 仍可下调。streams为 0/1–32，window_chunks为 0/1–64；streams > 1要求parallel_range，internal_parallel_read驱动拒绝多流。probe test read的 streams/window 覆盖，以及probe bench read/probe bench upload，用于先测 provider 上限再改 profile。full-config.md、support-drivers.md的“并行读”列、troubleshooting、architecture、driver-development、debug、FAQ，OpenSpec 变更已归档。RACE_P可覆盖），测试集合与-race覆盖不变。测试证据
scripts/ci/check-all.sh通过（Go 1.27.3，输出== local CI check passed ==）。TestRunDriverReadBenchmark读取 2 MiB 内存数据低于 Windows 单调时钟粒度，time.Since为 0 导致吞吐为 0；测试驱动增加可观测延迟，断言保持不变。openspec/changes/archive/2026-11-08-add-read-profile-override/benchmark-baseline.md。风险与回滚
未设置
[mounts.read_profile]时行为由驱动 profile 和能力决定；Koofr、Drime、123 默认单流不变，interactive/probe始终单流。该配置只影响bulk读，改完需重启qrypt mount。回滚本 PR 即可退回原读路径。检查清单
scripts/ci/check-all.sh通过