Skip to content

test(file-provider): add an end-to-end benchmark harness - #10733

Open
juliusvaart wants to merge 2 commits into
nextcloud:stable-34.0from
juliusvaart:macos/vf/benchmarks
Open

test(file-provider): add an end-to-end benchmark harness#10733
juliusvaart wants to merge 2 commits into
nextcloud:stable-34.0from
juliusvaart:macos/vf/benchmarks

Conversation

@juliusvaart

Copy link
Copy Markdown

Summary

Performance claims about this package have so far been made from figures taken on one person's account, which reproduce nowhere, or from a mocked server, which measures the mock. This adds a harness that measures the package's own code against a real Nextcloud over real HTTP, and that can run the same measurement on an unmodified baseline revision and print the difference.

docker compose -f Benchmarks/docker-compose.yml up -d
Benchmarks/provision.sh
Benchmarks/compare.sh upstream/stable-34.0

compare.sh builds a detached worktree at the baseline ref, copies only this directory into it, builds both revisions in release, runs both against the same server and prints a table per scenario. The baseline stays the baseline: nothing crosses into that worktree except the measuring instrument.

Everything below the File Provider observer is production code — real NextcloudKit, real WebDAV, real XML parsing, the package's own Enumerator and ingestion, and a real Realm database. The observers themselves are stand-ins, because NSFileProviderChangeObserver and NSFileProviderEnumerationObserver are supplied by the framework inside a running extension and cannot be obtained outside one; the harness implements what the framework does, draining moreComing batches until the enumerator reports the last one, and timestamps what arrives. CountingRemoteInterface replaces nothing: it forwards every call to the real interface and records how many requests were issued and how many overlapped.

Four scenarios, reporting durations and counts side by side. A duration alone cannot separate code that got smarter from a server that was warmer; the counts are what a claim can be held to, and they do not move between machines.

working-set-scan one full working-set walk, nothing changed remotely
working-set-scan-change the same walk with one file genuinely changed
item-enumeration-cold first enumeration of a wide directory
item-enumeration-repeat a second enumeration of the same unchanged directory

The reference server is Nextcloud 31 in a container, on SQLite and without Redis: the benchmarks measure the client, and a heavier server stack only adds variance. provision.sh writes the fixture into the data directory and indexes it with occ files:scan, which is far faster than several thousand WebDAV uploads and leaves the same server-side state. NFPK_BENCH_URL, NFPK_BENCH_USER and NFPK_BENCH_PASSWORD point the harness at any other instance.

This commit changes no production code. What the harness cannot see — the framework's own scheduling and retry behaviour — is documented in the doc:PerformanceBenchmarks article, together with the log lines to count for it from an installed build.

Assisted-by: Claude Code:claude-opus-5

TODO

  • Confirm the methodology is what you want before the eight follow-up PRs go up
  • Pin the reference server image by digest — nextcloud:31-apache is a floating tag, so two runs months apart are not strictly the same server
  • Decide whether this can run in CI at all: GitHub macOS runners have no Docker, and the package is macOS-only so a Linux job cannot build it. Local/manual for now
  • Verify the harness against server 32 and 33 (only 31 exercised so far)

Checklist

AI (if applicable)

Performance claims about this package have so far been made from figures taken
on one person's account, which reproduce nowhere, or from a mocked server, which
measures the mock. This adds a harness that measures the package's own code
against a real Nextcloud over real HTTP, and that can run the same measurement
on an unmodified baseline revision and print the difference.

    docker compose -f Benchmarks/docker-compose.yml up -d
    Benchmarks/provision.sh
    Benchmarks/compare.sh upstream/stable-34.0

`compare.sh` builds a detached worktree at the baseline ref, copies only this
directory into it, builds both revisions in release, runs both against the same
server and prints a table per scenario. The baseline stays the baseline: nothing
crosses into that worktree except the measuring instrument.

Everything below the File Provider observer is production code — real
`NextcloudKit`, real WebDAV, real XML parsing, the package's own `Enumerator`
and ingestion, and a real Realm database. The observers themselves are
stand-ins, because `NSFileProviderChangeObserver` and
`NSFileProviderEnumerationObserver` are supplied by the framework inside a
running extension and cannot be obtained outside one; the harness implements
what the framework does, draining `moreComing` batches until the enumerator
reports the last one, and timestamps what arrives. `CountingRemoteInterface`
replaces nothing: it forwards every call to the real interface and records how
many requests were issued and how many overlapped.

Four scenarios, reporting durations and counts side by side. A duration alone
cannot separate code that got smarter from a server that was warmer; the counts
are what a claim can be held to, and they do not move between machines.

  working-set-scan          one full working-set walk, nothing changed remotely
  working-set-scan-change   the same walk with one file genuinely changed
  item-enumeration-cold     first enumeration of a wide directory
  item-enumeration-repeat   a second enumeration of the same unchanged directory

The reference server is Nextcloud 31 in a container, on SQLite and without
Redis: the benchmarks measure the client, and a heavier server stack only adds
variance. `provision.sh` writes the fixture into the data directory and indexes
it with `occ files:scan`, which is far faster than several thousand WebDAV
uploads and leaves the same server-side state. `NFPK_BENCH_URL`,
`NFPK_BENCH_USER` and `NFPK_BENCH_PASSWORD` point the harness at any other
instance.

This commit changes no production code. What the harness cannot see — the
framework's own scheduling and retry behaviour — is documented in the
<doc:PerformanceBenchmarks> article, together with the log lines to count for it
from an installed build.

Signed-off-by: Julius van der Vaart <julius@vanderva.art>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Assisted-by: Claude Code:claude-opus-5
`nextcloud:31-apache` is a floating tag. Two comparison runs months apart would
otherwise measure against different servers while reporting the difference as a
property of the client.

Signed-off-by: Julius van der Vaart <julius@vanderva.art>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Assisted-by: Claude Code:claude-opus-5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant