Skip to content

perf(skills): batch remote skill discovery reads#31507

Draft
nornagon-openai wants to merge 1 commit into
mainfrom
nornagon/codex/batched-remote-skill-reads
Draft

perf(skills): batch remote skill discovery reads#31507
nornagon-openai wants to merge 1 commit into
mainfrom
nornagon/codex/batched-remote-skill-reads

Conversation

@nornagon-openai

@nornagon-openai nornagon-openai commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Summary

Add a generic batched remote-filesystem read primitive and use it in the legacy skill loader so remote thread startup no longer fans out into one RPC per discovered skill file.

The change adds ExecutorFileSystem::read_files and the generic fs/readFiles exec-server RPC with per-path success/error results, a 128-path request cap, and a 64 MiB aggregate decoded-response cap. Oversized files fail individually without consuming the remaining batch budget, so later small files can still succeed. Mixed-version clients fall back to individual fs/readFile calls when an older exec server does not know the new method.

Skill discovery now uses the existing bounded server-side fs/walk inventory. Walk can report canonical paths while following directory symlinks, preserving canonical skill identity without a client-side canonicalize RPC per skill on current servers; older walk responses retain the compatibility fallback. Hidden descendants are pruned the same way as the legacy scanner, while the root .agents/skills directory itself remains discoverable.

After inventory, the loader reads discovered SKILL.md files and candidate agents/openai.yaml metadata through the batched primitive, then keeps parsing, warnings, precedence, ordering, and namespace policy in Codex core. Plugin namespace manifest batching is deliberately left to the per-root namespace work in #31348 rather than duplicating that change.

Why

Browser-created remote-environment thread/start has a 30 second client timeout. The legacy loader performed remote directory discovery and file reads as many serialized filesystem RPCs, so cold skill discovery could spend most of that budget on network latency instead of parsing.

Validation

  • just test -p codex-core-skills (112 passed)
  • just test -p codex-exec-server-protocol (5 passed)
  • just test -p codex-exec-server batched_reads_return_per_path_errors_and_reject_too_many_paths remote_file_system_batches_paths_and_keeps_per_path_errors
  • just test -p codex-exec-server file_system_walk_handles_directory_symlinks file_system_walk_returns_a_bounded_tree
  • just fix -p codex-file-system -p codex-exec-server-protocol -p codex-exec-server -p codex-core-skills
  • just fmt
  • git diff --check

just test -p codex-exec-server was also run: 271 passed, 34 failed, and 2 skipped. The failures are the existing macOS sandbox integration cases in this runner, all failing with sandbox-exec: sandbox_apply: Operation not permitted; the new batched-read tests passed.

🧵 Codex Thread

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant