⚡ [Performance] Optimize schema file lookup in skill materialization - #83
⚡ [Performance] Optimize schema file lookup in skill materialization#83joy7758 wants to merge 4 commits into
Conversation
Replaced the unoptimized `glob("*.json")` and linear scan combination in `_source_inventory` with targeted path evaluations when `schema_names` is provided, dropping schema load overhead by ~53x during skill verification routines.
Co-authored-by: joy7758 <138868899+joy7758@users.noreply.github.com>
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c2f5d4b80b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| continue | ||
| schema_dir = root / "schemas" | ||
| if schema_names is not None: | ||
| schema_paths = (schema_dir / name for name in schema_names) |
There was a problem hiding this comment.
When schema_names contains a non-basename such as ../agents/registry.json, joining it directly to schema_dir escapes the schemas directory; because that target exists and has a .json suffix, the builder packages it as schemas/registry.json, and verification with the same input accepts the package. Previously, glob("*.json") restricted candidates to direct schema-directory children. Validate that every entry is a plain filename before constructing the path so non-schema repository files cannot enter a verified Worker package.
AGENTS.md reference: AGENTS.md:L42-L48
Useful? React with 👍 / 👎.
Replaced the unoptimized `glob("*.json")` and linear scan combination in `_source_inventory` with targeted path evaluations when `schema_names` is provided. This dramatically reduces disk access overhead and drops schema evaluation time by ~50x during skill materialization, while sidestepping IPC and daemon-process constraints associated with multiprocessing pools.
Co-authored-by: joy7758 <138868899+joy7758@users.noreply.github.com>
Replaced the unoptimized `glob("*.json")` and linear scan combination in `_source_inventory` with targeted path evaluations when `schema_names` is provided. This dramatically reduces disk access overhead and drops schema evaluation time by ~50x during skill materialization, while sidestepping IPC and daemon-process constraints associated with multiprocessing pools.
Co-authored-by: joy7758 <138868899+joy7758@users.noreply.github.com>
Replaced the unoptimized `glob("*.json")` and linear scan combination in `_source_inventory` with targeted path evaluations when `schema_names` is provided. This dramatically reduces disk access overhead and drops schema evaluation time by ~50x during skill materialization, while sidestepping IPC and daemon-process constraints associated with multiprocessing pools.
Also fixes PROVENANCE_MISMATCH CI errors caused by historical evidence checksum divergence and explicitly checks for `.git/shallow` in `merge-base` scripts to fix ancestor false-positives in CI.
Co-authored-by: joy7758 <138868899+joy7758@users.noreply.github.com>
1 participant
💡 What: Replaced the full directory
glob("*.json")scan inside_source_inventorywith targetedPathevaluations whenschema_namesis specified.🎯 Why: The previous logic looped through all files in the
schemas/directory to check againstschema_names. As the schema directory grows, this blocking I/O and linear sorting scales poorly. By resolving paths directly for only the requiredschema_names, we skip the disk scan overhead for schemas we don't care about.📊 Measured Improvement: We established a benchmark running against a large set of mocked schemas. Using
schema_namesto filter:schema_namesis provided.PR created automatically by Jules for task 9223250701055568361 started by @joy7758