[#86] feat(blob_storage): support recursive walk for path LIKE predicate#87
Merged
jerryshao merged 2 commits intoApr 17, 2026
Merged
Conversation
…or path LIKE predicate When a path LIKE or path ILIKE predicate is present in a QUERY intent, switch from single-level iterdir() to os.walk() so that patterns crossing directory boundaries (e.g. 'docs/%.txt') can match entries at arbitrary depth. Ignored directories are pruned via os.walk topdown mode to avoid unnecessary I/O. Single-level listing is preserved when no path LIKE predicate is used. Add unit tests covering: - Top-level extension filter (regression) - Subdirectory prefix pattern (docs/%) - Arbitrary-depth nested pattern (a/b/%.csv) - Case-insensitive ILIKE matching - Ignore-pattern directory pruning Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
mchades
force-pushed
the
feat/path-like-recursive-walk
branch
from
April 9, 2026 14:22
2e46274 to
b9d6f9e
Compare
There was a problem hiding this comment.
Pull request overview
Adds recursive filesystem traversal to the local blob storage backend when a path LIKE / path ILIKE predicate is used, aligning query behavior with expected SQL LIKE semantics for patterns that can match across directory boundaries.
Changes:
- Updated
LocalFSBackend._execute_queryto switch to recursive enumeration whenpath LIKE/ILIKEis present. - Added
_has_path_like_predicateand_iter_entries_recursivehelper methods to support conditional recursion with ignore-directory pruning. - Added a new unit-test suite covering recursive LIKE/ILIKE behavior and ignore-pattern pruning.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
src/backends/blob_storage/local.py |
Adds conditional recursive walk for path LIKE/ILIKE queries via new helper methods. |
tests/unit/backends/test_blob_storage.py |
Adds unit tests validating recursive LIKE/ILIKE matching and ignore-pattern pruning behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
|
I have no comments on this PR, let AI to check if there's any logic problem. |
…arden symlink handling - Convert _iter_entries_recursive from returning list[Path] to Iterator[Path] (generator), so the can_early_stop limit optimization works correctly during recursive LIKE walks - Use followlinks=False in os.walk unconditionally to prevent symlinked directories from escaping the source directory or causing infinite cycles (Option A: symlinked dirs appear as leaf entries but are never descended into) - Unify per-level entry ordering: yield files and subdirectories mixed alphabetically by name, matching sorted(iterdir()) semantics - Fix docstring: align Yields/Returns wording with generator type; replace misleading 'sorted order' with 'deterministic traversal order' - Add two new tests: symlinked dir contents not traversed during recursive walk, self-referential symlink loop does not hang Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
mchades
force-pushed
the
feat/path-like-recursive-walk
branch
from
April 17, 2026 06:57
49d3494 to
dc0e154
Compare
Contributor
Author
|
all comments resolved |
jerryshao
approved these changes
Apr 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Added recursive directory traversal support to
LocalFSBackend._execute_querywhen apath LIKEorpath ILIKEpredicate is present. Two new private methods were introduced:_has_path_like_predicate(predicates): recursively checks whether any predicate targetspathwith aLIKEorILIKEoperator._iter_entries_recursive(root): usesos.walk(topdown=True)to enumerate all filesystem entries under a root directory, pruning ignored directories in-place before descending so that no unnecessary I/O occurs.When no
path LIKEpredicate is present, the original single-leveliterdir()path is used unchanged.Why are the changes needed?
Previously,
path LIKE 'docs/%'silently returned zero results because the listing only iterated one directory level, so entries likedocs/guide.mdnever appeared. SQLLIKEsemantics require%to match any character sequence including/, so the recursive walk is the correct behaviour.Fix: #86
Does this PR introduce any user-facing change?
Yes —
QUERYintents with apath LIKEorpath ILIKEpredicate now traverse the resource directory recursively, returning entries at arbitrary depth whosepathmatches the pattern. Queries without such a predicate are unaffected.How was this patch tested?
Five new unit tests in
TestLocalFSBackendPathLike:test_query_path_like_top_level_extension_filter—path LIKE '%.txt'still filters top-level files (regression).test_query_path_like_recursive_subdir_prefix—path LIKE 'docs/%'returns files insidedocs/.test_query_path_like_deep_nested—path LIKE 'a/b/%.csv'matches at arbitrary depth.test_query_path_ilike_case_insensitive—path ILIKE '%.TXT'works case-insensitively.test_query_path_like_respects_ignore_patterns— ignored directories are not traversed.