Anchor service metadata patterns so foreach paths do not prefix-match by winklemad · Pull Request #3357 · Netflix/metaflow

winklemad · 2026-08-31T00:52:44Z

PR Type

Bug fix

Summary

Handles the OSS service-provider side of #3341. Task.parent_tasks and
Task.child_tasks pass foreach execution-path patterns to the metadata
service, whose regex matching is unanchored. An exact path such as middle:1
could therefore also select middle:10 and middle:11.

The local-provider side is handled separately by #3342.

Root Cause

ServiceMetadataProvider.filter_tasks_by_metadata forwarded the raw pattern
to the metadata service:

query_params["pattern"] = pattern

The service applies that pattern as an unanchored regex search.

Fix

Wrap the forwarded pattern in full-string anchors:

query_params["pattern"] = "^(?:%s)$" % pattern

This makes middle:1 exact while preserving the descendant pattern
middle:1,.*. The existing .* match-all case remains short-circuited and
sends no pattern.

Tests

test/unit/test_service_metadata_provider.py covers:

exact path anchoring;
descendant pattern anchoring;
the match-all short circuit.

Run with:

python -m pytest test/unit/test_service_metadata_provider.py -q

Non-Goals

The local metadata provider and its tests are handled by Anchor metadata pattern matching so foreach paths do not prefix-match #3342.
No metadata-service repository change is required; anchoring happens at the
OSS client boundary.

AI Tool Usage

AI tools were used.

Tool disclosed by the author: Claude Code, used to help implement the original
change and tests. The service-only rescope retains that implementation.

…r-select Client.parent_tasks/child_tasks build a filter pattern from a task's foreach-execution-path and query filter_tasks_by_metadata. The local provider matched it with regex.match, which only anchors the start, so an exact path like "middle:1" also matched "middle:10" and "middle:11" once a foreach had 10+ splits, and parent_tasks/child_tasks returned the wrong tasks. Use regex.fullmatch in the local provider, and anchor the pattern the service provider forwards to the metadata service as ^(?:pattern)$ so it matches in full there too (the existing ".*" and "{path},.*" patterns already require the service to treat this as a regex). The ".*" match-all short-circuit and the "{path},.*" descendant form are unchanged. Fixes Netflix#3341

greptile-apps · 2026-08-31T00:55:04Z

Greptile Summary

The PR anchors metadata-service task-filter patterns so exact foreach execution paths no longer select tasks whose indices merely share a prefix.

Wraps non-match-all service patterns with full-string regex anchors.
Preserves the existing match-all behavior by omitting the pattern query parameter.
Adds unit coverage for exact, descendant, and match-all forwarding.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
metaflow/plugins/metadata_providers/service.py	Anchors forwarded task-metadata regex patterns while preserving the match-all short circuit.
test/unit/test_service_metadata_provider.py	Verifies exact and descendant patterns are anchored and match-all remains unfiltered.

_{Reviews (2): Last reviewed commit: "Scope fix to service metadata provider" | Re-trigger Greptile}

nileshpatil6 · 2026-08-31T16:18:18Z

Thanks for the credit and for laying out the overlap so clearly, that was a considerate way to handle it.

The split you proposed works well from my side: #3342 has been open since Aug 16 with the local provider fix (regex.match to regex.fullmatch) and a prefix-matching test that fails on master, and the service-side anchoring here is genuinely the piece it does not cover. So landing #3342 for the local half and scoping this one to the service provider seems like the cleanest path.

@talsperre happy to go with whatever ordering you prefer.

winklemad · 2026-08-31T17:45:31Z

Thanks @nileshpatil6, likewise, your #3342 nailed the local-provider side. Hoping we can get both landed.

codecov · 2026-08-31T23:06:40Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (master@ddef84a). Learn more about missing BASE report.

Additional details and impacted files

@@            Coverage Diff            @@
##             master    #3357   +/-   ##
=========================================
  Coverage          ?   31.02%           
=========================================
  Files             ?      382           
  Lines             ?    52717           
  Branches          ?     9303           
=========================================
  Hits              ?    16353           
  Misses            ?    35153           
  Partials          ?     1211

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

winklemad · 2026-09-01T08:34:53Z

Thanks for the review @talsperre 🙏

…#3342) Addresses the local-provider side of #3341. The service-provider side is handled by #3357. `LocalMetadataProvider.filter_tasks_by_metadata` matched metadata values with `regex.match`, which anchors only at the start. `Task.parent_tasks` / `child_tasks` build their pattern from `foreach-execution-path`, so once a foreach has 11 or more items the pattern `middle:1` also matched the values `middle:10` and `middle:11`, and the client resolved the wrong tasks. With `spin` this turns into a hard failure ("not a join step but gets multiple inputs"). The fix switches that one call to `regex.fullmatch`, so a pattern has to match the whole value. The reason `fullmatch` is safe here is that the patterns this function receives are already written to cover the part of the path they care about. From `metaflow/client/core.py`: - an exact ancestor path, e.g. `middle:1` — should match only that task, which is what this fixes - a descendant pattern built as `f"{current_path},.*"` — the trailing `.*` still consumes the rest of the value, so nested foreach lookups are unaffected - `".*"` for the match-all case — unaffected, and it is short-circuited before the regex anyway I checked the values that a 12 item foreach produces: | pattern | `re.match` (before) | `re.fullmatch` (after) | |---|---|---| | `middle:1` | `middle:1`, `middle:10`, `middle:11`, `middle:1,inner:0` | `middle:1` | | `middle:1,.*` | `middle:1,inner:0` | `middle:1,inner:0` | | `.*` | all | all | Only the over-matching case changes. The service metadata provider passes the pattern to the backend instead of matching locally, so this is the only client-side matcher affected. Added `test_filter_tasks_by_metadata_does_not_match_prefixes` in `test/unit/test_local_metadata_provider.py`, covering the exact path, the prefixed sibling, a nested descendant pattern and match-all. Verified it fails before the change (the `middle:1` lookup returns three extra tasks) and passes after; the existing test in that file still passes. --------- Co-authored-by: Shashank Srikanth <ssrikanth@netflix.com>

winklemad mentioned this pull request Aug 31, 2026

Client parent_tasks/child_tasks resolve wrong tasks when a foreach index is a string prefix of another #3341

Open

Scope fix to service metadata provider

85e071e

talsperre added a commit to nileshpatil6/metaflow that referenced this pull request Aug 31, 2026

Leave service provider fix to Netflix#3357

2cef1fc

talsperre changed the title ~~Match foreach execution paths in full so an index prefix does not over-select (#3341)~~ Anchor service metadata patterns so foreach paths do not prefix-match Aug 31, 2026

talsperre mentioned this pull request Aug 31, 2026

Anchor metadata pattern matching so foreach paths do not prefix-match #3342

Merged

talsperre approved these changes Aug 31, 2026

View reviewed changes

talsperre merged commit 4810787 into Netflix:master Sep 2, 2026
42 checks passed

Navigation Menu

Uh oh!

Anchor service metadata patterns so foreach paths do not prefix-match - #3357

Anchor service metadata patterns so foreach paths do not prefix-match#3357
talsperre merged 2 commits into
Netflix:masterfrom
winklemad:fix/foreach-path-prefix-match-3341

winklemad commented Aug 31, 2026 •

edited by talsperre

Loading

Uh oh!

greptile-apps Bot commented Aug 31, 2026 •

edited

Loading

Important Files Changed

Uh oh!

nileshpatil6 commented Aug 31, 2026

Uh oh!

winklemad commented Aug 31, 2026

Uh oh!

codecov Bot commented Aug 31, 2026

Uh oh!

winklemad commented Sep 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

winklemad commented Aug 31, 2026 • edited by talsperre Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Type

Summary

Root Cause

Fix

Tests

Non-Goals

AI Tool Usage

Uh oh!

greptile-apps Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

nileshpatil6 commented Aug 31, 2026

Uh oh!

winklemad commented Aug 31, 2026

Uh oh!

codecov Bot commented Aug 31, 2026

Codecov Report

Uh oh!

winklemad commented Sep 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

winklemad commented Aug 31, 2026 •

edited by talsperre

Loading

greptile-apps Bot commented Aug 31, 2026 •

edited

Loading