Anchor metadata pattern matching so foreach paths do not prefix-match

[nileshpatil6]

Addresses the local-provider side of #3341. The service-provider side is handled by #3357.

LocalMetadataProvider.filter_tasks_by_metadata matched metadata values with regex.match, which anchors only at the start. Task.parent_tasks / child_tasks build their pattern from foreach-execution-path, so once a foreach has 11 or more items the pattern middle:1 also matched the values middle:10 and middle:11, and the client resolved the wrong tasks. With spin this turns into a hard failure ("not a join step but gets multiple inputs").

The fix switches that one call to regex.fullmatch, so a pattern has to match the whole value.

The reason fullmatch is safe here is that the patterns this function receives are already written to cover the part of the path they care about. From metaflow/client/core.py:

• an exact ancestor path, e.g. middle:1 — should match only that task, which is what this fixes
• a descendant pattern built as f"{current_path},.*" — the trailing .* still consumes the rest of the value, so nested foreach lookups are unaffected
• ".*" for the match-all case — unaffected, and it is short-circuited before the regex anyway

I checked the values that a 12 item foreach produces:

pattern	re.match (before)	re.fullmatch (after)
middle:1	middle:1, middle:10, middle:11, middle:1,inner:0	middle:1
middle:1,.*	middle:1,inner:0	middle:1,inner:0
.*	all	all

Only the over-matching case changes.

The service metadata provider passes the pattern to the backend instead of matching locally, so this is the only client-side matcher affected.

Added test_filter_tasks_by_metadata_does_not_match_prefixes in test/unit/test_local_metadata_provider.py, covering the exact path, the prefixed sibling, a nested descendant pattern and match-all. Verified it fails before the change (the middle:1 lookup returns three extra tasks) and passes after; the existing test in that file still passes.

[greptile-apps]

Greptile Summary

The PR anchors local metadata pattern matching to the complete metadata value, preventing foreach indices such as middle:1 from matching middle:10 or middle:11.

• Replaces regex.match with regex.fullmatch in the local metadata provider.
• Adds regression coverage for exact paths, prefixed siblings, nested descendants, and match-all patterns.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
metaflow/plugins/metadata_providers/local.py	Narrows metadata filtering to whole-value matches while preserving explicit descendant wildcard patterns.
test/unit/test_local_metadata_provider.py	Adds focused regression coverage for foreach prefix collisions and supported wildcard cases.

_{Reviews (5): Last reviewed commit: "Merge branch 'master' into fix-foreach-p..." | Re-trigger Greptile}

[winklemad]

Thanks for picking this up. I checked the reasoning rather than taking it on trust, and the three pattern shapes are exactly as you describe — client/core.py builds pattern = current_path or ".*", the bare ".*", and f"{current_path},.*" at both the parent_tasks (~1266) and child_tasks (~1329) sites. fullmatch is safe for all three: the descendant form's trailing .* still consumes the remainder, and ".*" is short-circuited before the regex anyway. So the one-call change is right and I don't think it needs widening.

One thing I could not settle from this repo, which might matter for how the fix is framed:

ServiceMetadataProvider.filter_tasks_by_metadata (metadata_providers/service.py:328) doesn't do the matching locally — it forwards pattern as a query param to the metadata service and lets the server filter (query_params["pattern"] = pattern, then GET .../filtered_tasks?...). It has the same .* short-circuit, so the two providers agree on that case, but the anchoring behaviour for an exact path like middle:1 depends on whatever the service does with that pattern, and that code isn't here.

If the service side anchors the same way regex.match did, a user on the service provider would still see middle:1 pull in middle:10 and middle:11 after this merges, and the local and service providers would disagree. Probably a question for a maintainer rather than something to pull into this PR.

For context, I filed #3341 — glad it's getting fixed.

[nileshpatil6]

Thanks for checking it rather than taking it on trust, and for filing #3341 in the first place.

I went looking for the service side, and I think the question is answerable. The filtering happens in Netflix/metaflow-service, services/data/postgres_async_db.py, in get_filtered_task_pathspecs:

if pattern:
    conditions.append("regexp_match(value, %s) IS NOT NULL")
    values.append(pattern)

regexp_match() in PostgreSQL is an unanchored search: it returns the first substring matching the pattern anywhere in the value. So the service provider is not merely anchored the way re.match was, it is looser still. For pattern = "middle:1":

	middle:1	middle:10	xmiddle:1
old local, re.match	match	match (the bug)	no match
service, regexp_match	match	match	match
new local, re.fullmatch	match	no match	no match

So yes, the two providers diverge after this merges, and they already diverged before it on the leading-substring case. Anchoring server-side would mean wrapping the pattern, roughly value ~ ('^(?:' || %s || ')$'), which keeps the f"{current_path},.*" descendant form working since the trailing .* still consumes the remainder.

I agree that does not belong in this PR: it is a different repo, it is a behaviour change for anyone relying on the current substring semantics, and it wants a maintainer's call on whether to anchor the service or relax the client. Happy to open an issue on metaflow-service with the above if a maintainer thinks it is worth tracking.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (master@ddef84a). Learn more about missing BASE report.

Additional details and impacted files

@@            Coverage Diff            @@
##             master    #3342   +/-   ##
=========================================
  Coverage          ?   31.00%           
=========================================
  Files             ?      382           
  Lines             ?    52717           
  Branches          ?     9303           
=========================================
  Hits              ?    16346           
  Misses            ?    35163           
  Partials          ?     1208           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.