Skip to content

Add @package_sources for sibling source directories - #3351

Merged
talsperre merged 3 commits into
masterfrom
ssrikanth/package-sources
Aug 27, 2026
Merged

Add @package_sources for sibling source directories#3351
talsperre merged 3 commits into
masterfrom
ssrikanth/package-sources

Conversation

@talsperre

@talsperre talsperre commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Summary

Add @package_sources for including sibling or parent source directories in Metaflow code packages.

This supports project layouts where flows and shared application code live in separate directories:

from metaflow import FlowSpec, package_sources

@package_sources("../src/forecasting")
class TrainFlow(FlowSpec):
    ...

Motivating discussion: https://netflix.slack.com/archives/C02116BBNTU/p1785352517019289

Implementation

  • Resolve relative source paths from the flow file directory.
  • Support multiple sources, custom archive paths, and additional file suffixes.
  • Expose the mutator as metaflow.package_sources.
  • Reject unsafe archive paths and missing sources.

Tests

  • 649 passed, 1 skipped, 6 deselected
  • Pre-commit passes

@greptile-apps

greptile-apps Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR adds a package_sources flow mutator for including files and directories outside a flow’s directory in its code package.

  • Resolves relative paths from the flow file’s directory.
  • Supports multiple sources, custom archive names, and configurable suffix filters.
  • Registers the mutator as a top-level Metaflow API and adds focused unit coverage.

Confidence Score: 4/5

The PR does not appear safe to merge because the previously reported whole-source-root import failure remains unresolved.

The default archive mapping still preserves the src directory prefix, while the new importability test avoids that path by packaging the nested forecasting directory directly; remote code using @package_sources("../src") can therefore still fail to import forecasting.

Files Needing Attention: metaflow/plugins/package_sources.py, test/unit/test_package_sources.py

Important Files Changed

Filename Overview
metaflow/plugins/package_sources.py Implements source-spec parsing, archive-path validation, suffix filtering, and package-content generation.
metaflow/plugins/init.py Registers package_sources as a top-level plugin.
test/unit/test_package_sources.py Covers top-level exposure, relative source resolution, archive names, suffixes, validation, and missing sources.

Reviews (3): Last reviewed commit: "Document package_sources usage" | Re-trigger Greptile

Comment thread metaflow/plugins/package_sources.py
saikonen
saikonen previously approved these changes Aug 27, 2026
Comment thread metaflow/plugins/package_sources.py Outdated
@talsperre
talsperre merged commit ddef84a into master Aug 27, 2026
47 checks passed
@talsperre
talsperre deleted the ssrikanth/package-sources branch August 27, 2026 20:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants