Skip to content

feat: flatten_stream returns list[Path] (v2.0.4) - #17

Merged
scottdraper8 merged 5 commits into
mainfrom
v2.0.4
Mar 5, 2026
Merged

feat: flatten_stream returns list[Path] (v2.0.4)#17
scottdraper8 merged 5 commits into
mainfrom
v2.0.4

Conversation

@scottdraper8

@scottdraper8 scottdraper8 commented Mar 4, 2026

Copy link
Copy Markdown
Owner

Summary

  • flatten_stream now returns list[Path] (the paths of all files written) instead of None
  • Return value propagates up through writer.close()stream_process()flatten_stream()
  • Fully backwards-compatible — callers that discard the return value are unaffected

Changes

  • Writers (base.py, csv.py, arrow_base.py, avro.py): StreamingWriter.close() abstract signature updated to -> list[Path]; each concrete writer tracks and returns written file paths
  • Streaming (streaming.py): stream_process() captures and returns list[Path] from writer.close()
  • API (api.py): flatten_stream() captures and returns result of stream_process(); docstring updated with Returns: section
  • Tests: Updated assert result is None assertions to verify returned list is non-empty and all paths exist
  • Docs: Updated api.md and streaming.md with new return type and usage examples

Test plan

  • poetry run pytest tests/unit/test_api_flatten.py::TestFlattenStream -v
  • poetry run pytest tests/unit/test_streaming.py -v
  • poetry run pytest (full suite)

Note

Low Risk
Low risk: behavior is additive for callers who ignored the return value, and changes are limited to return-type plumbing and path tracking in streaming writers.

Overview
flatten_stream() now returns a list[Path] of all output files written (rather than None), by propagating the return value from StreamingWriter.close() through stream_process() to the public API.

To support this, streaming writers (CSV, PyArrow/Parquet/ORC, Avro) now track written file paths and return them on close(), and docs/tests were updated accordingly; version is bumped to 2.0.4.

Written by Cursor Bugbot for commit a5f8440. This will update automatically on new commits. Configure here.

@scottdraper8
scottdraper8 merged commit 68a2b79 into main Mar 5, 2026
8 checks passed
@scottdraper8
scottdraper8 deleted the v2.0.4 branch March 5, 2026 17:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant