feat: add IngestSpec value object (#59) - #63
Merged
Conversation
Introduce IngestSpec, a parsed, immutable ingestion specification that is the typed entry point for building an ingestion from Python. - IngestSpec.from_file / from_dict parse the spec YAML; .spec returns a copy (immutable); .to_bytes() serializes it (order-preserving) for submission; local_locations() reports locations whose path is not an s3:// URI. - IngestResource.create now accepts Path | str | bytes | IngestSpec. Given an IngestSpec it raises ValueError (naming the offending locations) if any location is still local, otherwise submits .to_bytes(). Raw inputs keep their passthrough behavior. - Promote pyyaml to a runtime dependency; export IngestSpec from the package. Staging (uploading local sources and rewriting their paths) is a separate slice (#61); this slice covers parsing and submission only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Collaborator
|
Reviewing this is bringing up a question: Should the I had also envisioned the IngestSpec would have a Pydantic model for validating the spec, which then implies the answer to the above is |
5 tasks
Collaborator
|
I think it makes sense to create a seperate issue for adding the pydantic model and syncing it with the ingest API. However for this PR, we should change |
Contributor
Author
|
Sure that sounds good |
…d it only accepts IngestSpec. IngestSpec now has a method.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements #59 — the first slice of the end-to-end ingest workflow (parent #56).
Adds
IngestSpec, a parsed, immutable ingestion specification that is the typed entry point for building an ingestion from Python, and teachesIngestResource.createto accept it.What's included
prescient_sdk/ingest_spec.py—IngestSpec:from_file(path)/from_dict(spec)parse the spec (viayaml.safe_load)..specreturns a deep copy (immutable);.to_bytes()serializes back to YAML with key order preserved.local_locations()reports locations whosepathis not ans3://URIIngestResource.createnow acceptsPath | str | bytes | IngestSpec. For anIngestSpecit raisesValueError(naming the offending locations) if any location is still local; otherwise it submits.to_bytes(). Rawbytes/str/Pathinputs are unchanged.pyyamlpromoted to a runtime dependency;IngestSpecexported fromprescient_sdk.Staging (uploading local sources + rewriting their paths) is intentionally out of scope here — that's #61 (blocked by this and #60).
Acceptance criteria
IngestSpec.from_file/from_dictparse a spec;.to_bytes()round-tripsIngestResource.createaccepts an all-s3://IngestSpecand submits the serialized speccreateraisesValueErrornaming the offending location when anIngestSpechas a non-s3://location pathpyyamldeclared as a runtime dependency;IngestSpecexported fromprescient_sdkIngestSpec, and the guardrailTesting
uv run pytest— 124 passeduv run ruff check prescient_sdk/ tests/— clean; files formatted withruff formatCloses #59
🤖 Generated with Claude Code