Skip to content

feat: add IngestSpec value object (#59) - #63

Merged
underchemist merged 2 commits into
developfrom
59-ingestspec
Jul 7, 2026
Merged

feat: add IngestSpec value object (#59)#63
underchemist merged 2 commits into
developfrom
59-ingestspec

Conversation

@underchemist

@underchemist underchemist commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Implements #59 — the first slice of the end-to-end ingest workflow (parent #56).

Adds IngestSpec, a parsed, immutable ingestion specification that is the typed entry point for building an ingestion from Python, and teaches IngestResource.create to accept it.

What's included

  • prescient_sdk/ingest_spec.pyIngestSpec:
    • from_file(path) / from_dict(spec) parse the spec (via yaml.safe_load).
    • .spec returns a deep copy (immutable); .to_bytes() serializes back to YAML with key order preserved.
    • local_locations() reports locations whose path is not an s3:// URI
  • IngestResource.create now accepts Path | str | bytes | IngestSpec. For an IngestSpec it raises ValueError (naming the offending locations) if any location is still local; otherwise it submits .to_bytes(). Raw bytes/str/Path inputs are unchanged.
  • pyyaml promoted to a runtime dependency; IngestSpec exported from prescient_sdk.

Staging (uploading local sources + rewriting their paths) is intentionally out of scope here — that's #61 (blocked by this and #60).

Acceptance criteria

  • IngestSpec.from_file / from_dict parse a spec; .to_bytes() round-trips
  • IngestResource.create accepts an all-s3:// IngestSpec and submits the serialized spec
  • create raises ValueError naming the offending location when an IngestSpec has a non-s3:// location path
  • pyyaml declared as a runtime dependency; IngestSpec exported from prescient_sdk
  • Unit tests cover parse, round-trip, create-with-IngestSpec, and the guardrail

Testing

  • uv run pytest — 124 passed
  • uv run ruff check prescient_sdk/ tests/ — clean; files formatted with ruff format

Closes #59

🤖 Generated with Claude Code

Introduce IngestSpec, a parsed, immutable ingestion specification that is
the typed entry point for building an ingestion from Python.

- IngestSpec.from_file / from_dict parse the spec YAML; .spec returns a
  copy (immutable); .to_bytes() serializes it (order-preserving) for
  submission; local_locations() reports locations whose path is not an
  s3:// URI.
- IngestResource.create now accepts Path | str | bytes | IngestSpec. Given
  an IngestSpec it raises ValueError (naming the offending locations) if any
  location is still local, otherwise submits .to_bytes(). Raw inputs keep
  their passthrough behavior.
- Promote pyyaml to a runtime dependency; export IngestSpec from the package.

Staging (uploading local sources and rewriting their paths) is a separate
slice (#61); this slice covers parsing and submission only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@michaelconnor00

Copy link
Copy Markdown
Collaborator

Reviewing this is bringing up a question: Should the IngestResource.create() take a file path (or bytes), or should it only take an IngestSpec, which will read from file, or take bytes??

I had also envisioned the IngestSpec would have a Pydantic model for validating the spec, which then implies the answer to the above is IngestResource.create() should only take IngestSpec and an argument.

@michaelconnor00

Copy link
Copy Markdown
Collaborator

I think it makes sense to create a seperate issue for adding the pydantic model and syncing it with the ingest API.

However for this PR, we should change IngestResource.create() to only take IngestSepc. Unless you have thoughts otherwise.

@underchemist

Copy link
Copy Markdown
Contributor Author

Sure that sounds good

…d it only accepts IngestSpec. IngestSpec now has a method.
@underchemist
underchemist merged commit e82bbd2 into develop Jul 7, 2026
4 checks passed
@underchemist
underchemist deleted the 59-ingestspec branch July 7, 2026 20:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants