Skip to content

Create an end to end Ingest workflow that include Upload and Execution #56

Description

@michaelconnor00

The intention of this effort is to provide a seamless local workflow that will allow a user to define the workflow via python(+yaml). Workflow in this context would be define->upload->ingest->cleanup. Initial thoughts are to create a SpecResource, which would represent the spec.yaml, but include upload logic (use existing SDK functions), and once complete substitute the location into the spec to be ingested.

Tasks:

  • Finalize requirements for different users, ie - notebook user, script user, non-interactive machine.
  • Finalize IAM, and bucket prefix. Applications repo will need a new Role that can be assume by web identity. Sort this out with @michaelconnor00
  • Create SpecResource, or what ever name makes sense.
  • IngestResource should be updated to take a SpecResource, where upload needs to run prior to ingest actions. Use the same rich library for live status of upload (I believe the current upload functions already have a status indicator, so maybe rich is not necessary).
  • Ensure test coverage.
  • Update documentation.
  • Create and Push another RC from develop branch.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions