Skip to content

feat: add Dockerized ETL ingestion stack#34

Merged
ZeroQuest merged 6 commits into
mainfrom
feat/dockerization
Jun 22, 2026
Merged

feat: add Dockerized ETL ingestion stack#34
ZeroQuest merged 6 commits into
mainfrom
feat/dockerization

Conversation

@ZeroQuest

Copy link
Copy Markdown
Owner

This PR introduces a Dockerized deployment workflow for the ETL pipeline, allowing the ingestion stack to run in a consistent, containerized environment. It also includes supporting refactors to improve pipeline portability, reduce log noise, and reorganize sample data.


Features

Dockerized ETL stack

  • Added Docker support for the ETL ingestion pipeline
  • Containerized application dependencies and runtime environment
  • Simplified local setup and execution through Docker
  • Established a consistent execution environment across development machines

Refactor

Pipeline robustness

  • Improved pipeline behavior to support execution within a containerized environment
  • Updated file handling and runtime assumptions to be Docker-compatible
  • Preserved existing ETL functionality during infrastructure changes

Sample data organization

  • Moved sample datasets into dedicated data source locations
  • Improved separation between application code and ingestion data
  • Simplified management of test and demonstration datasets

Logging improvements

  • Reduced unnecessary verbosity in ETL logging output
  • Improved readability of pipeline execution logs
  • Retained key operational and error information

Code quality

  • Applied Black formatting across the codebase
  • Ensured Ruff linting compliance
  • Cleaned up minor inconsistencies introduced during refactoring

Repository maintenance

Git configuration

  • Added coverage artifacts to .gitignore
  • Prevented generated coverage files from being tracked in version control

Dependencies

  • Added Docker configuration required to run the ETL pipeline in a containerized environment
  • No changes to core ETL processing dependencies

Testing

  • Verified ETL pipeline executes successfully within Docker environment
  • Confirmed existing ingestion workflows remain functional after refactoring
  • Validated sample data remains accessible through updated source locations
  • Ensured logging output remains useful while reducing noise

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment