Yields Data Pipeline

Overview

This project aims to create a data pipeline for yield farming data from the Yields API.

Terminal Commands

Install dependencies

pipenv install

Run black to format the code

pipenv run black

starting pipenv shell

pipenv shell

Run the pipeline

pipenv run python -m pipeline

Docker- to start the postgres container

docker compose up

Postgres- to access the postgres container

docker exec -it postgres psql -U dbuser -d mydb

To stop the postgres container

docker compose down

Initialize Alembic

alembic init migrations

Create initial migration

alembic revision --autogenerate -m "Create pools and pools_historical tables"

Apply migration

alembic upgrade head

Repo Structure

your_project/ ├── alembic/ │ ├── versions/ │ ├── env.py │ └── script.py.mako ├── app/ │ ├── init.py │ ├── models/ │ │ ├── init.py │ │ ├── base.py # Contains Base model and common configurations │ │ ├── mixins.py # Common model mixins (e.g., TimestampMixin) │ │ ├── pools.py # Pool model │ │ └── pools_historical.py # PoolHistorical model │ ├── schemas/ │ │ ├── init.py │ │ └── pools.py # Pydantic models for validation │ ├── database.py # Database connection handling │ └── config.py # Configuration settings ├── migrations/ │ └── env.py ├── tests/ │ ├── init.py │ └── test_models.py ├── .env ├── .gitignore ├── alembic.ini ├── requirements.txt └── README.md