ThrowbackMemo

Private, cost-efficient backend for storing, organizing, and sharing personal photos and videos. Built as a progressive learning project that evolves from a minimal FastAPI service to a production-style system with persistence, background processing, observability, and deployment.

Goals

• Ship a clean, typed Python backend you can deploy cheaply.
• Learn modern backend practices by building real features.
• Support your personal portfolio site with a real app behind it.

---

Tech Stack

Language: Python 3.12 Framework: FastAPI DB: PostgreSQL (SQLAlchemy 2.0 async, Alembic) Cache & Jobs: Redis, Celery Events (later): Kafka Object Storage: MinIO locally, S3/R2 in prod Testing & Quality: pytest, mypy, Ruff, Black, isort, pre-commit Packaging & Dev: Poetry, Docker Compose Observability: OpenTelemetry, Prometheus, Grafana CI: GitHub Actions

---

Learning Phases

Phase 1 — Foundation: FastAPI, Tests, Tooling

Outcomes

• Minimal FastAPI service with typed endpoints and Pydantic models.
• pytest suite with strict typing and pre-commit hooks (Black, Ruff, isort, mypy, pytest).
• CI on GitHub Actions mirroring local checks.

---

Phase 2 — Persistence: SQLAlchemy + Alembic + Postgres

Focus

• Async SQLAlchemy models and Alembic migrations.
• Core entities and endpoints:
  • Album: id, title, description
  • MediaAsset: id, album_id, media_type (photo|video), original_filename, mime_type, size_bytes, storage_key, width, height, duration_sec
  • Endpoints: create/list/get/update albums; create/list assets (metadata only).
• Integration tests against Postgres via Docker.

---

Phase 3 — Background Jobs & Object Storage

Focus

• S3-style storage (MinIO locally; S3/R2 in prod).
• Presigned upload URLs for direct browser → storage uploads.
• Celery worker pipeline:
  • Ingest uploaded files, probe metadata (e.g., image size, video duration).
  • Generate thumbnails (photos) and transcodes (videos).
• Idempotent tasks, retries, and progress flags.

---

Phase 4 — Delivery

Focus

• Docker multi-stage images for API, worker, and migrations (see infrastructure/Dockerfile).
• Local orchestration with Docker Compose: make db-up, make worker-up, make run for hot-reload API; make migrate to apply Alembic revisions.
• VPS deploys land in /srv/throwback-{APP_ENV} per environment to keep staging and production isolated.
• Delivery targets: compose stack for cheap VPS, or push images to Fly.io/Railway.
• Managed services: Neon/Supabase for Postgres, S3/R2 for object storage.
• Edge: Cloudflare DNS/TLS, Pages for the portfolio frontend.

---

Phase 5 — Platform Hardening (Auth & API Cleanup)

Focus

• Solidify auth/signup/login and JWT handling.
• Enforce rate limits and per-user quotas with consistent error responses.
• Clean API surface: consistent schemas, validation, and background task handoffs.

---

Phase 6 — Observability

Focus

• OpenTelemetry traces across API → DB → Celery.
• Prometheus metrics and Grafana dashboards.
• Request/task log correlation and golden signals (RPS, latency, errors, queue length).

---

Phase 7 — Front-end

Focus

• TBA

---

Managed Login & Guest Uploads (Roadmap)

Goal: Allow authenticated guests to upload into designated albums (e.g., “Guests’ Photos”) with proper access control.

Approach

• Managed auth (Clerk/Auth0/Cognito) to avoid password handling.
• Roles: owner (you), contributor (guests who can upload to specific albums), viewer (read-only).
• Entities: User, AlbumMember (album-level role binding), ShareLink (time-bounded public view).
• Flow: guest logs in → requests presigned upload for an assigned album → uploads → ingest task attaches asset to that album.

This can be introduced incrementally after Phase 2 or alongside Phase 3 without large refactors.

Current auth groundwork (in progress)

• All API routes now live under /api/...; album/asset routes require Authorization: Bearer <token>.
• Endpoints: POST /api/auth/signup (creates user + returns access token) and POST /api/auth/login (returns new token).
• Tokens are JWTs (HS256) with stored JTIs/expirations for revocation checks in the DB.
• Public exceptions: /api/health, /docs, /openapi.json.
• Next: move to managed auth, add refresh tokens, roles/ACLs, rate limiting, and guest upload permissions.

Postman collection

• Collection: postman/throwbackmemo.postman_collection.json
• Environments: postman/environments/*.postman_environment.json (local/staging/prod templates). Set api_base_url, auth_email, auth_password, and optionally access_token (Auth → Login/Signup will store it automatically).
• Requests cover health, auth, albums, and assets (upload uses /api/assets/upload; select a file manually and set album_id).
• Legacy curl examples live in tests/API_LOCAL_TEST_GUIDANCE.md.

---

Development Quality

• Pre-commit: automatic formatting, linting, typing, and quick tests on each commit.
• VS Code: Ruff + Black on save; optional mypy extension for inline type feedback.
• Makefile (optional): make run, make test, make lint, make type, make migrate.
• Docs: docs/code-quality-refactor.md (cleanup guardrails), docs/observability-roadmap.md (Phase 6 plan), docs/multi-tenancy-media-roadmap.md (tenant/media plans), docs/deployment-roadmap.md (VPS vs k8s/Helm path), docs/ci-optimization.md (skip CI/deploy on docs-only changes).

---

Environment files

• Local development uses a .env in the repo root (ignored by git); running docker compose -f infrastructure/docker-compose.yaml ... from the repo root will pick it up automatically, or pass --env-file .env explicitly if you prefer.
• Each deploy target lives in its own folder (e.g., /srv/throwback-production, /srv/throwback-staging), each carrying a plain .env so filenames stay simple now that environments are separated.
• CI deploy writes that per-folder .env before running docker compose --env-file /srv/throwback-<env>/.env ... on the VPS; reuse the same pattern for manual runs.
• REDIS_PASSWORD is required for staging/prod; the dev .env ships with a default for local use only. The dev compose binds Redis to 127.0.0.1 and requires that password; avoid publishing Redis to the internet.
• Auth config: set AUTH_SECRET_KEY (required for staging/prod) and optionally ACCESS_TOKEN_TTL_MINUTES (default 60).
• Rate limiting and quotas: tune REQUEST_RATE_LIMIT_PER_MINUTE (default 60) and DAILY_UPLOAD_LIMIT_BYTES (default 1 GB) as needed per environment.

Redis usage

• Request throttling via fastapi-limiter: per-user (or IP if unauthenticated) caps using REQUEST_RATE_LIMIT_PER_MINUTE.
• Per-user daily upload quota using Redis counters keyed by date and user (DAILY_UPLOAD_LIMIT_BYTES).
• Short-lived caching for album/asset list responses.
• Celery broker and result backend.

---

Philosophy

Build small, observable components with clear contracts. Keep feedback loops tight: test, type-check, and lint continuously.

---

Status

• Phase 1 ✅ complete (FastAPI + tests/tooling)
• Phase 2 ✅ complete (Postgres + Alembic migrations + schema)
• Phase 3 ✅ complete (storage + ingestion tasks)
• Phase 4 ✅ complete (delivery hardening and prod paths)
• Phase 5 ✅ complete (basic user, login auth, api cleanup)
• Phase 6 🚧 in progress (observability/logging dashboards/security)
• Phase 7 ⏳ not started (frontend)