Andrei

[sonder314]

Task 1: Unified Docker Compose

Description: Create a single docker-compose.yml that runs all services together

Mid-term (Apr 20):

• Create unified docker-compose.yml with all services:
  • orchestrator
  • training-worker
  • inference-service
  • dashboard
  • postgres
  • minio
  • mlflow
• Configure shared network
• Configure service names (no localhost)
• Configure environment variables
• Test docker-compose up works

End-term (Apr 21):

• Add persistent volumes for postgres and minio
• Add health checks for all services
• Add restart policies
• Test docker-compose down and docker-compose up
• Test data persists after restart

Deliverables:

• ✅ Single docker-compose.yml file
• ✅ All services running with one command
• ✅ No localhost references between containers
• ✅ Persistent storage configured

---

Task 2: Enable Tests in CI

Description: Enable pytest in GitHub Actions CI pipeline

Mid-term (Apr 21):

• Uncomment pytest in .github/workflows/ci.yml
• Add pytest configuration to pyproject.toml
• Add test requirements to CI
• Test CI runs tests on push
• Fix any failing tests

End-term (Apr 21):

• Add integration test step to CI
• Add test coverage report
• Add test artifacts (screenshots, logs)
• Test CI passes on all branches

Deliverables:

• ✅ Tests run on every push
• ✅ Tests pass on main and dev branches
• ✅ Test coverage report generated

---

Task 3: Add RAM/CPU Limits

Description: Add Docker resource limits to prevent OOM crashes

Mid-term (Apr 21):

• Add memory limits to docker-compose.yml:
  • orchestrator: 512MB
  • training-worker: 1GB
  • inference-service: 512MB
  • dashboard: 256MB
• Add CPU limits to docker-compose.yml:
  • orchestrator: 0.5 CPU
  • training-worker: 1 CPU
  • inference-service: 0.5 CPU
  • dashboard: 0.25 CPU
• Test limits work correctly
• Test services don't crash with limits

End-term (Apr 21):

• Add memory swap limits
• Add OOM kill policy
• Test services handle resource constraints
• Document limits in README

Deliverables:

• ✅ RAM/CPU limits set for all services
• ✅ Services don't crash with limits
• ✅ Limits documented

---

Task 4: Full Integration Test

Description: Create and run end-to-end integration test

Mid-term (Apr 23):

• Create integration test script:
  • Start docker-compose
  • Upload dataset
  • Trigger training
  • Wait for job completion
  • Check job status
  • Trigger inference
  • Check prediction
  • Stop docker-compose
• Test script runs successfully
• Add script to CI

End-term (Apr 23):

• Add test for crash recovery
• Add test for service restart
• Add test for multiple jobs
• Add test for concurrent requests

Deliverables:

• ✅ End-to-end integration test
• ✅ Test runs in CI
• ✅ All tests pass