Skip to content

developer_guide.md

Latest commit

 

History

History
217 lines (173 loc) · 6.55 KB

developer_guide.md

File metadata and controls

217 lines (173 loc) · 6.55 KB
Raw

Developer Setup Guide

Introduction

This document provides instructions for developers on how to set up their local development environment for Route66

Pre-requisites

  • Access to the test server which can launch docker container.
  • Docker and Docker Compose installed
  • Set up environment. You can create a protected environment file or just add the following to your ~/.bashrc file so the environments are ready whenever you want to set up the testing server.
export LDAP_BIND_PASSWORD=<LDAP_BIND_PASSWORD>
export SECRET_KEY=<SECRET_KEY>
export DB_USER=<DB_USER>
export DB_PASS=<DB_PASS>
export DATABASE_URL=<DATABASE_URL>

Set up reload mode for debugging on the testing server.

  1. Clone the repository:
git clone https://xxx/be.app.route66.git && cd be.app.route66 && git pull
  1. Set up the database container and development environment (this need to be done at least) Most of time, developer is working on source code development, they don't really need to touch the database container, so it is recommend to set up database container once and keep it running during the development cycle. However, debugging a fast api container is difficult, so you shall close the fast api container and use the conda environment for testing during development.

set up testing environment by docker container

# close fast api container
docker stop fastapi_container && docker rm fastapi_container

# set up both container for database and fastapi
docker-compose --env-file ~/.env.list build --no-cache
docker-compose --env-file ~/.env.list up -d
# if this is the first time you create this container, you might want to load testing data and user settings
docker exec fastapi_container sh -c "cd /usr/src/route66/tests && python load_dummy_data.py demo_setting.json demo_setting_schema.json"

If you want to set up testing environment in reload mode

#create conda environment for development, but you will still need a postgres docker container running in the back
conda create -n route66 python=3.11 && conda activate route66 && pip install -r requirements.txt && conda install postgresql
cd ~/be.app.route66 && uvicorn route66.main:app --reload --host 0.0.0.0 --port 9000

Set up debug mode in vscode

Create a launch.json file:

  • Open your VS Code command palette (Ctrl+Shift+P or Cmd+Shift+P).
  • Type "Python: Create launch.json file" and select it.
  • Choose "FastAPI" as your launch configuration type, copy following as launch.json
{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "FastAPI: route66.main:app",
            "type": "debugpy",
            "request": "launch",
            "module": "uvicorn",
            "args": [
                "route66.main:app",
                "--reload",
                "--host",
                "0.0.0.0",
                "--port",
                "9000"
            ],
            "justMyCode": false  // Optional: debugging beyond your own code
        }
    ]
}

Launch test

docker build -t route66 . && docker run --env-file ~/.env.list -it --rm route66 python -m pytest --collect-only

Test if data can be loaded using dummy data:

docker exec -it fastapi_container /bin/bash python tests/test_database_connection.py python tests/load_dummy_data.py

clean up all existing data

psql $DATABASE_URL DROP SCHEMA public CASCADE; CREATE SCHEMA public;

Test the data in production

export DATABASE_URL=postgresql://user:pass@localhost:5432/db psql $DATABASE_URL

Task submission

python submission_task_and_inputs.py --dataset dataset_example.json --task task_minimal.json --type input --token xxx --schema task_json_schema python submission.py xxx --json_file task_minimal.json

launch dev on vm

conda activate route66 cd ~/repos/be.app.route66 uvicorn route66.main:app --reload --host 0.0.0.0 --port 9000

modify the database

docker exec -it postgresdb /bin/bash psql <DATABASE_URL>

mofidy enum table

example like below

step 1 is to replace the existing table with new states

ALTER TYPE appstate ADD VALUE 'ACTIVATE'; ALTER TYPE appstate ADD VALUE 'INACTIVATE'; UPDATE apps SET state = 'INACTIVATE' WHERE state = 'DEVELOPMENT'; UPDATE apps SET state = 'ACTIVATE' WHERE state = 'RELEASE';

step 2 is to create a new enum and replace the old enum with new enum for the table

CREATE TYPE appstate_new AS ENUM ('ACTIVATE','INACTIVATE'); ALTER TABLE apps ALTER COLUMN state TYPE appstate_new USING state::text::appstate_new; DROP TYPE appstate; ALTER TYPE appstate_new RENAME TO appstate;

If gets into problem with alembic

  1. docker exec -it fastapi_container /bin/bash
  2. remove all .py file in versions folder
  3. psql $DATABASE_URL
  4. DROP TABLE alembic_version;
  5. Ctrl+d. mkdir alembic/versions
  6. alembic revision --autogenerate -m "new change"
  7. alembic upgrade head

How to delete duplicated File object with duplicated s3_path and only keep one

-- Step 1: Delete associations from file_sample_association WITH duplicates AS ( SELECT id FROM ( SELECT id, s3_path, ROW_NUMBER() OVER (PARTITION BY s3_path ORDER BY id) AS rnum FROM files ) t WHERE t.rnum > 1 ) DELETE FROM file_sample_association WHERE file_id IN (SELECT id FROM duplicates);

-- Step 2: Delete associations from file_dataset_association WITH duplicates AS ( SELECT id FROM ( SELECT id, s3_path, ROW_NUMBER() OVER (PARTITION BY s3_path ORDER BY id) AS rnum FROM files ) t WHERE t.rnum > 1 ) DELETE FROM file_dataset_association WHERE file_id IN (SELECT id FROM duplicates);

-- Step 3: Delete associations from input_file_association WITH duplicates AS ( SELECT id FROM ( SELECT id, s3_path, ROW_NUMBER() OVER (PARTITION BY s3_path ORDER BY id) AS rnum FROM files ) t WHERE t.rnum > 1 ) DELETE FROM input_file_association WHERE file_id IN (SELECT id FROM duplicates);

-- Step 4: Delete associations from output_file_association WITH duplicates AS ( SELECT id FROM ( SELECT id, s3_path, ROW_NUMBER() OVER (PARTITION BY s3_path ORDER BY id) AS rnum FROM files ) t WHERE t.rnum > 1 ) DELETE FROM output_file_association WHERE file_id IN (SELECT id FROM duplicates);

-- Step 5: Delete duplicate entries from files WITH duplicates AS ( SELECT id FROM ( SELECT id, s3_path, ROW_NUMBER() OVER (PARTITION BY s3_path ORDER BY id) AS rnum FROM files ) t WHERE t.rnum > 1 ) DELETE FROM files WHERE id IN (SELECT id FROM duplicates);