TOO-MM APP

flask application for too-mm

Initial Setup

Generate a secret key for the application inside too_mm_app directory:

python generate_secret_key.py

This will create or update your instance/.env file with a secure SECRET_KEY for init.py inside too_mm_app.

To build the images:

docker compose --env-file instance/.env build

To start all containers:

docker compose --env-file instance/.env up

To start only one service:

docker compose --env-file instance/.env up <service-name>

Test

Pytest configuration is within the pyproject.toml. There is multiple test suites in the project.

• To run doctest and every tests without the need to have an app and a db setup:

pytest -m "not e2e"

• To run only the tests that need the in-memory sqlite db:

pytest -m "with_db"

• To run the e2e tests:

pdm run_e2e

⚠️ The e2e tests doesn't work outside the use of the above command, the user needs docker compose installed and has built the images defined in the docker-compose file before.

Configuration

The toO-MM app uses two different configuration files. These configuration files should be in the too_mm_app/config folder.

Application configuration

A .env file containing the main configuration to initialize the application, should be within the instance folder of the flask application

# args for docker builds
USR=template
USR_GROUP=template

# defined in tools/setup_FSC_env.sh
PROJECT_GID=0000
PROJECT_UID=0000
KC_PASSWORD=template


# database configuration
POSTGRES_DB=template
POSTGRES_USER=template
POSTGRES_PASSWORD=template
POSTGRES_ROOT_PASSWORD=template
POSTGRES_PORT=template
POSTGRES_HOST=template

# used in the config.py to set the database URL from the environment for the flask app
DATABASE_URL="postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DB}"


# celery configuration
REDIS_HOST=template
REDIS_PORT=0000
CELERY_BROKER_URL="redis://${REDIS_HOST}:${REDIS_PORT}/0"
CELERY_RESULT_BACKEND="redis://${REDIS_HOST}:${REDIS_PORT}/0"
timezone="Europe/Paris"

GCN stream configuration

A .toml file containing the GCN stream configuration The Kafka configuration keys can be found here

The CLIENT part is the credential for the GCN stream. An account should be created before on https://gcn.nasa.gov/.

The GCN_TOPICS part corresponds to the topic the GCN stream listens to. The dictionary associates a topic name (the key) with a class name (the value). It allows the instanciation of a class using the topic name in the code.

The GW_CONFIG contains the different values used to filter the gravitational wave before running the science pipeline.

[CLIENT]
id = "template" #string
secret = "template" #string

[KAFKA_CONFIG]
group_id = "template" #string
auto_offset_reset = "earliest" #string
enable_auto_commit = false #boolean

[GCN_TOPICS]
"igwn.gwalert" = "GW_alert"
"gcn.classic.voevent.ICECUBE_ASTROTRACK_GOLD" = "Icecube_alert"
"gcn.classic.voevent.ICECUBE_CASCADE" = "Icecube_alert"
"gcn.classic.voevent.ICECUBE_ASTROTRACK_BRONZE" = "Icecube_alert"

[PATH]
notice_path="template" #string

[GW_CONFIG]
BBH_thresh = 0.0  # probability to be a BBH, float
Dist_cut = 000    # mega parsec, float
sky_error = 000   # square degree, float
Significance = 0  # accept significant gw_alert or not, int
Has_ns = 0.0 #float
space_delay = 00 #float
space_score = 00 #float
ground_delay = 0 #float

How to access the database running on Portainer

1. Update the docker-compose.fsc.yml to create a new psql service with the following configuration:

  psql:
    <<: *shared-config
    image: alpine/psql
    command: ["tail", "-f", "/dev/null"]
    extra_hosts:
      postgres-server: "${IP_PGSQL_SERVER}"
    entrypoint: ["/bin/sh", "-c", 'exec "$$@"', "--"]

2. Select the branch where the docker-compose.fsc.yml is located and restart stack using the portainer interface.

3. Access the psql container using the Exec button in the container's page.

4. Tun the bin/sh command to access the container's shell and run :

export SQLALCHEMY_DATABASE_URI=$(cat $SQLALCHEMY_DATABASE_URI_FILE)
psql $SQLALCHEMY_DATABASE_URI