Skip to content

configuration_guide.md

Latest commit

 

History

History
578 lines (428 loc) · 20.2 KB

configuration_guide.md

File metadata and controls

578 lines (428 loc) · 20.2 KB
Raw

Configuration Guide

Route66 uses INI-based configuration files to manage all application settings. This guide covers every configuration section, key, and pattern you need to deploy and maintain a Route66 instance.

Table of Contents

How Configuration Works

File Locations and Precedence

Route66 reads configuration from two INI files, merged in order:

Priority Path Purpose
1 (lower) configs/config.ini Repository defaults, shipped with the codebase
2 (higher) ~/.config.ini User/deployment overrides in the home directory

Keys in ~/.config.ini override the same keys in configs/config.ini. If neither file exists, the application raises a FileNotFoundError at startup.

All keys are case-sensitive (the loader preserves case via optionxform = str).

Setting Up Your Config File

# 1. Copy the example
cp config.ini.example ~/.config.ini

# 2. Edit with your actual values
nano ~/.config.ini

# 3. Restrict permissions (the file contains secrets)
chmod 600 ~/.config.ini

The example file (config.ini.example) contains placeholder values and comments explaining every key. Use it as your starting template.

Deployment Configuration

Environment Sections (S3 Storage)

Route66 supports multiple deployment environments (dev, test, prod, etc.). Each is an INI section that defines where files, task outputs, and logs are stored in S3.

[dev]
S3_BUCKET = my-org-dev-data
S3_PREFIX = route66_dev

[prod]
S3_BUCKET = my-org-prod-data
S3_PREFIX = route66

The active environment is selected by DEPLOYMENT in [environment_variables]:

[environment_variables]
DEPLOYMENT = dev

At runtime the app reads config.get(DEPLOYMENT, "S3_BUCKET") and config.get(DEPLOYMENT, "S3_PREFIX") to resolve the correct bucket. S3 paths are constructed as:

s3://<S3_BUCKET>/<S3_PREFIX>/projects/<project_id>/tasks/<task_id>/output/

You can define as many environment sections as you need. Just add a new [section] and set DEPLOYMENT to match.

Database Configuration

Database connection is built from individual components in [environment_variables]:

[environment_variables]
DB_USER = route66_user
DB_PASS = secure_password
DB_HOST = localhost
DB_PORT = 5432
DB_NAME = db

These are assembled into a PostgreSQL connection URL at runtime:

postgresql://<DB_USER>:<DB_PASS>@<DB_HOST>:<DB_PORT>/<DB_NAME>

When running via Docker Compose, DB_HOST is automatically overridden to db (the Docker service name) by setup.sh.

Database Migrations (Alembic)

Route66 uses Alembic for schema migrations. The configuration is in alembic.ini and migration scripts are in alembic/versions/.

# Run pending migrations (inside the container or locally)
alembic upgrade head

# Create a new migration
alembic revision --autogenerate -m "description"

Alembic reads the database URL from the same config system via configs/config_loader.py in alembic/env.py.

Web Server Options

Variable Section Default Description
WEB_CONCURRENCY [environment_variables] 4 Number of Uvicorn worker processes
ROUTE66_ENDPOINT [environment_variables] — Public URL of the Route66 instance (used for callbacks from jobs)
UPLOAD_SERVICE_ENDPOINT [environment_variables] — URL of the external file upload service
NATIVE_APP_TEST_ENDPOINT [environment_variables] — URL for native app test execution

Providers Configuration

Provider Registry

Providers are execution environments where jobs run. They are configured entirely in config.ini, with no code changes needed to add new environments.

The [providers] section lists all active providers as a comma-separated string:

[providers]
providers = AWS_BATCH, MY_HPC_CLUSTER, EXTERNAL_SERVICE

Each listed name must have a corresponding INI section with at least a class key:

[AWS_BATCH]
class = aws
JOB_QUEUE = arn:aws:batch:us-east-1:123456789:job-queue/my-queue
JOB_ROLE_ARN = arn:aws:iam::123456789:role/my-role

At startup, route66/main.py registers the provider classes and loads the config:

provider_registry.register_class("aws", AWSProvider)
provider_registry.register_class("custom", CustomProvider)
provider_registry.load_from_config(config)

Provider names (the section names) are used in task.execution_environment and app.allowed_executors throughout the application.

AWS Provider

Class name: aws

The AWS provider handles S3 storage operations and AWS Batch compute. Config keys:

Key Required Description
class Yes Must be aws
JOB_QUEUE Yes ARN of the AWS Batch job queue
JOB_ROLE_ARN Yes ARN of the IAM role for job containers
AWS_ACCESS_KEY_ID No Per-provider AWS key override
AWS_SECRET_ACCESS_KEY No Per-provider AWS secret override

If AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY are not set in the provider section, the provider relies on the standard boto3 credential chain (process environment variables, IAM role, etc.). In a Docker deployment, setup.sh exports the global AWS keys from [environment_variables] as OS environment variables, so boto3 picks them up automatically.

Example with multiple AWS environments:

[providers]
providers = DEV_BATCH, PROD_BATCH

[DEV_BATCH]
class = aws
JOB_QUEUE = arn:aws:batch:us-east-1:111111111:job-queue/dev-queue
JOB_ROLE_ARN = arn:aws:iam::111111111:role/dev-role

[PROD_BATCH]
class = aws
JOB_QUEUE = arn:aws:batch:us-east-1:222222222:job-queue/prod-queue
JOB_ROLE_ARN = arn:aws:iam::222222222:role/prod-role

Custom Provider

Class name: custom

The custom provider is a stub for externally-managed, complex execution environments not supported by Route66. It accepts job submissions without directly issuing any computational resources. This allows you to implement your own external job monitoring services with undefined behaviors while using the Route66 API and/or the app engine's primitives to connect to any system of your choice.

[ARVADOS]
class = custom

Use this when an external system handles job orchestration and only needs to appear as a selectable environment in the Route66 UI.

Adding a New Provider

  1. Add a new INI section with the desired name and class key:

    [MY_NEW_ENV]
class = aws
JOB_QUEUE = arn:aws:batch:...
JOB_ROLE_ARN = arn:aws:iam::...

  2. Add the name to the providers list:

    [providers]
providers = AWS_BATCH, MY_NEW_ENV

  3. Restart the application. No code changes are needed.

For a truly new provider type (not aws or custom), implement a subclass of BaseProvider in route66/providers/, register it in route66/main.py, and use your new class name.

Authentication Configuration

Choosing an Auth Method

Authentication is controlled by the [authentication] section:

[authentication]
method = ldap
Value Provider Description
ldap LDAPProvider Authenticates against an LDAP/Active Directory server
local LocalProvider Username/password stored locally with Argon2 hashing

LDAP Authentication

[authentication]
method = ldap
LDAP_SERVER = ldap://ldap.example.com
LDAP_BIND_DN = svc-account@example.com
LDAP_BIND_PASSWORD = service_account_password
LDAP_BASE_DN = OU=people,DC=example,DC=com
Key Required Description
LDAP_SERVER Yes LDAP server URL (e.g., ldap:// or ldaps://)
LDAP_BIND_DN Yes Service account DN for binding
LDAP_BIND_PASSWORD Yes Password for the bind DN
LDAP_BASE_DN Yes Base DN to search for users

Users are authenticated by binding to LDAP with the provided credentials. On success, a JWT token is issued.

Local Authentication

[authentication]
method = local
min_password_length = 12
max_password_length = 128
max_failed_attempts = 5
lockout_minutes = 15
# common_passwords_path = /path/to/common_passwords.txt
Key Required Default Description
min_password_length No 12 Minimum password length
max_password_length No 128 Maximum password length
max_failed_attempts No 5 Failed logins before account lockout
lockout_minutes No 15 How long accounts stay locked (minutes)
common_passwords_path No built-in list Path to a file of common/banned passwords (one per line)

Passwords are hashed with Argon2 (time_cost=3, memory_cost=64 MB, parallelism=4). The password policy enforces:

  • Minimum and maximum length
  • Password must not match the username
  • Password must not be in the common passwords list

Local auth supports password changes and account lockout after too many failed attempts.

Token and Session Settings

[authentication]
SECRET_KEY = your-secret-key-for-jwt
TOKEN_EXPIRE_DAYS = 30.44
Key Section Default Description
SECRET_KEY [authentication] — Secret used for signing JWT tokens (HS256). Required.
TOKEN_EXPIRE_DAYS [authentication] 30.44 How long user session tokens remain valid

The SECRET_KEY is also used for CSRF protection. Generate a strong random value:

python -c "import secrets; print(secrets.token_urlsafe(32))"

Job Variables

How Job Variables Work

When Route66 submits a job to AWS Batch, it constructs the container environment from three sources:

  1. Automatic variables injected by the JobService (callback URL, Route66 auth token, git branch, etc.)
  2. [job_environment_variables] section, passed directly to the container
  3. [environment_variables] keys for AWS and GitHub credentials

Built-in Job Variables

These are set automatically by JobService.env_vars() and do not need to be configured:

Variable Source Description
ROUTE66_TOKEN Generated JWT token for the job to call back to Route66 (72-hour expiry)
ROUTE66_ENDPOINT [environment_variables] URL the job uses for HTTP callbacks
TASK_USER Task metadata Username of the person who submitted the task
TASK_USER_EMAIL Task metadata Email of the submitter
R66_GIT_BRANCH [environment_variables] Git branch or commit to checkout in the job container
GITHUB_TOKEN [environment_variables] For cloning public GitHub repos
GITHUB_ENTERPRISE_TOKEN [environment_variables] For cloning from GitHub Enterprise

User-Defined Job Variables

The [job_environment_variables] section is entirely user-controlled. Add any key-value pairs your jobs need:

[job_environment_variables]
# Nextflow Tower
TOWER_ACCESS_TOKEN = your-tower-token
TOWER_API_ENDPOINT = https://tower.example.com/api
TOWER_WORKSPACE_ID = org/workspace

# External database
ODS_USER = data_loader
ODS_PASSWORD = secure_password
ODS_HOST = database.example.com
ODS_PORT = 5432
ODS_SCHEMA = production

# AI/ML API keys
AZURE_OPENAI_API_KEY = your-openai-key
ANTHROPIC_API_KEY = your-anthropic-key

# Add any custom variables your apps need
MY_CUSTOM_VAR = my_value

All keys in this section are forwarded to every AWS Batch job container.

Environment Variables Injected at Runtime

The native_app_runner.sh script runs inside the job container and expects these environment variables (set by the combination above):

Used By Variables
Git clone GITHUB_TOKEN or GITHUB_ENTERPRISE_TOKEN, ROUTE66_GITHUB_URL
S3 sync AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
Callback ROUTE66_TOKEN, ROUTE66_ENDPOINT
App-specific Anything from [job_environment_variables]

GitHub Integration

Route66 supports both public GitHub and GitHub Enterprise for app repository access.

[environment_variables]
# Public GitHub (github.com)
GITHUB_TOKEN =
GITHUB_API_URL = https://api.github.com

# GitHub Enterprise (optional)
GITHUB_ENTERPRISE_URL = https://github.mycompany.com
GITHUB_ENTERPRISE_TOKEN = ghp_xxxxxxxxxxxx
GITHUB_ENTERPRISE_API_URL = https://github.mycompany.com/api/v3

# Route66 repository (cloned in job containers)
ROUTE66_GITHUB_URL = https://github.com/your-org/be.app.route66
Key Required Description
GITHUB_TOKEN No Personal access token for github.com. Leave empty for public-only access.
GITHUB_API_URL No Defaults to https://api.github.com
GITHUB_ENTERPRISE_URL No Base URL of your GitHub Enterprise instance
GITHUB_ENTERPRISE_TOKEN No PAT for GitHub Enterprise
GITHUB_ENTERPRISE_API_URL No API URL for GitHub Enterprise
ROUTE66_GITHUB_URL Yes URL used to git clone Route66 in job containers

Tokens are optional. If omitted, only public repositories are accessible. If a feature requires a token that is not configured, a clear error is raised at the point of use.

LLM / AI Assistant

Route66 includes an AI assistant feature powered by Anthropic's API.

[environment_variables]
ANTHROPIC_MODEL = claude-sonnet-4-5-20250929
ANTHROPIC_MAX_TOKENS = 2000
ANTHROPIC_BASE_URL = https://api.anthropic.com
ANTHROPIC_API_KEY = your-api-key
Key Required Description
ANTHROPIC_MODEL No Model identifier to use
ANTHROPIC_MAX_TOKENS No Maximum tokens per response
ANTHROPIC_BASE_URL No API endpoint (use a proxy URL if needed)
ANTHROPIC_API_KEY Yes (for AI features) API key for authentication

If the Anthropic keys are not configured, AI assistant features are unavailable but the rest of the application works normally.

UI Refresh Settings

The task detail page polls for updated logs and reports at configurable intervals:

[environment_variables]
LOG_REFRESH_DELAY = 120
REPORT_REFRESH_DELAY = 180
Key Default Description
LOG_REFRESH_DELAY Sensible default Seconds between log content polls in the browser
REPORT_REFRESH_DELAY Sensible default Seconds between HTML report iframe reloads

Both are optional. If omitted, the application uses built-in defaults.

Job Monitor Daemon

The monitor daemon (r66_monitor_daemon) periodically checks the status of running jobs and updates the database. It runs as a separate Docker container.

[environment_variables]
JOB_DAEMON_INTERVAL_MIN = 10
Key Required Default Description
JOB_DAEMON_INTERVAL_MIN No Not set (disabled) Interval in minutes between monitoring cycles

When this variable is set, setup.sh automatically starts the monitor service via the --profile monitor flag. When unset, no monitor runs and job status must be updated via callbacks alone.

Full Configuration Reference

Below is a condensed reference of every configuration section and key. See config.ini.example for a fully commented template.

[environment_variables]

Key Required Description
DB_USER Yes PostgreSQL username
DB_PASS Yes PostgreSQL password
DB_HOST Yes Database host (db inside Docker)
DB_PORT Yes Database port (default: 5432)
DB_NAME Yes Database name (default: db)
DEPLOYMENT Yes Active environment section name
ROUTE66_ENDPOINT Yes Public application URL for job callbacks
AWS_ACCESS_KEY_ID Yes AWS access key
AWS_SECRET_ACCESS_KEY Yes AWS secret key
ANTHROPIC_MODEL No LLM model identifier
ANTHROPIC_MAX_TOKENS No LLM max tokens
ANTHROPIC_BASE_URL No LLM API endpoint
ANTHROPIC_API_KEY No LLM API key
GITHUB_TOKEN No GitHub.com PAT
GITHUB_API_URL No GitHub API URL
GITHUB_ENTERPRISE_URL No GHE base URL
GITHUB_ENTERPRISE_TOKEN No GHE PAT
GITHUB_ENTERPRISE_API_URL No GHE API URL
ROUTE66_GITHUB_URL Yes Route66 repo URL for job cloning
UPLOAD_SERVICE_ENDPOINT No External upload service URL
NATIVE_APP_TEST_ENDPOINT No Native app test endpoint
NFTOWER_LAUNCHER_IMAGE No Docker image for NF Tower launcher
WEB_CONCURRENCY No Uvicorn worker count (default: 4)
R66_GIT_BRANCH No Git branch to checkout in jobs
LOG_REFRESH_DELAY No Log poll interval (seconds)
REPORT_REFRESH_DELAY No Report reload interval (seconds)
JOB_DAEMON_INTERVAL_MIN No Monitor daemon interval (minutes)

[authentication]

Key Required Description
method Yes ldap or local
SECRET_KEY Yes JWT/CSRF signing secret
TOKEN_EXPIRE_DAYS No Session token lifetime (default: 30.44)
LDAP_SERVER LDAP only LDAP server URL
LDAP_BIND_DN LDAP only Service account DN
LDAP_BIND_PASSWORD LDAP only Service account password
LDAP_BASE_DN LDAP only User search base DN
min_password_length Local only Min length (default: 12)
max_password_length Local only Max length (default: 128)
max_failed_attempts Local only Lockout threshold (default: 5)
lockout_minutes Local only Lockout duration (default: 15)
common_passwords_path Local only Path to banned passwords file

[job_environment_variables]

User-defined. All keys are forwarded to job containers. Common examples:

Key Description
TOWER_ACCESS_TOKEN Nextflow Tower API token
TOWER_API_ENDPOINT Nextflow Tower API URL
TOWER_WORKSPACE_ID Nextflow Tower workspace
ODS_USER External database username
ODS_PASSWORD External database password
ODS_HOST External database host
ODS_PORT External database port
ODS_SCHEMA External database schema
AZURE_OPENAI_API_KEY Azure OpenAI API key

[<DEPLOYMENT>] (e.g., [dev], [prod])

Key Required Description
S3_BUCKET Yes S3 bucket for this environment
S3_PREFIX Yes Path prefix within the bucket

[providers]

Key Required Description
providers Yes Comma-separated list of active provider section names

[<PROVIDER_NAME>] (e.g., [AWS_BATCH])

Key Required Description
class Yes Provider class (aws or custom)
JOB_QUEUE AWS only AWS Batch job queue ARN
JOB_ROLE_ARN AWS only IAM role ARN for job containers
AWS_ACCESS_KEY_ID No Per-provider AWS key override
AWS_SECRET_ACCESS_KEY No Per-provider AWS secret override