Adding and maintaining web (and other) services in braingeneers infrastructure

Basic usage:

Prerequisite:

1. Log into our server: braingeneers.gi.ucsc.edu, see the permissions page for access, then clone the repo into your home directory on braingeneers server as shown below.
2. Copy the NRP Kubernetes config file for service accounts to your home directory under ~/.kube/config. Download config file here. This file allows the services to access our namespace secrets, which is where we store all our credentials files.

# clone the repo
> git clone git@github.com:braingeneers/mission_control.git
> cd mission_control

Using Codex for service deployment work

When you want Codex to help add, update, deploy, or troubleshoot a lab service in this repo, ask it to use the mission-control-services-management skill. The skill is intended for developers working with mission_control services on braingeneers.gi.ucsc.edu.

Use this skill for work involving:

• private web services behind the default browser authentication
• intentionally public web services with host-specific service-proxy overrides
• headless or direct-port services that should not be routed through the web proxy
• MCP services that need bearer-token forwarding and backend token validation
• outbound Slack and email notifications through the shared notification service
• Kubernetes secret wiring through secret-fetcher and /secrets
• published container image workflows, including service Makefile build, push, local-test, and shell targets
• targeted service pull, recreate, log, and status checks for conservative deploys

The skill first selects the required evidence or action surface: protected HTTPS API, Data Explorer content API, rendered browser UI, operator-run server command, or implementation change. It then inspects only the matching Compose, proxy, service, wiki, and focused reference sources under skills/mission-control-services-management/. Developers should have GI server access, Braingeneers GitHub access, NRP namespace access for secret-related work, and registry credentials when publishing custom images.

The skill source is skills/mission-control-services-management/SKILL.md.

Before committing changes to Compose or service-proxy, run the complete local configuration gate:

make test

This validates the Compose model, nginx authentication and identity-header inheritance, the notification-service routing and outbound-mail contracts, and the production/development bucket and immutable-image contracts.

Manage individual services

You can also start and stop a single services, this is the normal case so you don't interfere with other running services, it's perfectly safe to do this while other services are running:

The name my_service is defined in the docker-compose.yaml file under services: for example mqtt, notification-service, etc. are services in the docker-compose.yaml file

# Restart a service (cleaner than using docker compose restart)
docker compose up -d --force-recreate --pull always my-service

# Stop (remove) a specific service
docker compose rm -sf my-service

# Pull the latest image for a specific service
docker compose pull my-service

# Start a specific service in the background (removed -d, detached, to see logs in the foreground, useful during testing)
docker compose up -d my-service

# See a list of all braingeneers service names (these are defined in the docker-compose.yaml file but can be quickly listed with a double tab)
docker compose up -d <tab><tab>

# See logs for a specific service
docker compose logs my-service

# See process status for all services
docker compose ps

Scheduled data lifecycle backup

The AWS/Glacier backup pipeline has been onboarded as the braingeneers-data-lifecycle-backup Nextflow workflow in the workflows repository. Workflows launches each pipeline stage as a Kubernetes Job, keeps at most one backup run active, and publishes durable run artifacts under s3://braingeneers/services/data-lifecycle/runs/.

The intended production schedule is 9:00 PM America/Los_Angeles on Monday, Tuesday, and Friday with overlap handling set to skip. Manage this schedule from the Workflows Schedules page.

Each successful backup advances the small s3://braingeneers/services/data-lifecycle/latest-backup-state.json manifest only after its immutable run bundle is complete. Data Explorer rebuilds a local SQLite status projection from that evidence, and the separate monthly Data Retention Policy Report consumes the same pointer. Neither consumer performs a live Glacier inventory query.

Data Explorer keeps dataset management subordinate to normal browsing. One collapsed panel groups backup coverage, the active online cleanup date, and live DANDI publication state. Plain-language controls can keep an atomic dataset or individual file online longer with a zero-byte DATA_LIFECYCLE_RETENTION marker and can create or reverse a folder NOBACKUP. The marker's Ceph LastModified is authoritative and user data is never self-copied. Cleanup removes only the active S3 copy after its Glacier backup is confirmed; the Glacier archive remains restorable, and Data Explorer does not perform cleanup itself. The local volume holds only a disposable, restart-persistent index; immutable request/result audits remain in S3.

Mission Control owns the registry image source and lifecycle policy in data-lifecycle/. The Nextflow source and catalog definitions remain in the sibling workflows repository. Build, test, and publish the task image from this repository:

make data-lifecycle-build
make data-lifecycle-test
make data-lifecycle-push

The retired Data Lifecycle review website is not a Compose service. Data Explorer is the sole user interface for retention renewal and NOBACKUP.

Data Explorer DANDI publication

Data Explorer provides push-button publication of an exact NWB selection to a new DANDI Sandbox Dandiset. The browser can separately select raw ephys and derived spike/telemetry NWBs, choose open or embargoed access at creation, sync the mutable draft, publish an OPEN Valid draft with release notes, and later start an irreversible unembargo. It displays the Dandiset, validation/access states, workflow outcome, every known DOI, and the source NWBs represented by each immutable version.

The service owns a data_explorer schema in shared sql-db. Its entrypoint runs Alembic before FastAPI and keeps SQLAlchemy table auto-create disabled. Unique immutable request and result objects live beneath s3://braingeneers/services/data-explorer/dandi/sandbox/. Data Explorer sends schema-v2 launch messages to the optional Workflows MQTT ingress and reconciles request IDs through the internal Workflows API. It has no Compose dependency on MQTT or Workflows, so either integration can recover independently.

The credential-bearing dandi-publication task runs on NRP from the sibling workflows catalog. Data Explorer itself never receives a DANDI API key. The workflow expects the existing s3-credentials Secret keys access_key and secret_key, plus the operator-owned dandi-sandbox-api-key Secret key api_key. Verify those exact names and keys before enabling a real test; Secret creation or changes remain operator-owned.

Create the Sandbox token Secret from an operator-protected file, then verify only its key name (the command does not print the token):

kubectl -n braingeneers create secret generic dandi-sandbox-api-key \
  --from-file=api_key=/path/to/dandi-sandbox-api-key
kubectl -n braingeneers get secret dandi-sandbox-api-key \
  -o go-template='{{range $key, $_ := .data}}{{$key}}{{"\n"}}{{end}}'

After publishing aligned Data Explorer, Workflows, and worker images, an operator can refresh only the affected server services:

docker compose pull sql-db workflows-backend workflows data-explorer
docker compose up -d sql-db
docker compose up -d --force-recreate workflows-backend workflows data-explorer
docker compose ps sql-db workflows-backend workflows data-explorer
docker compose logs --tail=200 workflows-backend data-explorer

Replicated volume backup

replicated-volume-backup is a headless Mission Control infrastructure service. Every day at 2:00 AM America/Los_Angeles it copies new and changed files from the shared read-only replicated volume to s3://braingeneersdev/services/replicated/. It never deletes destination objects and excludes dot-prefixed and *.tmp incomplete files.

The registry-published image and its scripts live in replicated-volume-backup/. It reads the existing prp-s3-credentials/credentials file from secret-fetcher; no additional Kubernetes Secret is required. Lock and status files live under local:/local/replicated-volume-backup, and the service is unhealthy when no successful sync has been recorded in the last 36 hours.

Build, test, and publish the image from this repository:

make replicated-volume-backup-test
make replicated-volume-backup-push

Run a manual sync when validating or recovering this service:

docker compose pull replicated-volume-backup
docker compose run --rm replicated-volume-backup sync
docker compose up -d --force-recreate replicated-volume-backup
docker compose ps replicated-volume-backup
docker compose logs --tail=200 replicated-volume-backup

Routine updates target only this service:

docker compose pull replicated-volume-backup
docker compose up -d --force-recreate replicated-volume-backup
docker compose ps replicated-volume-backup
docker compose logs --tail=200 replicated-volume-backup

Shared SQL database service

sql-db is the shared internal SQL database service for Mission Control applications. It is not published outside the Docker network and uses public default credentials as an internal compatibility guard, not as a security boundary:

POSTGRES_DB=services
POSTGRES_USER=services
POSTGRES_PASSWORD=services

Clients join braingeneers-net, wait for the sql-db health check, and own a normalized application-named schema in the shared services database. They must configure their connection and migration tooling to use that schema instead of creating tables in public.

The sql-db image makes an unconfigured default connection fail closed with no current schema, while schema-aware client connections override that default. This guardrail is installed automatically for fresh clusters. It must not be applied to an existing database until every client selects its owned schema and public contains no application relations; see the sql-db guide for the backup, verification, rollout, and rollback procedure.

sql-db stores active database files under local:/local/sql-db. Its image runs a daily backup at 08:00 UTC and writes one custom-format dump per day to replicated:/replicated/sql-db/postgres using 30 rolling filename slots. Older slots are overwritten by filename rather than deleted.

See sql-db/README.md for schema provisioning, connection URLs, Compose wiring, migrations, backup/restore, and troubleshooting.

Deploy or refresh only the shared database service on braingeneers.gi.ucsc.edu:

docker compose pull sql-db
docker compose up -d --force-recreate sql-db
docker compose logs -f sql-db

Notification service

notification-service is the shared outbound Slack and email boundary for new integrations. Compose peers call http://notification-service:8000 directly. External callers use https://notifications.braingeneers.gi.ucsc.edu through the normal authenticated proxy, which accepts existing service-account JWTs and signed-in browser sessions. The application does not implement another bearer token scheme and does not use PostgreSQL or MQTT.

Slack delivery is synchronous through POST /v1/slack, supports channel IDs or direct delivery to stable user IDs, and exposes GET /v1/slack/destinations for friendly user and joined-channel pickers. Email is accepted with POST /v1/email into the persisted outbound Postfix queue. The email endpoint supports plain text, an optional HTML alternative, and bounded uploads; all mail uses notifications@braingeneers.gi.ucsc.edu. See the service README, user documentation, and skills/mission-control-services-management/references/notifications.md for the complete contracts and examples.

Two operator-owned Kubernetes Secrets provide the notification credentials:

• slack-token-braingeneersbot-gi, key slack-token-braingeneersbot-gi: the braingeneersbot token installed in the ucsc-gi Slack workspace. Mission Control configures the API to read /secrets/slack-token-braingeneersbot-gi/slack-token-braingeneersbot-gi.
• notification-service, key dkim-private-key: the private key for selector notifications and domain braingeneers.gi.ucsc.edu.

The Slack endpoint alone returns 503 while its token is absent. The mail relay waits without sending unsigned mail while its DKIM key is absent; the API and Slack channel remain independently healthy. Secret creation and replacement are operator-owned, and secret values must not enter this checkout or logs.

The deployed outbound identity is:

• A and PTR: braingeneers.gi.ucsc.edu / 128.114.198.51
• SPF: v=spf1 ip4:128.114.198.51 -all
• DKIM selector: notifications at notifications._domainkey.braingeneers.gi.ucsc.edu
• DMARC: v=DMARC1; p=none at _dmarc.braingeneers.gi.ucsc.edu

The server must retain outbound TCP 25 access. No inbound SMTP port, inbox, or MX record is required; delayed bounce processing is out of scope. The DKIM public record must match the operator-owned private key but is intentionally not copied into this repository so key rotation does not leave stale docs.

Deploy or refresh only the notification components:

docker compose pull notification-mail-relay notification-service
docker compose up -d --force-recreate notification-mail-relay notification-service
docker compose ps notification-mail-relay notification-service
docker compose logs --tail=200 notification-mail-relay notification-service
docker compose exec notification-mail-relay postqueue -p

Recreate service-proxy only when its configuration changes. After initial deployment or credential/DNS rotation, use #braingeneers-test and a controlled email recipient for acceptance. Slack must return 200 delivered; email must return 202 queued, arrive, and show spf=pass, dkim=pass, and dmarc=pass in Authentication-Results.

The existing MQTT↔Slack slack-bridge remains a separate integration for its current publishers and inbound consumers. It has no scheduled retirement and is not the default interface for new direct notification callers.

Workflows web service

The workflows service serves https://workflows.braingeneers.gi.ucsc.edu as a private web app behind the shared browser authentication and SSL proxy. The frontend and backend images are built and pushed from the workflows repository. The immutable deployed image tags are recorded in docker-compose.yaml rather than duplicated here.

The protected production route has been validated to provide the authenticated user's usable email in X-Email. In the same deployment, X-User contains an opaque CILogon subject and the other configured identity headers are empty. Workflows therefore uses only X-Email for browser action attribution. This is authoritative only because the private proxy overwrites the header; prod-local and other no-proxy requests may omit it, in which case Workflows records the friendly initiator User. Do not add identity-header debug logging: these values are user information and the production behavior is now documented.

The backend uses the shared secret-fetcher volume. It expects:

• /secrets/prp-s3-credentials/credentials for S3 access.
• /secrets/kube-config/config for Kubernetes launch and run monitoring.
• /secrets/github-readonly/braingeneers-runtime-readonly.private-key.pem for read-only workflow-definition refreshes from the private Braingeneers GitHub organization. The public App ID is configured directly in Compose as GITHUB_APP_ID; the backend exchanges the App ID and secret-mounted PEM for a short-lived installation token and does not store a long-lived Git access token.

The backend uses the shared internal sql-db database service and owns the workflows schema. Its production image selects that schema and verifies it before Alembic migrations or application startup. Ensure sql-db is already running; refresh it separately only when the shared database service changes.

The backend mounts the shared local volume at /local and stores collected run diagnostics under /local/workflows/runs. These launch-file copies, Nextflow traces and reports, task logs, resource telemetry, and collection markers survive backend container recreation but remain disposable. They do not belong in the backed-up replicated volume. Kubernetes continues to use the PVC-backed /workspace/runs path, which can lazily repopulate an older run's local collected cache while the PVC data remains available.

The backend is also the sole managed MQTT workflow launcher. It subscribes to the internal mqtt service on workflows/launch with QoS 1 and applies the same catalog validation, durable request idempotency, provenance, and Kubernetes launch path used by the web API. The retired standalone nextflow-launcher service and arbitrary-Git-URL protocol are not deployed. The retired mqtt-job-listener, job-scanner, and maxwell-dashboard definitions remain commented in Compose for reference and are not deployed. MQTT is an optional ingress path, so Workflows has no Compose startup dependency on the broker; broker availability must not block the web app or API.

Report workflows publish channel-neutral artifacts and do not select Slack channels, email recipients, or delivery behavior. Workflows schedules own any number of email and Slack recipients plus per-artifact link or inline delivery. Those settings are snapshotted when each scheduled run launches and delivered through the shared notification-service after terminal artifact collection. Workflows intentionally has no Compose startup dependency on it.

The backend also owns the durable schedule runner. Operators can create, preview, pause, resume, update, and delete schedules at /schedules; weekly, monthly, daily, and five-field cron schedules use explicit IANA time zones. Each schedule also has shared multiline notes displayed beneath its owner; existing schedules receive a blank note when the database migration runs. Missed occurrences are coalesced after downtime, and the configured overlap policy prevents a scheduled workflow from exceeding its catalog-defined active run limit. The data-lifecycle workflow is catalog-limited to one active run.

Deploy or refresh the workflows service group on braingeneers.gi.ucsc.edu:

docker compose pull workflows workflows-backend
docker compose up -d --force-recreate workflows-backend workflows
docker compose ps workflows-backend workflows
docker compose logs --tail=200 workflows-backend workflows

After deployment, check /api/admin/system-status for scheduler health and confirm the existing Data Lifecycle schedules retain their intended time zone, overlap policy, and enabled state.

The intended Data Retention Policy Report schedule runs on the 15th of each month at 09:00 in America/Los_Angeles, with overlap set to skip. Validate the interactive, printable, machine-readable, and Slack-ready report artifacts independently of future notification delivery.

The legacy Data Lifecycle review app is retired. If its stopped container still exists, remove it before pulling the Compose revision that deletes the service definition:

docker compose rm -f data-lifecycle
git pull --ff-only
docker compose pull workflows-backend workflows
docker compose up -d --force-recreate workflows-backend workflows
docker compose ps workflows-backend workflows
docker compose logs --tail=200 workflows-backend workflows

The published braingeneers/data-lifecycle image remains required by the two Nextflow workflows even though no long-running Compose service has that name.

The uploader publishes selected Ephys workflow requests to the same internal MQTT broker. Refresh it alongside Workflows when the launch contract or uploader image changes:

docker compose pull uploader workflows workflows-backend
docker compose up -d --force-recreate workflows-backend workflows uploader
docker compose logs -f workflows-backend workflows uploader
make verify-uploader-deployment SERVICE=uploader

For side-by-side uploader acceptance testing, uploader remains the production service at https://uploader.braingeneers.gi.ucsc.edu and uploader-dev runs the candidate image at https://uploader-dev.braingeneers.gi.ucsc.edu. The candidate service uses PROD=true, so dataset discovery, metadata operations, and uploads all use the production braingeneers bucket. Treat it as production-data access despite the -dev hostname. It stores its metadata templates separately under /replicated/uploader-dev/metadata-templates. Its optional AI metadata prefill reads the NRP LLM key from the shared secret-fetcher volume at /secrets/nrp-llm-api-key; if that secret is unavailable, the candidate keeps ordinary upload and metadata editing enabled while disabling only AI prefill.

Deploy or refresh only the acceptance-test service:

docker compose pull uploader-dev
docker compose up -d --force-recreate uploader-dev
docker compose ps uploader uploader-dev
docker compose logs --tail=100 uploader-dev
make verify-uploader-deployment SERVICE=uploader-dev

The deployment verifier compares the Compose image reference, the pulled image ID, the running container image ID, APP_VERSION, and PROD. Run it after every uploader image update so an old container left behind by a pull or restart is reported immediately.

If shared Kubernetes secrets such as prp-s3-credentials or kube-config were changed, refresh secret-fetcher first:

docker compose up -d --force-recreate secret-fetcher
docker compose logs -f secret-fetcher

Shared local volumes

New services should use the shared Docker volumes local and replicated instead of adding service-specific top-level volumes. Each service owns a directory under the volume root, such as /local/sql-db or /replicated/sql-db.

• local is restart-persistent local state that may be lost without breaking the service permanently. Active file changes belong here.
• replicated is for backed-up static files. Services should stage changing files in local and publish completed artifacts into replicated.
• Dot-prefixed temporary publish files in replicated should be treated as incomplete and ignored by backup tooling.

Uploader versions that support metadata presets store their versioned template JSON in an environment-specific service directory. Production uses /replicated/uploader/metadata-templates; the acceptance-test service uses /replicated/uploader-dev/metadata-templates. They publish completed records with atomic renames from dot-prefixed temporary files, so the daily replicated-volume sync copies only complete template revisions.

Managing all services

This should only be done when the server is rebooted, under normal conditions you will be managing individual services as describe above.

# Pull latest version of all services
docker compose pull

# Start all services
docker compose up -d

# Stop all services
docker compose down

How to Add a New Service

Step 1: Clone the Repository

Clone the mission_control repo where our Docker Compose configuration resides (this command assumes GitHub SSH clone).

git clone git@github.com:braingeneers/mission_control.git
cd mission_control

Step 2: Edit the Docker Compose File

Open the docker-compose.yaml file located in the mission_control directory in a text editor.

Adding a New Service

Add a new service definition for your container under the services section, similar to the existing services. Here is an example service you can start with:

  # Describe your service in comments and let other people know who manages it
  your-service:                                                 # give your service a meaningful name, replace "your-service" with something meaningful like "supervisualizer"
    image: jwilder/whoami:latest                                # your docker image name as hosted on Docker Hub (or other image hosting service)
    expose:
      - "8000"                                                  # the port(s) that your service is listening on inside your container
    environment:
      VIRTUAL_HOST: "your-service.braingeneers.gi.ucsc.edu"     # choose an appropriate domain name for your service, for example: your-service.braingeneers.gi.ucsc.edu
      VIRTUAL_PORT: "8000"                                      # same as what you listed in expose
      LETSENCRYPT_HOST: "your-service.braingeneers.gi.ucsc.edu" # same as VIRTUAL_HOST
      LETSENCRYPT_EMAIL: "braingeneers-admins-group@ucsc.edu"   # don't change this
    networks:
      - braingeneers-net                                        # don't change this

Setting the Virtual Host and LetsEncrypt environment variables

The VIRTUAL_HOST & LETSENCRYPT_HOST environment variables in your service's definition determine the subdomain your service will be accessible from. For instance, if VIRTUAL_HOST is set to your-service. Your service will be accessible from https://your-service.braingeneers.gi.ucsc.edu. You can choose any valid hostname under the braingeneers.gi.ucsc.edu domain. The VIRTUAL_PORT defines what port(s) your service listens to. Your service can listen on any port, this is only used internally between the frontend nginx web server and your service.

Configuring Shared Secrets

IMPORTANT Never include credentials in your docker image, that image is public and those credentials will be found and stolen if you do. Credentials are stored in our braingeneers namespace in Kubernetes. See the administration page on the wiki for more details.

If your service requires access to shared secrets such as the S3 credentials, add a volume mount from the shared secrets volume. The secrets will be available in the following directory structure. All secrets in our namespace are downloaded by the secret-fetcher service. If you've added a new secret to the Kubernetes namespace you can simply restart the secret-fetcher service to make it available and watch the secret-fetcher logs to confirm your new secret was pulled.

/secrets/
  ├── prp-s3-credentials/
  │   └── credentials
  └── other-k8s-secret/
      └── other-files

You can add the following to your yaml to add this volume, you will see other services in the yaml that use this structure:

    volumes:
      - secrets:/secrets
    depends_on:
      secret-fetcher:
        condition: service_healthy

Using entrypoint-secrets-setup.sh

This is an advanced option.

The entrypoint-secrets-setup.sh script wraps the original entrypoint and allows copying files from the dynamic secrets volume to the correct location, as well as exporting environment variables from a specified file. Include this script in the service definition as shown in the example below if you need credentials files moved to the proper location, or environment variables set before launching your process.

Note the use of one or more --copy (or --env) commands which are processed by entrypoint-secrets-setup.sh to copy files to the correct location. You may use the --env option followed by the path to an environment file to export variables defined within that file into the environment.

Example of --env usage:

--env /secrets/k8s-secret-name/mysecrets.env

The above line will export all key-value pairs defined in /secrets/k8s-secret-name/mysecrets.env as environment variables, excluding lines starting with #. See the wiki permissions page on how to maintain kubernetes namespace secrets.

After processing the --copy and --env options, your original entrypoint command is executed with its arguments.

services:
  your-service:
    image: your-image:latest
    entrypoint: /secrets/entrypoint-secrets-setup.sh
    command:
      - "--copy"
      - "/secrets/prp-s3-credentials/credentials:/root/.aws/credentials"
      - "--copy"
      - "/secrets/other-k8s-secret/another-file:/path/to/another-file"
      - "--env"
      - "/secrets/k8s-secret-name/mysecrets.env"
      - "original-entrypoint-command"
      - "arg1"
      - "arg2"
    environment:
      VIRTUAL_HOST: "your-service.braingeneers.gi.ucsc.edu"
      VIRTUAL_PORT: "8000"
      LETSENCRYPT_HOST: "your-service.braingeneers.gi.ucsc.edu"
      LETSENCRYPT_EMAIL: "braingeneers-admins-group@ucsc.edu"

    # This section adds the secrets volume to your service, secrets are stored in-memory so they must depend on the
    # secret-fetcher service which populates that volume at startup.
    volumes:
      - secrets:/secrets
    depends_on:
      secret-fetcher:
        condition: service_healthy

Use the --env option in combination with the --copy option as needed to set up the environment for your containerized application.

If your service uses braingeneerspy and needs a JWT service-account token at runtime, do not copy /secrets/service-accounts/config.json into braingeneers/iot/service_account/config.json. Long-running unattended services should instead mount the refreshed secret maintained by service-account-jwt-token-refresh:

    command:
      - "--copy"
      - "/secrets/braingeneers-jwt-service-account-token/config.json:/usr/local/lib/python3.10/site-packages/braingeneers/iot/service_account/config.json"
      - "python3"
      - "your_app.py"

/secrets/service-accounts/config.json is not the regularly refreshed runtime token source. The service-account-jwt-token-refresh service updates the braingeneers-jwt-service-account-token Kubernetes secret daily, and that is the correct secret to mount for non-interactive services that rely on braingeneerspy.

Commit the Changes

After verifying your service works correctly, commit the changes to the docker-compose.yaml file back to the mission_control repository.

An Overview of Our Infrastructure

This section describes the automatic service discovery, automatic SSL certificate management, and automatic authentication provided.

We use a reverse proxy (service-proxy) which terminates SSL, maintains certificates automatically, checks user and service-account authentication, and provides a secure front end to all our web-services. The service-proxy auto discovers docker based web-services in our environment based on a few simple environment variables set by the service. Nothing needs to be updated or changed when a new service is brought online. SSL, DNS, authentication, and network security are all provided and managed automatically. By default, virtual hosts inherit browser authentication from service-proxy/default; a host-specific file under service-proxy/ can override that behavior for intentionally public services such as spikelab.braingeneers.gi.ucsc.edu. Because service-proxy bind-mounts each host override explicitly in docker-compose.yaml, adding a new host override also requires adding the matching volume mount there.

MCP Services

MCP services are a special case:

• they stay behind the shared service-proxy edge and TLS setup
• they do not use the normal browser-oriented oauth2-proxy runtime auth flow on /mcp
• the MCP backend validates bearer tokens itself as the OAuth protected resource
• the proxy layer must preserve the original Authorization header end to end

Use docs/mcp-onboarding.md when adding or updating an MCP service. It defines the standard compose wiring, proxy override, IAM mount, issuer environment variables, and end-to-end test steps.

MCP Broker Host

MCP now also has a dedicated self-hosted broker path under:

• https://oauth2.braingeneers.gi.ucsc.edu

This host is intentionally separate from:

• https://auth.braingeneers.gi.ucsc.edu

Use the new host for MCP-only OIDC broker work. Leave the existing auth.braingeneers.gi.ucsc.edu flow attached to oauth2-proxy and the current web Auth0 login stack.

Broker artifacts live in:

• oauth2-broker/

%%{init: {'flowchart': {'curve': 'basis'} } }%%
flowchart TD
    A["User (Browser) Request"] --> B{Service Proxy\nnginx}
    B -->|auth_request| C{OAuth2}
    C -->|Return 2XX or 40X\nbased on auth session\nor jwt token| B
    B -->|If OAuth2 = 40X| E[Auth0]
    E -->|University and\nother auth providers\nsupported| F[CILogon]
    F --> E
    E -->|Verify User Roles &\nCreate auth session| B
    B -->|If OAuth2 = 2XX| D["All Web Services [1]"]
    
    G[Service Account Request\n`Authorization: Bearer jwt_token`] --> B
    B -->|If OAuth2 = 2XX| H["Service Accounts App [2]"]
    H --> I[Generate JWT via Auth0]

    B <--->|Create and refresh\nSSL certificates| J[Let's Encrypt\nService]

    classDef default fill:#ffffff,stroke:#333,stroke-width:2px,color:#000000;
    classDef proxy fill:#4a90e2,stroke:#333,stroke-width:2px,color:#000000;
    classDef auth fill:#ffd700,stroke:#333,stroke-width:2px,color:#000000;
    classDef service fill:#87cefa,stroke:#333,stroke-width:2px,color:#000000;
    classDef helper fill:#90EE90,stroke:#333,stroke-width:2px,color:#000000;

    class B,C,J proxy;
    class E,F,I auth;
    class D,H service;

Loading

[1] Examples: https://whoami.briangeneers.gi.ucsc.edu, https://your-service.braingeneers.gi.ucsc.edu
[2] https://service-accounts.braingeneers.gi.ucsc.edu/generate_token

Nginx Reverse Proxy

The nginx-proxy is a Docker container running Nginx and docker-gen. docker-gen generates reverse proxy configurations for Nginx and reloads Nginx when containers are started and stopped. This setup allows us to route incoming requests to different Docker containers (our services), each possibly running a different application, all on the same host machine.

Shared Secrets Fetcher

The secret-fetcher service is a special Docker container that fetches shared secrets from a Kubernetes secret store. It does this on behalf of the other services running in the same Docker Compose setup. The secrets are retrieved when the services are started and stored in an in-memory volume accessible to all services. This ensures that each service has access to the same secrets without requiring them to retrieve the secrets individually. The only requirement is that the user running the Docker Compose stack has access to the Kubernetes namespace containing the secrets.

Let's Encrypt for SSL Certificates

The LetsEncrypt container automates the creation and renewal of SSL certificates used by the oauth2-proxy. It communicates with the Let's Encrypt service to generate valid certificates for the domains specified via environment variables. The generated SSL certificates are stored in a shared volume and used by the oauth2-proxy to secure the client communication via HTTPS. This streamlines the management of our SSL certificates and enhances the security of our user-facing services.

Authentication via Oauth2 Proxy / Auth0 / cilogon.org

All requests are first handled by service-proxy (nginx). Service-proxy performs an internal authentication request against oauth2-proxy, which will check for a valid session or JWT token. Oauth2 proxy returns a 2XX or 40X response to indicate if the user is authenticated or not. If the user is not authenticated, the request is redirected to Auth0, and Auth0 forwards to CILogon for authentication. Once the user is authenticated, the user's roles are verified by Auth0 and an auth session is created. The user is then redirected back to the service-proxy, which performs another internal authentication request against oauth2-proxy which will now succeed.

The internal authentication subrequest must use proxy_pass_request_body off and clear Content-Length. Nginx does not provide the original body to an auth_request subrequest, so retaining its length causes OAuth2 Proxy to wait for nonexistent bytes and prevents authenticated POST, PUT, or PATCH requests from reaching their application.

For virtual hosts that inherit service-proxy/default, service-proxy maps fields from the successful authentication response into these application-facing request headers:

• X-User
• X-Email
• X-Groups
• X-Name
• X-Given-Name
• X-Family-Name
• X-Preferred-Username
• X-Subject

This list describes the headers configured in service-proxy/default; it does not guarantee that every header has a value. The oauth2-proxy version, identity-provider claims, and authentication method determine which source fields are returned. A header whose source field is unavailable may be empty or omitted, so verify the deployed route before making an application depend on a particular field.

For the current Workflows production route, that verification found a usable X-Email, an opaque CILogon subject in X-User, and no values in the remaining configured identity headers. This observation is deployment-specific rather than a general guarantee for every service or identity provider.

On the normal authenticated path, service-proxy overwrites these header names with values from the authentication subrequest and removes the downstream Authorization header after using it for authentication. Public virtual hosts configured with auth_request off do not provide trusted proxy-derived identity, so their backends must not treat client-supplied identity headers as authenticated user information. MCP virtual hosts are a separate case: they explicitly strip these identity headers and preserve Authorization so the backend can validate the bearer token itself.

This remains the current browser-oriented web-service authentication path. MCP-specific broker work under oauth2.braingeneers.gi.ucsc.edu is separate and should not be treated as a change to the existing web flow.

Service Accounts

JWT tokens are generated by auth0 for service accounts. The service accounts app is used to generate these tokens initially. Go to https://service-accounts.braingeneers.gi.ucsc.edu/generate_token to generate a token. All HTTP requests must include the Authorization header with the Bearer token. braingeneerspy will handle this automatically except for the first time, which requires manual authentication:

python -m braingeneers.iot.authenticate

That braingeneerspy command will (auto) bootstrap two credentials:

• the existing broad Auth0-backed service-account token from https://service-accounts.braingeneers.gi.ucsc.edu/generate_token
• a narrower interactive user token from the self-hosted Keycloak broker at https://oauth2.braingeneers.gi.ucsc.edu/realms/braingeneers

The interactive bridge token is a normal Keycloak-issued OIDC token with a refresh token. The remote MCP service continues to trust signed bearer-token claims, not proxy-injected identity headers.

Once the first token bundle is obtained manually:

• the service-account token keeps the current long-lived refresh behavior
• the interactive user token refreshes itself through the standard Keycloak token endpoint
• if either token is revoked or its refresh window expires, the user must rerun python -m braingeneers.iot.authenticate

Bridge-issuer endpoints:

• issuer:
  • https://oauth2.braingeneers.gi.ucsc.edu/realms/braingeneers
• JWKS:
  • https://oauth2.braingeneers.gi.ucsc.edu/realms/braingeneers/protocol/openid-connect/certs
• device authorization:
  • https://oauth2.braingeneers.gi.ucsc.edu/realms/braingeneers/protocol/openid-connect/auth/device
• token endpoint:
  • https://oauth2.braingeneers.gi.ucsc.edu/realms/braingeneers/protocol/openid-connect/token

For services deployed under mission_control, the usual pattern is different from local interactive use: do not depend on manual braingeneerspy token bootstrap inside the container. Instead, mount /secrets/braingeneers-jwt-service-account-token/config.json to braingeneers/iot/service_account/config.json. That secret is refreshed daily by the service-account-jwt-token-refresh service and is the expected source for unattended service-to-service access.

Security considerations

All services exist on an internal docker network named braingeneers-net, this is inaccessible to the outside world and access to it is restricted by the Service Proxy, which checks authentication on every request. Internally services can communicate over http safely because the Service Proxy is the only entry point to the network.

Service account JWT tokens provide full access to all web services, they expire after 4 months and auto refresh themselves in braingeneerspy, this is an accepted security risk, trading off some security for convenience. If a token is compromised all tokens can be revoked in Auth0, however individual tokens cannot be revoked.

Best Practices

To ensure security and maintainability:

1. The services are designed to be stateless except for the ~/.kube/config requirement to retrieve the secrets.
2. Services can rely on the Kubernetes secrets and can access any state files via our standard S3 service at s3://braingeneers/ or other buckets.
3. Services that need local files should use service-scoped directories under the shared local or replicated Docker volumes. Mutable files belong in local; completed files that should be backed up belong in replicated.