Poc/global redash

[imenattatra]

Closes https://github.com/metr-systems/backlog/issues/4651

Link to the Gemini summary of the refinement https://docs.google.com/document/d/12LUTT6X05AEdyO8HrtCoHTl-7zjGNkQ2JBLRoWaenIM/edit?tab=t.n7gc0govp32y

AI Disclaimer: I used clause sonnet 4 for brainstorming - help generating code faster

Summary

This PR implements a Proof of Concept for a separate global-redash application within the same repository, designed to handle cross-organizational operations. The implementation demonstrates the viability of sharing Redash models while maintaining service separation for global administrative tasks. It may not be merged at the end but it could be a base for something big so I am openning it as PR.

Code Strategy

1. Implementing a Global Redash Application

Separate Flask application that reuses existing Redash models and database , it has a clear separation between routes, services, and authentication for reusability. Offering an Independent deployment capability while sharing the same codebase.

2. Adding an Organization Creation Service

We added RESTful API endpoint: POST /global-api/organizations . This one does a complete organization setup: Creates org, admin/default groups, and admin user. It does that via a secure authentication: Token-based authentication via X-Global-Api-Token header

3. Handling Infrastructure & Deployment

We implemented a docker integration: adding a new global_server command in docker-entrypoint using a separate port 5001. It guarantees a separate worker configuration also (REDASH_GLOBAL_WEB_WORKERS)

4. Added REDASH_GLOBAL_API_TOKEN to redash staging secrets with a proper value

5 Implemented the infrastructure part of it that guaranteed a clear separation between redash and global-redash

check this PR: https://github.com/metr-systems/infrastructure/pull/84

6. We also added the dns entry to ionos.

QA

We did test it manually throught these steps :

• Deploy (check infrastructure part also)
• Run this command

curl -X POST https://global-dashboard.staging.metr.systems/global-api/organizations \
  -H "Content-Type: application/json" \
  -H "X-Global-Api-Token: ---" \
  -d '{
    "name": "new org",
    "slug": "new-org",
    "admin_email": "---",
    "admin_name": "---",
    "password": "---"
  }'

• Check that organization was created with
  kubectl -n staging exec -it dashboards-scheduler-68ccbb578d-tntv9 -- python manage.py org list

[thiagogds]

Nice! I think we can move forward with this strategy

[berinhard]

I left some comments but the blockers for me are related to:

1. HTTP return code for missing API tokens;
2. Proper handle of missing REDASH_GLOBAL_API_TOKEN;

All the other comments are refactoring suggestions or things to consider during future works in this PoC. But I really liked how that's plugable to Redash!

[berinhard]

Should we add more validation to this data here? For example, the admin_email is an email? THe slug is a valid slug?

I'd recommend marshmallow since it has an API that is closer to Django and, thus, it can mean a lower barrier of entrance for new developers.

[berinhard]

Pydantic can be an alternative too

[imenattatra]

Closing this PR since it will not be merged and will use the knowledge we got from reviews in the following POC : https://github.com/metr-systems/backlog/issues/4673