Dockerization for the madrona portal - #62
Draft
pollardld wants to merge 162 commits into
Draft
Conversation
…ved compatibility
…r next software team meeting
…ortal - Create .env.example for environment variable configuration. - Update Dockerfile to streamline environment variable setup and application source copying. - Revise docker-compose.yml to enhance service configuration and health checks. - Modify entrypoint.sh for improved database connection handling and fixture loading. - Update requirements.txt and docker-requirements.txt for dependency management. - Introduce pyproject.toml for project metadata and tooling configuration. - Revise dev_requirements.txt to include testing and linting tools.
- Simplified URL patterns in `urls.py` by removing version checks for Wagtail and consolidating imports. - Updated URL patterns to ensure proper routing and removed deprecated endpoints. - Enhanced error handling for loading project-specific URLs. - Streamlined model imports in `models.py` by removing conditional imports based on Wagtail version. - Ensured unique constraints in `PortalRendition` model are consistently defined.
- entrypoint.sh: switch fixture from initial_data.json to initial_data_prod.json; also load scenarios/fixtures/initial_data.json for reference data; raise fresh-DB threshold from ==0 to <5 pages (initial_data migration creates 1 page); clear django_site before load to avoid PK conflict - urls.py: add social_django.urls under 'social' namespace to fix NoReverseMatch at /account/ - settings.py: add APPEND_SLASH=True Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
wagtail.search.models.Query was removed in Wagtail 7. The site fixture no longer uses natural-foreign keys so the Query table cleanup is no longer needed anyway. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The initial_data migration creates placeholder Wagtail pages whose tree paths conflict with pages in the fixture. Delete depth>1 pages before loaddata so the fixture can load its own page tree cleanly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- correct env file location (.env at project root, not docker/.env) - correct variable names (DB_* not SQL_*) - correct ports (5432, 6379, not 65432/8379) - document --profile full requirement - document buildx build requirement and why (BuildKit git-context caching) - document entrypoint sequence including fixture seeding and superuser bootstrap - add dev-only infrastructure mode (no app container) - update all docker compose commands with correct flags Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ssing sub-app package clones
…velopment overrides
…loyment instructions
…created before loading data
- Clear default values for sensitive information in .env.example and config.docker.ini.template - Change region name to West Coast Ocean in config.docker.ini.template - Add instructions for creating ini file in README.md
Add compose.base.yml to the madrona-portal
Continue with Docker decoupling by removing wcoa from Dockerfile
…and other references to wcoa
…xample.com and clarify URL patterns comment
… have all been moved to the `wcoa` project app repo.
…, and enhance documentation for madrona-portal base image
Remove legacy coupled files and CI split
…improve error handling
Update Docker workflow: set working directory for SHA extraction and …
…ture and project overlays
There was a problem hiding this comment.
Pull request overview
This PR shifts madrona-portal toward a Docker-first architecture by introducing a new Ubuntu-based base image build, a base compose layer, updated runtime bootstrap behavior, and refreshed documentation/config standards to support “thin” portal overlays (e.g., WCOA/MidA) maintained in their own repos.
Changes:
- Added a new core Docker build (Ubuntu 24.04) and runtime entrypoint, plus base compose scaffolding and env templates.
- Modernized configuration handling (typed env/config helpers + pytest coverage) and introduced a JSON-RPC compatibility shim for legacy clients.
- Cleaned up legacy artifacts (removed old roadmap/dev requirements/old Dockerfile) and updated docs to reflect the new workflow.
Reviewed changes
Copilot reviewed 112 out of 117 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| Vagrantfile | Updates synced-folder paths to madrona-portal naming. |
| scripts/vagrant_provision.sh | Marks Vagrant provisioning as legacy and points to Docker docs. |
| scripts/db-restore.sh | Adds a Docker-based Postgres restore helper script. |
| ROADMAP.md | Removes obsolete roadmap content. |
| requirements.txt | Reorganizes and tightens dependency constraints for Django/Wagtail-era upgrades. |
| README.md | Rewrites primary docs around Docker base-image + overlay model and local workflows. |
| pyproject.toml | Adds pytest/coverage/ruff/mypy configuration and minimal project metadata. |
| marco/portal/welcome_snippet/migrations/0006_welcomepageentry_media_image.py | Removes legacy unicode_literals import. |
| marco/portal/welcome_snippet/migrations/0005_welcomepage_body.py | Removes legacy unicode_literals import. |
| marco/portal/welcome_snippet/migrations/0004_auto_20150502_1649.py | Removes legacy unicode_literals import. |
| marco/portal/welcome_snippet/migrations/0003_auto_20150501_1850.py | Removes legacy unicode_literals import. |
| marco/portal/welcome_snippet/migrations/0002_welcomepage_use_on_site.py | Removes legacy unicode_literals import. |
| marco/portal/welcome_snippet/migrations/0001_initial.py | Removes legacy unicode_literals import. |
| marco/portal/pages/migrations/0003_auto_20150112_2308.py | Removes legacy unicode_literals import. |
| marco/portal/pages/migrations/0002_page_description.py | Removes legacy unicode_literals import. |
| marco/portal/pages/migrations/0001_initial.py | Removes legacy unicode_literals import. |
| marco/portal/ocean_stories/templates/ocean_stories/extra_js.html | Removes obsolete polyfill include. |
| marco/portal/ocean_stories/migrations/0012_auto_20160225_0012.py | Removes legacy unicode_literals import. |
| marco/portal/ocean_stories/migrations/0011_oceanstory_display_home_page.py | Removes legacy unicode_literals import. |
| marco/portal/ocean_stories/migrations/0010_auto_20150603_1721.py | Removes legacy unicode_literals import. |
| marco/portal/ocean_stories/migrations/0009_oceanstorysection_map_legend.py | Removes legacy unicode_literals import. |
| marco/portal/ocean_stories/migrations/0008_auto_20150203_2337.py | Removes legacy unicode_literals import. |
| marco/portal/ocean_stories/migrations/0007_auto_20150122_2225.py | Removes legacy unicode_literals import. |
| marco/portal/ocean_stories/migrations/0006_auto_20150121_2319.py | Removes legacy unicode_literals import. |
| marco/portal/ocean_stories/migrations/0005_auto_20150112_2302.py | Removes legacy unicode_literals import. |
| marco/portal/ocean_stories/migrations/0004_auto_20141219_2132.py | Removes legacy unicode_literals import. |
| marco/portal/ocean_stories/migrations/0003_oceanstory_feature_image.py | Removes legacy unicode_literals import. |
| marco/portal/ocean_stories/migrations/0002_auto_20141211_0121.py | Removes legacy unicode_literals import. |
| marco/portal/ocean_stories/migrations/0001_initial.py | Removes legacy unicode_literals import. |
| marco/portal/news/migrations/0003_auto_20160225_0012.py | Removes legacy unicode_literals import. |
| marco/portal/news/migrations/0002_auto_20150522_0047.py | Removes legacy unicode_literals import. |
| marco/portal/news/migrations/0001_initial.py | Removes legacy unicode_literals import. |
| marco/portal/menu/migrations/0007_auto_20171201_2126.py | Removes legacy unicode_literals import. |
| marco/portal/menu/migrations/0006_auto_20150521_2053.py | Removes legacy unicode_literals import. |
| marco/portal/menu/migrations/0005_auto_20150518_2309.py | Removes legacy unicode_literals import. |
| marco/portal/menu/migrations/0004_auto_20150122_2150.py | Removes legacy unicode_literals import. |
| marco/portal/menu/migrations/0003_menuentry_page.py | Removes legacy unicode_literals import. |
| marco/portal/menu/migrations/0002_menuentry_show_divider_underneath.py | Removes legacy unicode_literals import. |
| marco/portal/menu/migrations/0001_initial.py | Removes legacy unicode_literals import. |
| marco/portal/initial_data/migrations/0002_create_pages.py | Removes legacy unicode_literals import. |
| marco/portal/initial_data/migrations/0001_initial_data.py | Removes legacy unicode_literals import. |
| marco/portal/home/models.py | Simplifies Wagtail imports to a single modern path (drops version branching). |
| marco/portal/home/migrations/0018_auto_20171130_2057.py | Removes legacy unicode_literals import. |
| marco/portal/home/migrations/0017_auto_20171130_2030.py | Removes legacy unicode_literals import. |
| marco/portal/home/migrations/0016_auto_20171130_1854.py | Removes legacy unicode_literals import. |
| marco/portal/home/migrations/0015_homepagecardset.py | Removes legacy unicode_literals import. |
| marco/portal/home/migrations/0014_auto_20171123_0025.py | Removes legacy unicode_literals import. |
| marco/portal/home/migrations/0013_auto_20171123_0013.py | Removes legacy unicode_literals import. |
| marco/portal/home/migrations/0012_homepagecards.py | Removes legacy unicode_literals import. |
| marco/portal/home/migrations/0011_remove_homepagecarousel_link.py | Removes legacy unicode_literals import. |
| marco/portal/home/migrations/0010_auto_20171121_2246.py | Removes legacy unicode_literals import. |
| marco/portal/home/migrations/0009_homestream.py | Removes legacy unicode_literals import. |
| marco/portal/home/migrations/0008_auto_20171121_0042.py | Removes legacy unicode_literals import. |
| marco/portal/home/migrations/0007_auto_20171120_2023.py | Removes legacy unicode_literals import. |
| marco/portal/home/migrations/0006_auto_20171120_2017.py | Removes legacy unicode_literals import. |
| marco/portal/home/migrations/0005_remove_homepage_feature_image.py | Removes legacy unicode_literals import. |
| marco/portal/home/migrations/0004_auto_20171118_0027.py | Removes legacy unicode_literals import. |
| marco/portal/home/migrations/0003_homepagecarousel.py | Removes legacy unicode_literals import. |
| marco/portal/home/migrations/0002_create_homepage.py | Removes legacy unicode_literals import. |
| marco/portal/home/migrations/0001_initial.py | Removes legacy unicode_literals import. |
| marco/portal/grid_pages/migrations/0004_auto_20160225_0012.py | Removes legacy unicode_literals import. |
| marco/portal/grid_pages/migrations/0003_auto_20150429_1844.py | Removes legacy unicode_literals import. |
| marco/portal/grid_pages/migrations/0002_auto_20150429_1843.py | Removes legacy unicode_literals import. |
| marco/portal/grid_pages/migrations/0001_initial.py | Removes legacy unicode_literals import. |
| marco/portal/data_gaps/migrations/0005_auto_20150121_2319.py | Removes legacy unicode_literals import. |
| marco/portal/data_gaps/migrations/0004_auto_20150112_2303.py | Removes legacy unicode_literals import. |
| marco/portal/data_gaps/migrations/0003_auto_20141217_2301.py | Removes legacy unicode_literals import. |
| marco/portal/data_gaps/migrations/0002_auto_20141205_0024.py | Removes legacy unicode_literals import. |
| marco/portal/data_gaps/migrations/0001_initial.py | Removes legacy unicode_literals import. |
| marco/portal/data_catalog/migrations/0002_datacatalog_description.py | Removes legacy unicode_literals import. |
| marco/portal/data_catalog/migrations/0001_initial.py | Removes legacy unicode_literals import. |
| marco/portal/calendar/migrations/0007_auto_20150121_2313.py | Removes legacy unicode_literals import. |
| marco/portal/calendar/migrations/0006_auto_20150112_2303.py | Removes legacy unicode_literals import. |
| marco/portal/calendar/migrations/0005_auto_20150109_0053.py | Removes legacy unicode_literals import. |
| marco/portal/calendar/migrations/0004_event_location.py | Removes legacy unicode_literals import. |
| marco/portal/calendar/migrations/0003_auto_20141218_0154.py | Removes legacy unicode_literals import. |
| marco/portal/calendar/migrations/0002_auto_20141205_0036.py | Removes legacy unicode_literals import. |
| marco/portal/calendar/migrations/0001_initial.py | Removes legacy unicode_literals import. |
| marco/portal/base/models.py | Simplifies Wagtail imports; removes debug print; consolidates Meta definition. |
| marco/portal/base/migrations/0004_auto_20200613_0023.py | Removes legacy unicode_literals import. |
| marco/portal/base/migrations/0003_auto_20200526_2357.py | Removes legacy unicode_literals import. |
| marco/portal/base/migrations/0002_auto_20200526_2354.py | Removes legacy unicode_literals import. |
| marco/marco/urls.py | Refactors URL config; adds API auto-discovery and RPC compat view routing. |
| marco/marco/tests/test_config_helpers.py | Adds tests enforcing config helper precedence/parsing. |
| marco/marco/tests/test_api_url_discovery.py | Adds tests asserting auto-mounted /api/ routes resolve. |
| marco/marco/tests/init.py | Adds test package init. |
| marco/marco/settings.py | Major modernization: typed env/config precedence, updated app lists, Redis/Celery changes, stricter SECRET_KEY handling. |
| marco/marco/rpc_compat.py | Adds JSON-RPC 2.0 compat shim backing /rpc/. |
| marco/marco/config_helpers.py | Adds typed helper functions for env/config parsing. |
| marco/marco/celery.py | Moves dbwatch startup to Celery worker lifecycle instead of Django AppConfig. |
| marco/marco/apps.py | Removes AppConfig.ready side effects and keeps config minimal. |
| marco/marco_site/static/js/jsonrpc.js | Updates RPC endpoint URL to /rpc/. |
| marco/config.ini.dev | Removes legacy Vagrant-era config. |
| marco/config.docker.ini.template | Adds Docker-focused ini template for WCOA-style dev. |
| logs_and_config.md | Removes outdated server paths note. |
| docs/DOCKER_README.md | Adds Docker development guide. |
| docs/CONFIGURATION_STANDARD.md | Documents typed configuration precedence standard. |
| docs/AWS_DEPLOY.md | Adds an extensive AWS/EC2 deployment guide for containerized prod. |
| Dockerfile | Removes legacy top-level Alpine Dockerfile. |
| docker/nginx-dev.conf | Adds an nginx config intended for dev proxying. |
| docker/media/init.py | Keeps docker/media/ as a package/directory placeholder. |
| docker/entrypoint.sh | Reworks entrypoint to wait for DB, manage static/compress, optionally init DB, and select server mode. |
| docker/Dockerfile | Adds new Ubuntu-based base image build that copies in core + sub-app repos. |
| docker/docker-requirements.txt | Updates Docker requirements to modern Django/Wagtail and adds gunicorn/DRF/etc. |
| docker/docker-compose.yml | Removes legacy compose file. |
| docker/compose.base.yml | Adds a base compose layer for app/db/redis service skeleton. |
| docker/backups/elasticsearch/blank.txt | Placeholder to keep backup directory in repo. |
| docker/.env.example | Adds a core .env template for Docker usage. |
| docker/.env | Removes tracked .env. |
| dev_requirements.txt | Removes legacy dev requirements file. |
| deployment/crontab.template | Adds template cron jobs for backups/snapshots/import tasks. |
| backups/load_sql_dump.sh | Adds a helper to load a SQL dump into the running db container. |
| backups/dump_fixtures.sh | Adds a helper to regenerate fixture files from a running DB. |
| backups/db_dump.sh | Rewrites DB dump script to operate against docker compose DB service. |
| backups/create_elastic_snapshot.sh | Adds a helper to trigger Elasticsearch snapshots. |
| .gitignore | Updates ignores for docker runtime dirs and local env/config artifacts. |
| .github/workflows/create-and-publish-docker-images.yml | Adds GH Actions workflow to build/push base image to GHCR on main merges. |
Comments suppressed due to low confidence (1)
README.md:240
- Same issue as the earlier
docker compose upcommand: without specifying-f compose.base.yml, this won't run because there is no default compose file inmadrona-portal/docker.
Once you have a populated DB (either dummy or with migrated data) omit the `DB_INIT=1`:
```bash
docker compose up
</details>
---
💡 <a href="/Ecotrust/madrona-portal/new/main?filename=.github/instructions/*.instructions.md" class="Link--inTextBlock" target="_blank" rel="noopener noreferrer">Add Copilot custom instructions</a> for smarter, more guided reviews. <a href="https://docs.github.com/en/copilot/customizing-copilot/adding-repository-custom-instructions-for-github-copilot" class="Link--inTextBlock" target="_blank" rel="noopener noreferrer">Learn how to get started</a>.
Comment on lines
+44
to
+46
| COPY madrona-portal/marco ./marco | ||
| COPY madrona-portal/apps/__init__.py ./apps/__init__.py | ||
| COPY madrona-portal/assets ./assets |
Comment on lines
+1
to
+3
| services: | ||
| app: | ||
| environment: |
Comment on lines
+180
to
+188
| if [ "${DJANGO_ENV:-}" = "production" ] || [ "${DJANGO_DEBUG}" = "false" ]; then | ||
| echo "Starting gunicorn (production mode)..." | ||
| exec gosu madrona_user gunicorn marco.wsgi:application \ | ||
| --bind 0.0.0.0:8008 \ | ||
| --workers "${GUNICORN_WORKERS:-3}" \ | ||
| --timeout "${GUNICORN_TIMEOUT:-120}" \ | ||
| --chdir marco \ | ||
| --access-logfile - \ | ||
| --error-logfile - |
Comment on lines
+130
to
+134
| # Default routing - everything else goes to Django app | ||
| location / { | ||
| set $app_backend http://app:8008; | ||
| proxy_pass $app_backend; | ||
| proxy_redirect off; |
Comment on lines
+50
to
+51
| COMPOSE_DEV="docker/docker-compose.yml" | ||
| COMPOSE_PROD="docker/docker-compose.prod.yml" |
|
|
||
| PROJECT_SETTINGS_FILE = app_cfg.get('PROJECT_SETTINGS_FILE', False) | ||
| if PROJECT_SETTINGS_FILE and not PROJECT_SETTINGS_FILE == 'False': | ||
| PROJECT_SETTINGS_FILE = env_bool('PROJECT_SETTINGS_FILE', app_cfg, 'PROJECT_SETTINGS_FILE', '') |
Comment on lines
+20
to
+29
| # Load credentials from the Docker env file (same directory as docker-compose.yml) | ||
| set -a | ||
| # shellcheck source=/dev/null | ||
| source "$(dirname "$0")/../docker/.env.dev" | ||
| set +a | ||
|
|
||
| echo "Importing $(basename "$DUMP_FILE") into database '$SQL_DATABASE' ..." | ||
|
|
||
| docker compose --env-file docker/.env.dev -f docker/docker-compose.yml exec -T db \ | ||
| psql -U "$SQL_USER" -d "$SQL_DATABASE" < "$DUMP_FILE" |
Comment on lines
+222
to
+224
| ```bash | ||
| DB_INIT=1 docker compose up | ||
| ``` |
This was referenced Jul 29, 2026
…tform support for Docker images
| @@ -0,0 +1,45 @@ | |||
| #!/bin/bash | |||
| # Regenerate WCOA fixture files from the current running database. | |||
There was a problem hiding this comment.
should this be in here if its wcoa specific?
| @@ -1,6 +1,5 @@ | |||
| # -*- coding: utf-8 -*- | |||
| # Generated by Django 1.9.13 on 2020-05-26 23:54 | |||
| from __future__ import unicode_literals | |||
There was a problem hiding this comment.
what is the need for this change?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR is the core platform side of the Docker decoupling effort. It restructures madrona-portal into a reusable base image + base compose layer, then aligns dev and production workflows around that model so portal-specific repos (notably
wcoaandmida-portal) can build thin overlay images on top of the shared core.TLDR; madrona-portal now owns shared runtime/platform concerns, while portal repos own their project-specific Dockerfiles, compose overlays, config, and release pipelines.
This PR introduces Docker in a way that:
What this PR does
Adds Docker to main (new capability)
Defines decoupled architecture for shared platform
madrona-portal now owns shared/container-platform concerns:
... TODO