Refresh dev environment on deploy

[tnetennba3]

Context

We would like to change the way that we handle and make use of the dev/preprod environments. To do this we want to reset the dev env back to a standard set of consultations in the db and in s3 on every deploy. This originally did include preprod, but there shouldn't be any preprod changes left in the code.

This will allow us to test on a fresh env in dev when we need to, but maintain preprod as a long-lived mirror of prod to enable design reviews and evaluation testing etc.

There will be a follow-up ticket to add all admin users to the created consultations, but this PR is big enough already

Changes proposed in this pull request

• Ruff has gone a bit crazy, so a lot of files are just superficial changes
• Added a script to generate dummy data
• Added a conditional check to only deploy to dev
• Updated tests to match

Guidance to review

• Check that dev gets redeployed with fresh data
• Check that deploying to preprod doesn't wipe the env

The files that are only ruff changes

• backend/consultations/api/serializers.py
• backend/consultations/api/views/consultation.py
• backend/consultations/api/views/question.py
• backend/consultations/api/views/response.py
• backend/consultations/migrations/0099_alter_consultation_stage_default.py
• backend/consultations/migrations/0100_remove_legacy_stage_choices.py
• backend/consultations/migrations/0102_responsereadby_alter_response_read_by.py
• backend/consultations/migrations/0103_add_response_question_id_index.py
• backend/consultations/utils/s3.py
• backend/data_pipeline/s3.py
• backend/data_pipeline/sync/candidate_themes.py
• backend/data_pipeline/sync/consultation_setup.py
• backend/data_pipeline/sync/response_annotations.py
• backend/factories.py
• backend/rq_context.py
• backend/settings/local.py
• backend/tests/unit/data_pipeline/sync/test_consultation_setup.py
• backend/tests/unit/data_pipeline/test_batch.py
• backend/tests/unit/data_pipeline/test_s3.py
• backend/tests/unit/test_healthcheck_worker.py
• backend/tests/unit/test_middleware.py
• backend/tests/unit/test_rq_context.py

These can likely be ignored or quickly reviewed

[linear]

PRO-350

[252afh]

prepare_s3.py:219 — help says "Only runs on deployed dev environments" but there's no is_deployed() guard. Running locally with ENVIRONMENT=DEV triggers real boto3 S3 calls and will error. Since prepare_environment calls prepare_s3, a local run with ENVIRONMENT=DEV wipes the DB and then blows up on S3.

This isn't a concern, we don't run locally with dev environment set

[github-actions]

Claude finished @252afh's task in 49s —— View job

---

PR Review

• Gather context
• Read key changed files
• Provide review feedback

---

Clean implementation — dev-only allowlist (is_dev()) in both commands, fully deterministic dummy data, and the duplicate tests are gone. Previous review concerns are resolved.

prepare_s3.py:214 — help says "Only runs on deployed dev environments" but there's no is_deployed() guard. Minor wording mismatch with what the code actually enforces.

⚪️ NIT — Update help string to match reality: "Only runs on dev environments (ENVIRONMENT=DEV)." or add the is_deployed() guard the string implies.

[georgia-i-ai]

LGTM, apart from the concern about syncing between s3 and db, but not a major issue for dummy data.

[georgia-i-ai]

Right, makes sense. Don't really like that these are set up quite separately - is there a danger of them getting out of sync? I know it's dummy data, but seems like it could get quite messy. Not something we need to fix here, but maybe something to raise as a tech debt issue to think about later if so.

[github-actions]

Claude finished @252afh's task in 1m 13s —— View job

---

PR Review

• Gather context
• Read key changed files
• Provide review feedback

---

Clean implementation — is_dev() guard is consistent across both commands, dummy data is deterministic, and the test refactor (patching the full class rather than individual methods) is cleaner.

prepare_environment.py:15 — No is_deployed() guard. Running locally with ENVIRONMENT=DEV drops the entire schema (DROP SCHEMA public CASCADE) and then calls prepare_s3 which will attempt real boto3 calls. prepare_s3 has the same gap. Author has noted this is intentional, but the destructive path (schema drop) has no safeguard.

🔵 CONSIDER — Add an is_deployed() guard, or at minimum a comment explaining why local ENVIRONMENT=DEV is safe. A schema drop without that guard is a footgun.

prepare_s3.py:214 — help text still says "Only runs on deployed dev environments" but the guard is just is_dev(), not is_dev() and is_deployed(). Flagged in the previous review as a NIT, still present.

⚪️ NIT — Fix this → Update help string to "Only runs on dev environments (ENVIRONMENT=DEV)." to match the actual guard.

[github-actions]

Claude finished @252afh's task in 37s —— View job

---

PR Review

• Gather context
• Read key changed files
• Provide review feedback

---

Clean implementation. The is_dev() guard is consistent, tests are clear, and the HostingEnvironment class patching is now simpler.

prepare_environment.py:15 — is_dev() alone gates the schema drop. Running locally with ENVIRONMENT=DEV will DROP SCHEMA public CASCADE and then fail on prepare_s3 (no real S3). Author confirmed this is intentional, but the destructive path has no safeguard.

🔵 CONSIDER — Add a comment or an is_deployed() guard to make the intent explicit and prevent accidental local data loss.

prepare_s3.py:214 — help text still reads "Only runs on deployed dev environments" but the guard at line 219 is just is_dev(), not is_dev() and is_deployed(). Flagged in two prior reviews, still present.

⚪️ NIT — Fix this → Update help to "Only runs on dev environments (ENVIRONMENT=DEV)." to match the actual guard.

[github-actions]

Claude finished @252afh's task in 49s —— View job

---

PR Review

• Gather context
• Read key changed files
• Provide review feedback

---

Clean implementation overall — is_dev() guard is consistent, dummy data is deterministic, and the DB/S3 seeding separation is clear.

prepare_s3.py:214 and prepare_environment.py:10 — Both help strings claim "deployed dev" but neither has an is_deployed() guard — only is_dev(). The prepare_s3.py NIT has been flagged in three prior reviews.

⚪️ NIT — Fix this → Fix both help strings to drop "deployed" — they only check is_dev().

[github-actions]

Claude finished @252afh's task in 26s —— View job

---

PR Review

• Gather context
• Read key changed files
• Provide review feedback

---

Clean implementation — is_dev() guard is consistent across both commands, prepare_environment.py:10 help text is now accurate ("deployed dev only"), and the create_response non-determinism is fixed (deterministic index via themefinder_id).

prepare_s3.py:214 — help text still reads "Only runs on deployed dev environments" but the guard at line 219 is just is_dev() — no is_deployed() check. This NIT has been raised in four prior reviews and remains unaddressed.

⚪️ NIT — Fix this → Change "Only runs on deployed dev environments." → "Only runs on dev environments (ENVIRONMENT=DEV).".