[Task]: Add performance monitoring to measure impact of MT-to-Celery migration

[hannaseithe]

⚙️ Task

Task Description

What

Add observability metrics to track machine translation (MT) request performance, so we can measure the impact of the async Celery migration (#4504) and detect any future regressions.

Why

PR #4504 moves bulk machine translation from synchronous request handling to a Celery task queue. The core motivation was preventing request timeouts: in the old code, a single bulk MT action could block a web worker for several minutes, causing cascading 504 errors across the whole system under concurrent load.

To demonstrate that the change has the intended effect — and to guard against regressions — we need metrics that capture MT endpoint behaviour both before and after the migration. Currently no such metrics exist.

---

Task Category

📊 Monitoring / Observability

---

Steps / Implementation Plan

• Clarify with admin what the existing latency measurement in the cms InfluxDB database is tracking and whether it is still actively fed
• Decide on approach for capturing HTTP response time (e.g. adding %D to Apache log format)
• Extend monitor-cms.sh with MT-specific metrics (request count, response time, 504 error count) — to be pushed before Move machine translation processing to Celery (Backend) #4504 deploys to establish a baseline
• After Move machine translation processing to Celery (Backend) #4504 deploys: add Celery-specific metrics (queue depth, task completions) in a follow-up salt PR
• Verify metrics appear correctly in Grafana (cms database, "Content Management Systems" dashboard)

---

Acceptance Criteria

• MT request count and response time are visible in Grafana before the Celery change goes live
• 504 error rate is tracked and visible
• After the Celery change deploys, queue depth and task completion rate are also tracked
• The before/after difference in MT endpoint response time is observable in the dashboard

---

Technical Details

Affected Systems / Services

• Apache on the production CMS server (potential log format change)
• monitor-cms.sh (salt-managed, runs every minute)
• InfluxDB cms database
• Grafana "Content Management Systems" dashboard

Configuration Changes

• Possibly add %D to Apache access log format to capture request duration — needs admin review
• Two salt PRs: one before Move machine translation processing to Celery (Backend) #4504 deploys (baseline), one after (Celery metrics)

---

Risks & Impact

Risks

• Changing the Apache log format affects all access log entries and any tooling that parses them (e.g. jpal-logs.sh)

Blast Radius

• Services affected: Apache logging, monitor-cms.sh
• Teams affected: Admin / DevOps
• User-facing impact: None

Coordination Required

• Coordinate with admin on Apache log format change and existing latency measurement

---

Parent / Related Issues

• Parent: Move machine translation processing to Celery (Backend) #4504 (Move machine translation processing to Celery)