test infra: backend suite speedup (Phase A1) — Python 3.12 + sysmon coverage (#1767)

* test infra: backend suite speedup — Python 3.12 + sysmon coverage + worksteal

Two changes that together produced a 4x speedup on the hot test files in
local A/B measurement (19:34 -> 4:50 on the WebSocket / extract /
structured-response cluster; see plan §6). Targets the two highest-ROI
findings from a CI runtime audit of the ~47-min backend job:

A1. Bump test/runtime Python image from 3.11.15 to 3.12.7 and set
    COVERAGE_CORE=sysmon in the test env. Coverage on 3.11 was measured at
    +156% over the no-cov baseline on the hot files; sys.monitoring is
    ~5-10x cheaper. Codecov receives the same coverage.xml — only the
    collection path changes. Dropped django_coverage_plugin from setup.cfg
    because Coverage 7.x silently falls back to the C-trace path whenever
    any file-tracer plugin is configured, which would defeat sysmon (the
    project owns 7 templates total, all 4xx/5xx pages plus a couple of
    admin views — a trivial coverage slice for a ~150% wall-time tax).

B1. Switch the pytest CI invocation from --dist loadscope to
    --dist worksteal so large classes like TestStructuredResponseAPI
    (843 s / 27 tests) can fan out across all workers instead of pinning
    to one. Conftest gains an auto-tag that pins django.test.TestCase
    subclasses with class-scoped state (setUpTestData overridden or
    fixtures set) to a per-class xdist_group, so worksteal still respects
    that binding where it matters. TransactionTestCase subclasses are
    deliberately not pinned — they rebuild per-test regardless, so they
    benefit from free redistribution.

Plan also covers Phase A2 (drop redundant migrate step), A3 (ghcr image
cache), A4 (explicit -n), B2 (slim 17 MB fixture), B3 (class-once fixture
load), B4 (TransactionTestCase audit). Tracked in
docs/refactor_plans/2026-05-23-test-suite-speedup-A1-B1.md so we can
re-measure after this lands and decide which further phases are needed.

* test infra: revert Phase B1 (worksteal + auto-tag); keep A1 only

The combined PR #1767 ran in ~31:53 (vs 41-min baseline, −22%) but failed
with 39 tests broken — psycopg2 UniqueViolation on the `admin` user
across plain TestCase subclasses (UserTypePrivacyTestCase,
TestSearchAgentsForMention, TestOpenContractsAnalyzers, etc).

Root cause: worksteal interleaved tests from these classes with tests
from other classes on the same worker, exposing state leaks that
loadscope's class-pinning was hiding. The auto-tag only pinned
TestCase subclasses with setUpTestData/fixtures, leaving plain
TestCase classes vulnerable to interleave-on-worker pollution from
sibling classes' uncommitted state.

This commit:
- Reverts .github/workflows/backend.yml to `--dist loadscope`.
- Reverts conftest.py::pytest_collection_modifyitems to its prior shape
  (serial marker handling only).
- Keeps the Phase A1 changes (Python 3.12.7 image, COVERAGE_CORE=sysmon,
  dropped django_coverage_plugin) — they ran cleanly and account for the
  ~22% wall-time reduction independently.
- Updates the plan doc (§10 post-mortem) with what we learned: the local
  A/B 4× speedup was specific to the 3 hottest files (~30% of CI time),
  not a sound extrapolation to the full suite. Coverage cost was a
  multiplier on the per-test 17 MB fixture-reload base cost, not the
  base itself. After A1, that fixture reload is the binding constraint
  and Phase B3 is the right next lever.

Worksteal can be re-attempted only after a dedicated test-isolation
sweep that finds and fixes the admin-user collision pattern (probably
in factories + Celery-eager + migration seeds).

* test: accept text/javascript or application/javascript in storage test

Python 3.12's mimetypes module returns ``text/javascript`` for ``.js``
files (per RFC 9239, which deprecates ``application/javascript``). The
test was hardcoded to the legacy value and broke when CI bumped the
django image to Python 3.12 in this PR. Both forms are RFC-valid and
both browsers serve them identically — relax the assertion so the test
passes on either runtime.

---------

Signed-off-by: JSIV <5049984+JSv4@users.noreply.github.com>

Author: JSv4

.envs/.test/.django

@@ -51,6 +51,15 @@ DJANGO_SETTINGS_MODULE=config.settings.test
 # ------------------------------------------------------------------------------
 TESTING=true
 
+# Coverage backend
+# ------------------------------------------------------------------------------
+# Use CPython 3.12+'s ``sys.monitoring`` for coverage.py instead of the legacy
+# C trace function. On the backend test suite this reduces coverage overhead
+# from ~+156% to ~+5-15% over the no-cov baseline (measured 2026-05-23 against
+# the heavy WebSocket / extract / structured-response files). Requires the
+# Django image to be built on Python 3.12+ — see compose/local/django/Dockerfile.
+COVERAGE_CORE=sysmon
+
 # Pipeline Vars
 # ------------------------------
 EMBEDDINGS_MICROSERVICE_URL=http://vector-embedder:8000

.github/workflows/backend.yml

@@ -164,13 +164,19 @@ jobs:
       - name: Run Backend Test Suite
         timeout-minutes: 100
         run: |
-          # Coverage instrumentation (coverage.py's C trace function under
-          # Python 3.11 — sys.monitoring/COVERAGE_CORE=sysmon needs 3.12+)
-          # adds noticeable wall time to the run, but Codecov's PR patch
-          # status needs a fresh report for the PR head commit — without one
-          # it compares against carryforward data and reports phantom
-          # coverage for newly added files. Measure coverage on both push
-          # and pull_request events so the patch gate reflects reality.
+          # Coverage instrumentation runs on both push and pull_request events
+          # so Codecov's PR patch status has a fresh report for the PR head
+          # commit. The Django image is built on Python 3.12 with
+          # ``COVERAGE_CORE=sysmon`` (set in .envs/.test/.django), so coverage
+          # uses ``sys.monitoring`` rather than the legacy C trace function —
+          # an order-of-magnitude cheaper instrumentation path on Python 3.12+.
+          #
+          # ``--dist loadscope`` is retained: a worksteal trial (PR #1767,
+          # initial commit) exposed ~39 test-order-dependency failures across
+          # plain ``TestCase`` subclasses that worked under loadscope's
+          # class-pinning. The underlying isolation bugs are tracked in
+          # docs/refactor_plans/2026-05-23-test-suite-speedup-A1-B1.md §6
+          # and need to be resolved before worksteal can be re-attempted.
           #
           # pytest-cov + pytest-xdist handle per-worker coverage merging.
           docker compose -f test.yml run django pytest --cov --cov-report=xml -n auto --dist loadscope

CHANGELOG.md

@@ -9,6 +9,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+- **Backend test suite speedup — Phase A1: Python 3.12 + `sys.monitoring` coverage** (plan in `docs/refactor_plans/2026-05-23-test-suite-speedup-A1-B1.md`). Targets the highest-ROI lever from a CI runtime audit of `gh run view 26334920376` (47-min wall, 41-min pytest step): coverage on Python 3.11 measured at **+156% over the no-cov baseline** on the hot WebSocket / extract / structured-response files. Measured CI delta after this change alone: backend pytest step ~41 min → ~32 min (~22% reduction; full audit of remaining bottlenecks captured in the plan doc for follow-up). A companion `--dist worksteal` change was trialled in PR #1767 and reverted — it exposed pre-existing test-isolation bugs across ~39 plain `TestCase` subclasses that were latent under loadscope's class-pinning, and needs a dedicated test-isolation sweep before it can be re-attempted.
+  - **Python 3.12.7 in the test/runtime image** (`compose/local/django/Dockerfile`, `compose/production/django/Dockerfile`). Bumps `ARG PYTHON_VERSION` from `3.11.15-slim-bookworm` to `3.12.7-slim-bookworm`. CI's linter job has been running on Python 3.12 for weeks, so dependency compatibility is already proven; this brings the django image in line.
+  - **`COVERAGE_CORE=sysmon` in the test env** (`.envs/.test/.django`). Switches coverage.py to `sys.monitoring`-backed instrumentation (the `SysMonitor` tracer class), ~5-10× cheaper than the legacy C trace function used on Python 3.11. Codecov receives the same `coverage.xml` content — only the collection path changes.
+  - **Dropped `django_coverage_plugin`** (`setup.cfg`). Coverage 7.x silently falls back to the legacy C trace path whenever any file-tracer plugin is configured, defeating `COVERAGE_CORE=sysmon`. The project owns 7 Django templates total (4xx/5xx pages plus a couple of admin views), so template coverage was a trivial slice of total coverage but the plugin was forcing the ~150% wall-time tax. Restore if/when coverage.py supports sysmon-compatible plugin tracers.
+
 - **`DocumentKnowledgeBase` desktop polish — consolidate the two right-edge control systems into one unified rail** (issue #1734; `frontend/src/components/knowledge_base/document/styled/SidebarTabs.tsx`, `frontend/src/components/knowledge_base/document/document_kb/SidebarTabs.tsx`, `frontend/src/components/knowledge_base/document/FloatingDocumentControls.tsx`, `frontend/src/components/knowledge_base/document/layouts/DesktopDocumentLayout.tsx`). At a desktop viewport the document layer's right edge previously rendered two competing vertical stacks: a rotated-text tab rail (`INDEX`/`CHAT`/`FEED`/`DISCUSSIONS`) and, just to its left, a separate floating action-button cluster (settings gear + colored circular FABs for extracts / analyses / create-analysis). The audit flagged this as visual clutter that made it ambiguous which control does what, and called out the rotated text labels as hard to scan.
   - **Visual unification.** When the right panel is closed, both control groups now render inside a single new `RightEdgeRail` wrapper (`DesktopDocumentLayout.tsx`), vertically centered against the viewport's right edge. The navigation tabs sit on top, a 1-pixel `RailDivider` separates the two control intents, and the document tool pills (settings, extracts, analyses, create-analysis) sit below — one coherent column with consistent pill shape, sizing, and active-state treatment.
   - **No more rotated text.** `SidebarTab` (styled) drops its `writing-mode: vertical-rl` label rendering. The `.tab-label` span stays in the DOM as a screen-reader-only string (visually clipped to 1×1px) so assistive tech still announces it; the visible affordance is the icon plus a CSS-only hover tooltip driven by the new optional `data-tooltip` attribute. The tooltip flips to the opposite side of the pill based on `$panelOpen` so it never overflows the viewport edge.

compose/local/django/Dockerfile

@@ -1,7 +1,12 @@
-# Pin to a specific patch (not the floating 3.11 alias) so rebuilds on
+# Pin to a specific patch (not the floating 3.12 alias) so rebuilds on
 # different dates do not silently pick up a new CPython point release.
 # Keep this in sync with compose/production/django/Dockerfile.
-ARG PYTHON_VERSION=3.11.15-slim-bookworm
+#
+# 3.12 unlocks ``sys.monitoring``-backed coverage (``COVERAGE_CORE=sysmon``),
+# which is ~5-10x cheaper than the C-trace path required on 3.11. With the
+# test suite running ``--cov`` on every push, that's the biggest single
+# wall-time lever in CI.
+ARG PYTHON_VERSION=3.12.7-slim-bookworm
 
 # ---------------------------------------------------------------------------- #
 # Build stage: full toolchain for compiling wheels (psycopg2, opencv, etc.)    #

compose/production/django/Dockerfile

@@ -1,8 +1,11 @@
-# Pin to a specific patch (not the floating 3.11 alias) so rebuilds on
+# Pin to a specific patch (not the floating 3.12 alias) so rebuilds on
 # different dates do not silently pick up a new CPython point release that
 # could change runtime behaviour. Bump deliberately when upstream ships a
 # CVE/feature release and CI is green against the new patch.
-ARG PYTHON_VERSION=3.11.15-slim-bookworm
+#
+# Kept in sync with compose/local/django/Dockerfile. 3.12 is required for the
+# ``sys.monitoring``-backed coverage path used in CI; see that file's header.
+ARG PYTHON_VERSION=3.12.7-slim-bookworm
 
 # ---------------------------------------------------------------------------- #
 # Build stage: full toolchain for compiling wheels (psycopg2, opencv, etc.)    #