12 - API (feat): Switch to Advanced Alchemy

api/docs/adr/000-vertical-slice-architecture.md

@@ -86,7 +86,7 @@ damnit_api/
 ├── auth/            # OIDC flow, sessions, tokens, users, authz policy
 ├── contextfile/     # context-file (REST) endpoints
 ├── mymdc/           # MyMdC port
-├── appdb/           # application-DB engine/session/models
+├── appdb/           # application-DB engine/session/models; see ADR-010
 │
 └── graphql/         # transport composition only (ADR-007):
     ├── schema.py    #   assemble Query/Subscription from feature gql modules

api/docs/adr/009-channels-subscriptions.md

@@ -8,7 +8,7 @@ date: 2026-07-08
 
 The frontend's live table updates use a GraphQL subscription (`latest_data`) over `graphql-transport-ws`. DAMNIT proposal databases are plain SQLite files written by an external process. There is no change feed to subscribe to today, so the server must produce one by polling.
 
-That will not stay true. Candidate event sources exist on the horizon: the facility Kafka bus, or Postgres `LISTEN/NOTIFY` should the application database move to Postgres. The subscription design must not weld resolvers to any one change-detection mechanism.
+That will not stay true. Candidate event sources exist on the horizon: the facility Kafka bus, or Postgres `LISTEN/NOTIFY` should the application database ([ADR-010](010-two-databases.md)) move to Postgres. The subscription design must not weld resolvers to any one change-detection mechanism.
 
 Two forces shape the design. Naive per-client polling multiplies identical reads: N subscribers to one proposal would issue N queries per tick against a GPFS-hosted SQLite file. And any process-local coordination state is a deployment constraint that must be a recorded decision with a retirement path, not an accident.
 

api/docs/adr/010-two-databases.md

@@ -0,0 +1,37 @@
+---
+date: 2026-07-08
+---
+
+# ADR-010 - Two databases: application DB vs per-proposal DAMNIT DBs
+
+## Context and Problem Statement
+
+The API touches two databases with different owners and lifecycles.
+
+Per-proposal DAMNIT databases (`{proposal_dir}/{damnit_subdir}/runs.sqlite`) are external. One file exists per proposal, on GPFS, written by the DAMNIT listener. The API reflects their schema at runtime and treats them as read-only.
+
+The application database (`dw_api.sqlite`) is owned by this service. Its schema and lifecycle are ours to control, and it is the only database this service writes to.
+
+Nothing structurally prevents application code from wiring the two together through a shared, generic `db.py`/`get_session` pair, which would let a query meant for one database silently run against the other.
+
+## Considered Options
+
+- Structural separation: disjoint packages, entry points, and names per database.
+- A generic shared DB layer (one `db.py`, one `get_session`) serving both.
+- Convention and documentation only.
+
+## Decision Outcome
+
+Chosen option: "structural separation", because the two stacks get disjoint entry points and distinct types, so the wrong-database mistake cannot compile.
+
+`appdb`-owned code (`state.py`, `main.py`'s `SQLAlchemyAsyncConfig`, `metadata/repository.py`) is the only writer of `dw_api.sqlite`. `runs/` repositories ([ADR-005](005-repository-pattern.md)) remain the only reader of DAMNIT proposal databases. The naming convention is fixed: "app DB" always means `dw_api.sqlite`; "DAMNIT DB" or "proposal DB" always means a `runs.sqlite`.
+
+### Consequences
+
+- Good: the wrong-database failure mode disappears structurally instead of by discipline.
+- Good: the app DB's engine/session lifecycle has one home (`main.py`'s `SQLAlchemyPlugin`), so a future second config (a shared or Postgres backend) has a clear place to slot in without disturbing `runs/`.
+- Bad: there are two parallel data-access stacks, with no shared session helper by design.
+
+## Details
+
+Possible future: a move to Postgres for the app DB would also make `LISTEN/NOTIFY` available as a push publisher for run-update subscriptions - see [ADR-009](009-channels-subscriptions.md)'s publisher selection.

api/docs/architecture.md

@@ -23,7 +23,7 @@ For more information, see [ADR-000](adr/000-vertical-slice-architecture.md).
 | `auth/`                           | Authentication and authorisation        | OAuth flow, sessions, token store, `User`, permission classes, the membership policy                                      | Partial | Policy still in `metadata/services.py`                                    |
 | `contextfile/`                    | Context-file viewing                    | File reading, watching, its routes                                                                                        | Done    | As-is                                                                     |
 | `graphql/`                        | GraphQL transport only                  | Schema assembly, context, directives, controller binding - no resolvers, no domain logic (see [ADR-007](adr/007-graphql-transport-only.md)) | Partial | Assembly still in `shared/gql.py`; resolvers still here                   |
-| `appdb/`                          | The app's own database (infrastructure) | Models, engine/session plumbing for `dw_api.sqlite`                                                                       | Planned | `_db/`                                                                    |
+| `appdb/`                          | The app's own database (infrastructure) | Models, engine/session plumbing for `dw_api.sqlite` (see [ADR-010](adr/010-two-databases.md))                            | Partial | `_db/` (Advanced Alchemy `SQLAlchemyPlugin`); `metadata/repository.py`    |
 | `mymdc/`                          | MyMdC client (infrastructure)           | Ports, clients, vendored models                                                                                           | Planned | `_mymdc/`                                                                 |
 | `core/`                           | Cross-cutting, framework-free           | Shared error classes (see [ADR-001](adr/001-error-classes.md)), `DamnitType`, value types (see [ADR-004](adr/004-proposal-path-locator.md)), converters | Planned | `shared/` + `utils.py`                                                    |
 | `main.py` / `app.py` / `state.py` | Composition root                        | `AppState`, `create_*` factories (see [ADR-002](adr/002-no-global-mutable-state.md)), `create_app()` - the only place that may import everything and read settings | Partial | `main.py` + `state.py`                                                            |

api/pyproject.toml

@@ -38,6 +38,7 @@ dependencies = [
     "sqlmodel>=0.0.31",
     "pyyaml>=6.0.3",
     "litestar~=2.24.0",
+    "advanced-alchemy>=1.11.0",
 ]
 
 [dependency-groups]

api/src/damnit_api/_db/dependencies.py

@@ -1,17 +1,7 @@
-"""Litestar dependency helpers for database sessions."""
+"""Database session type alias for Litestar handlers."""
 
-from collections.abc import AsyncIterator
+from sqlalchemy.ext.asyncio import AsyncSession
 
-from litestar.datastructures import State
-from sqlmodel.ext.asyncio.session import AsyncSession
-
-
-async def get_session(state: State) -> AsyncIterator[AsyncSession]:
-    """Provide a database session from the application state."""
-
-    async with state.app_state.db_sessionmaker() as session:  # type: ignore[attr-defined]
-        yield session
-
-
-# Plain type alias; Litestar injects by the parameter name `session`.
+# Type alias used for annotations in other modules; the session itself is
+# provided by the Advanced Alchemy plugin (session_dependency_key="session").
 DBSession = AsyncSession

api/src/damnit_api/auth/dependencies.py

@@ -6,7 +6,7 @@
 from authlib.integrations.httpx_client import AsyncOAuth2Client
 from litestar import Request
 from litestar.datastructures import State
-from sqlmodel.ext.asyncio.session import AsyncSession
+from sqlalchemy.ext.asyncio import AsyncSession
 
 from .._mymdc.dependencies import MyMdCClient
 from .models import OAuthUserInfo as _OAuthUserInfo

api/src/damnit_api/main.py

@@ -5,7 +5,6 @@
 from litestar.exceptions import HTTPException
 
 from . import contextfile, metadata
-from ._db.dependencies import get_session
 from ._mymdc.dependencies import get_mymdc_client
 from .auth.dependencies import get_oauth_user_info, get_user
 
@@ -14,6 +13,11 @@
 
 
 def create_app():
+    from advanced_alchemy.extensions.litestar import (
+        AsyncSessionConfig,
+        SQLAlchemyAsyncConfig,
+        SQLAlchemyPlugin,
+    )
     from litestar import Request, Response
     from litestar.channels import ChannelsPlugin
     from litestar.channels.backends.memory import MemoryChannelsBackend
@@ -23,6 +27,7 @@ def create_app():
     from litestar.stores.file import FileStore
     from litestar.stores.memory import MemoryStore
     from litestar.stores.registry import StoreRegistry
+    from sqlmodel import SQLModel
 
     from . import _logging, auth, get_logger
     from .auth.oauth import SESSION_COOKIE_KEY, create_oauth_client
@@ -34,8 +39,6 @@ def create_app():
     from .shared.settings import settings
     from .state import (
         AppState,
-        create_db_engine,
-        create_db_sessionmaker,
         create_mymdc_client,
         create_repositories,
         provide_app_state,
@@ -50,6 +53,20 @@ def create_app():
     # (in-memory locally, file-backed otherwise) and selected below.
     session_config = ServerSideSessionConfig(key=SESSION_COOKIE_KEY)
 
+    # ── App database (ADR-010: `appdb`, dw_api.sqlite) ───────────────────────
+    # Advanced Alchemy owns engine/session lifecycle and provides the
+    # per-request `session` dependency (commit-on-success). Distinct
+    # dependency/state keys so a second config (e.g. a future DAMNIT
+    # Postgres) can coexist.
+    alchemy_config = SQLAlchemyAsyncConfig(
+        connection_string=f"sqlite+aiosqlite:///{settings.db_path}",
+        session_config=AsyncSessionConfig(expire_on_commit=False),
+        session_dependency_key="session",
+        engine_dependency_key="appdb_engine",
+        metadata=SQLModel.metadata,
+        create_all=True,
+    )
+
     # ── Channels (run-update pub/sub) ─────────────────────────────────────────
     # Subscribers consume per-proposal channels; the composition-selected
     # publisher (built in the lifespan) produces the events. The in-memory
@@ -114,8 +131,6 @@ async def lifespan(app: Litestar):
 
         logger.info("Starting application lifespan")
 
-        engine = create_db_engine(settings)
-
         oauth_client = create_oauth_client(settings)
         if oauth_client is not None:
             await oauth_client.load_server_metadata()
@@ -127,8 +142,7 @@ async def lifespan(app: Litestar):
         )
 
         app.state.app_state = AppState(
-            db_engine=engine,
-            db_sessionmaker=create_db_sessionmaker(engine),
+            db_sessionmaker=alchemy_config.create_session_maker(),
             mymdc_client=create_mymdc_client(settings),
             oauth_client=oauth_client,
             repositories=repositories,
@@ -140,7 +154,6 @@ async def lifespan(app: Litestar):
             yield
         finally:
             await run_update_publisher.aclose()
-            await engine.dispose()
 
     def _file_store(name: str) -> FileStore:
         # FileStore does not create its directory on write; the session read
@@ -176,7 +189,7 @@ def _file_store(name: str) -> FileStore:
             "oauth_config": Provide(
                 auth.dependencies.get_oauth_client, sync_to_thread=False
             ),
-            "session": Provide(get_session),
+            # The `session` dependency comes from the Advanced Alchemy plugin.
             "mymdc": Provide(get_mymdc_client, sync_to_thread=False),
             "user": Provide(get_user),
             "oauth_user": Provide(get_oauth_user_info, sync_to_thread=False),
@@ -186,7 +199,7 @@ def _file_store(name: str) -> FileStore:
             ),
             "repositories": Provide(get_repositories, sync_to_thread=False),
         },
-        plugins=[channels_plugin],
+        plugins=[channels_plugin, SQLAlchemyPlugin(config=alchemy_config)],
         stores=stores,
         middleware=[
             session_config.middleware,

api/src/damnit_api/metadata/repository.py

@@ -0,0 +1,22 @@
+"""App-DB repository for cached proposal metadata (Advanced Alchemy).
+
+This is the app database's data access (ADR-010: `dw_api.sqlite`, writable) -
+distinct from the read-only `DamnitRepository` over per-proposal DAMNIT files.
+"""
+
+from advanced_alchemy.repository import SQLAlchemyAsyncRepository
+
+from .models import ProposalMeta
+
+
+class ProposalMetaRepository(
+    SQLAlchemyAsyncRepository[ProposalMeta]  # ty: ignore[invalid-type-arguments]
+):
+    """CRUD/upsert access to the proposal-metadata cache.
+
+    SQLModel `table=True` models satisfy Advanced Alchemy's `ModelProtocol` at
+    runtime (they have `__table__`/`__mapper__`), but ty cannot verify that
+    structurally.
+    """
+
+    model_type = ProposalMeta

api/src/damnit_api/metadata/routers.py

@@ -2,7 +2,7 @@
 
 from litestar import Router, get
 from litestar.di import Provide
-from sqlmodel.ext.asyncio.session import AsyncSession
+from sqlalchemy.ext.asyncio import AsyncSession
 
 from .._mymdc.dependencies import MyMdCClient
 from ..auth.models import User

api/src/damnit_api/metadata/services.py

@@ -5,13 +5,14 @@
 from pathlib import Path
 from typing import TYPE_CHECKING
 
+from advanced_alchemy.filters import CollectionFilter
 from anyio import Path as APath
-from sqlmodel import col, select
 
 from .. import get_logger
 from ..shared.errors import ForbiddenError
 from ..shared.models import ProposalNumber
 from .models import ProposalMeta, ProposalMetaBase
+from .repository import ProposalMetaRepository
 
 logger = get_logger()
 
@@ -192,21 +193,17 @@ async def _get_proposal_meta(
 ) -> ProposalMeta:
     """Get proposal metadata by proposal number, using the repository and/or provided
     MyMdC Client."""
+    repo = ProposalMetaRepository(session=session, auto_commit=True)
 
-    statement = select(ProposalMeta).where(ProposalMeta.number == proposal_number)
-    result = (await session.exec(statement)).one_or_none()
-
+    result = await repo.get_one_or_none(number=proposal_number)
     if result:
         await logger.ainfo(
             "Loaded proposal metadata from repository", proposal=proposal_number
         )
         return result
 
     fetched = await _fetch_proposal_meta(client, proposal_number)
-    result = ProposalMeta(**fetched.model_dump())
-    session.add(result)
-    await session.commit()
-    return result
+    return await repo.add(ProposalMeta(**fetched.model_dump()))
 
 
 def _chunks(list_, n=10):
@@ -221,9 +218,7 @@ async def _get_proposal_meta_many(
     only_with_damnit: bool = True,
     start_after: datetime | None = None,
 ) -> list[ProposalMeta]:
-    statement = select(ProposalMeta).where(
-        col(ProposalMeta.number).in_(proposal_numbers)
-    )
+    repo = ProposalMetaRepository(session=session, auto_commit=True)
 
     filters = []
     if only_with_damnit:
@@ -232,7 +227,11 @@ async def _get_proposal_meta_many(
     if start_after:
         filters.append(lambda p: p.start_date and p.start_date >= start_after)
 
-    results = list((await session.exec(statement)).all())
+    results = list(
+        await repo.get_many(
+            CollectionFilter(field_name="number", values=proposal_numbers)
+        )
+    )
     missing = set(proposal_numbers) - {p.number for p in results}
 
     if not missing:
@@ -248,9 +247,7 @@ async def _get_proposal_meta_many(
             for p in new_fetched
             if not isinstance(p, BaseException)
         ]
-        session.add_all(new)
-        await session.commit()
-        results.extend(new)
+        results.extend(await repo.add_many(new))
 
     return [p for p in results if all(f(p) for f in filters)]
 
@@ -280,24 +277,10 @@ async def _update_proposal_meta(
 ) -> ProposalMeta:
     fetched = await _fetch_proposal_meta(client, proposal_number)
 
-    # Upsert into DB
-    statement = select(ProposalMeta).where(ProposalMeta.number == proposal_number)
-    result = (await session.exec(statement)).one_or_none()
-
-    if not result:
-        new = ProposalMeta(**fetched.model_dump())
-        session.add(new)
-        await session.commit()
-        return new
-
-    for key, value in fetched.model_dump().items():
-        if getattr(result, key) != value:
-            setattr(result, key, value)
-
-    await session.commit()
-    await session.refresh(result)
-
-    return result
+    repo = ProposalMetaRepository(session=session, auto_commit=True)
+    return await repo.upsert(
+        ProposalMeta(**fetched.model_dump()), match_fields=["number"]
+    )
 
 
 async def update_proposal_meta(

api/src/damnit_api/state.py

@@ -14,11 +14,12 @@
 from litestar.datastructures import (
     State as LitestarState,  # noqa: TC002 - Litestar inspects annotations at runtime via get_type_hints
 )
-from sqlalchemy.ext.asyncio import AsyncEngine, async_sessionmaker, create_async_engine
-from sqlmodel.ext.asyncio.session import AsyncSession
 
 if TYPE_CHECKING:
+    from collections.abc import Callable
+
     from litestar.channels import ChannelsPlugin
+    from sqlalchemy.ext.asyncio import AsyncSession
 
     from ._mymdc.clients import MyMdCClient
     from .auth.oauth import OAuthClient
@@ -29,24 +30,19 @@
 
 @dataclass(frozen=True)
 class AppState:
-    db_engine: AsyncEngine
-    db_sessionmaker: async_sessionmaker[AsyncSession]
+    # Session factory from the Advanced Alchemy config (main.py); held here
+    # for non-request contexts (e.g. the proposal-membership guard).
+    # Advanced Alchemy's create_session_maker() is typed as this Callable,
+    # not as async_sessionmaker[AsyncSession] (its actual runtime type in
+    # the non-routing case) - match its declared type here.
+    db_sessionmaker: Callable[[], AsyncSession]
     mymdc_client: MyMdCClient
     oauth_client: OAuthClient | None  # None when auth is disabled
     repositories: DamnitRepositoryRegistry
     channels: ChannelsPlugin
     run_update_publisher: RunUpdatePublisher
 
 
-def create_db_engine(settings: Settings) -> AsyncEngine:
-    db_url = f"sqlite+aiosqlite:///{settings.db_path}"
-    return create_async_engine(db_url, echo=False, future=True)
-
-
-def create_db_sessionmaker(engine: AsyncEngine) -> async_sessionmaker[AsyncSession]:
-    return async_sessionmaker(bind=engine, class_=AsyncSession, expire_on_commit=False)
-
-
 def create_mymdc_client(settings: Settings) -> MyMdCClient:
     from ._mymdc import clients
     from ._mymdc.settings import MyMdCHTTPSettings, MyMdCMockSettings