Backend reorganization: api / services / database / tasks / utils (Phase 7)

.github/instructions/backend.instructions.md

@@ -45,22 +45,32 @@ backend/
 │   ├── logging_context.py     # Logging context utilities
 │   └── timing_middleware.py   # Request timing middleware
 │
-├── core/                      # Core business logic
-│   ├── files_handler.py       # File operations handler (needs to be break down and move to files package)
-│   ├── base/                  # Shared base components
-│   │   ├── database/          # Database layer
-│   │   │   ├── models/        # SQLModel table definitions
-│   │   │   ├── manager/       # CRUD operations per model
-│   │   │   └── utils/         # DB engine, session management
-│   │   ├── arr_manager/       # Base Arr API client
-│   │   ├── connection_manager.py # Arr Connection manager (refresh data)
-│   │   └── utils/             # Shared utilities
-│   ├── radarr/                # Radarr-specific logic
-│   ├── sonarr/                # Sonarr-specific logic
-│   ├── download/              # download logic (images, trailers)
-│   ├── files/                 # File management
-│   ├── tasks/                 # Background task definitions
-│   └── updates/               # Update checking logic
+├── services/                  # Business logic
+│   ├── connections/           # Arr and Plex integrations
+│   │   ├── base.py            # BaseConnectionManager (shared refresh, CRUD)
+│   │   ├── probe.py           # Network probes against a server
+│   │   ├── service.py         # Create and update a connection
+│   │   ├── arr/               # Base Arr client, with radarr/ and sonarr/
+│   │   └── plex/              # Plex client, parser and models
+│   ├── trailers/              # Trailer download, search and conversion
+│   ├── files/                 # File operations and the media scanner
+│   ├── diagnostics/           # Connection Doctor, health checks, cookies
+│   ├── images/                # Image download
+│   ├── notifications/         # Apprise dispatch
+│   ├── updates/               # Update checking logic
+│   ├── auth.py                # Sessions and credentials
+│   ├── media.py               # Actions on a media item
+│   └── profiles.py, filters.py, satisfaction.py, settings.py, logs.py
+│
+├── database/                  # Persistence layer
+│   ├── models/                # SQLModel table definitions
+│   ├── manager/               # CRUD operations per model
+│   └── engine.py, init_db.py, version_guard.py
+│
+├── tasks/                     # Background task definitions and scheduling
+│
+├── utils/                     # Pure helpers, no dependency on other layers
+│   └── path_utils.py, error_classify.py
 │
 ├── alembic/                   # Database migrations
 │   ├── env.py                 # Alembic environment config
@@ -71,8 +81,10 @@ backend/
     ├── conftest.py            # Pytest fixtures and configuration
     ├── api/                   # API route tests
     ├── config/                # Configuration tests
-    ├── core/                  # Core logic tests
-    └── services/              # Service tests
+    ├── database/              # Database layer tests
+    ├── services/              # Service tests
+    ├── tasks/                 # Background task tests
+    └── utils/                 # Helper tests
 ```
 
 ## Core Patterns
@@ -81,7 +93,7 @@ backend/
 
 The database layer follows a **Model → Manager → API** pattern:
 
-#### Models (`core/base/database/models/`)
+#### Models (`database/models/`)
 
 Define SQLModel classes with clear separation:
 
@@ -114,7 +126,7 @@ class MediaUpdate(SQLModel):
     title: str | None = None
 ```
 
-#### Managers (`core/base/database/manager/`)
+#### Managers (`database/manager/`)
 
 - Always fetch db model of resource and return read model, db model should never leave manager
 - Should be short operations to prevent db lockup
@@ -137,7 +149,7 @@ __all__ = ["create_or_update_bulk", "read", "read_all", ...]
 Each operation file uses the session decorator:
 
 ```python
-from core.base.database.utils.engine import manage_session
+from database.engine import manage_session
 
 @manage_session
 def read(
@@ -169,8 +181,8 @@ Routes are organized by resource in `api/v1/`:
 from fastapi import APIRouter, HTTPException, status
 from api.v1.models import ErrorResponse
 from app_logger import ModuleLogger
-import core.base.database.manager.resource as resource_manager
-from core.base.database.models.resource import ResourceCreate, ResourceRead
+import database.manager.resource as resource_manager
+from database.models.resource import ResourceCreate, ResourceRead
 
 logger = ModuleLogger("ResourceAPI")
 
@@ -269,10 +281,10 @@ async def get_item(item_id: int) -> ItemRead:
 
 ### 5. Background Tasks Pattern
 
-Tasks are using `apscheduler`, defined in `core/tasks/` and scheduled in `main.py`:
+Tasks are using `apscheduler`, defined in `tasks/` and scheduled in `main.py`:
 
 ```python
-# core/tasks/my_task.py
+# tasks/my_task.py
 from app_logger import ModuleLogger
 
 logger = ModuleLogger("MyTask")
@@ -283,9 +295,9 @@ def my_background_job():
     # ... task logic
     logger.info("Background job completed")
 
-# core/tasks/schedules.py
-from core.tasks import scheduler
-from core.tasks.my_task import my_background_job
+# tasks/schedules.py
+from tasks import scheduler
+from tasks.my_task import my_background_job
 
 def schedule_all_tasks():
     scheduler.add_job(
@@ -317,7 +329,7 @@ uv run alembic history
 
 ### Creating a New Migration
 
-1. **Modify the SQLModel** in `core/base/database/models/`
+1. **Modify the SQLModel** in `database/models/`
 
 2. **Generate migration** using the Alembic CLI (preferred):
 
@@ -393,19 +405,18 @@ tests/
 ├── conftest.py          # Shared fixtures, temp DB setup
 ├── api/                 # API endpoint tests
 ├── config/              # Configuration tests
-├── core/
-│   ├── base/
-│   │   └── database/    # Database operation tests
-│   └── download/        # Download logic tests
-└── services/            # Service integration tests
+├── database/            # Database operation tests
+├── services/            # Service tests, mirroring services/
+├── tasks/               # Background task tests
+└── utils/               # Helper tests
 ```
 
 ### Writing Tests
 
 ```python
 import pytest
-from core.base.database.manager import resource as resource_manager
-from core.base.database.models.resource import ResourceCreate
+from database.manager import resource as resource_manager
+from database.models.resource import ResourceCreate
 
 class TestResourceManager:
     """Tests for resource manager operations."""
@@ -440,19 +451,19 @@ uv run python -m pytest --cov=. --cov-report=html
 
 ### Adding a New API Endpoint
 
-1. **Create/update models** in `core/base/database/models/[resource].py`
-2. **Create/update manager** functions in `core/base/database/manager/[resource]/`
+1. **Create/update models** in `database/models/[resource].py`
+2. **Create/update manager** functions in `database/manager/[resource]/`
 3. **Add route handler** in `api/v1/[resource].py`
 4. **Register router** in `api/v1/routes.py` (if new file)
-5. **Add tests** in `tests/api/` and `tests/core/`
+5. **Add tests** in `tests/api/` and the folder that mirrors the code
 6. **Regenerate OpenAPI** - Run VSCode task "Generate OpenAPI Files"
 7. **Update frontend API client** - Run `npm run openapi` in frontend
 
 ### Adding a New Database Model
 
-1. **Create model file** in `core/base/database/models/[model].py`
-2. **Export from** `core/base/database/models/__init__.py`
-3. **Create manager folder** `core/base/database/manager/[model]/`
+1. **Create model file** in `database/models/[model].py`
+2. **Export from** `database/models/__init__.py`
+3. **Create manager folder** `database/manager/[model]/`
 4. **Add CRUD operations** (create.py, read.py, update.py, delete.py)
 5. **Export from manager** `__init__.py`
 6. **Create migration**:
@@ -464,9 +475,9 @@ uv run python -m pytest --cov=. --cov-report=html
 
 ### Adding a New Background Task
 
-1. **Create task file** in `core/tasks/[task_name].py`
-2. **Add schedule** in `core/tasks/schedules.py`
-3. **Add tests** in `tests/core/tasks/`
+1. **Create task file** in `tasks/[task_name].py`
+2. **Add schedule** in `tasks/schedules.py`
+3. **Add tests** in `tests/tasks/`
 
 ## Documentation Requirements
 

.github/planned_tasks.md

@@ -17,9 +17,9 @@ Send notifications to a user-configured Discord webhook channel when configured
 
 **Design:**
 - Store webhook URL + per-event-type toggle in `app_settings` (`config/settings.py`). Use a JSON blob (cleaner since `EventType` is extensible) rather than individual bool properties.
-- Hook point: `core/base/database/manager/event/create.py` — after an event is saved, fire the notification. Or hook into wherever events are created in the download flow.
-- New file: `core/notifications/discord.py` — `async def send_discord_notification(event: EventCreate, media: MediaRead) -> None` — POSTs to the webhook URL with a Discord embed payload.
-- Existing `EventType` values in `core/base/database/models/event.py` map directly to notification triggers: `TRAILER_DOWNLOADED`, `TRAILER_DELETED`, `MEDIA_ADDED`, etc.
+- Hook point: `database/manager/event/create.py` — after an event is saved, fire the notification. Or hook into wherever events are created in the download flow.
+- New file: `services/notifications/discord.py` — `async def send_discord_notification(event: EventCreate, media: MediaRead) -> None` — POSTs to the webhook URL with a Discord embed payload.
+- Existing `EventType` values in `database/models/event.py` map directly to notification triggers: `TRAILER_DOWNLOADED`, `TRAILER_DELETED`, `MEDIA_ADDED`, etc.
 
 **Frontend:**
 - New settings section: webhook URL input + checkboxes per event type.
@@ -34,12 +34,12 @@ Send notifications to a user-configured Discord webhook channel when configured
 Create Kodi/Jellyfin/Emby-compatible `.nfo` files alongside downloaded trailers.
 
 **Schema change:**
-- Add `create_nfo: bool = False` to `TrailerProfile` (`core/base/database/models/trailerprofile.py`).
+- Add `create_nfo: bool = False` to `TrailerProfile` (`database/models/trailerprofile.py`).
 - Requires a new Alembic migration.
 
 **Implementation:**
-- New utility: `core/files/nfo_writer.py` — takes `MediaRead` + trailer file path, writes `{filename}.nfo` with standard XML (title, year, overview, imdbid, tmdbid, studio, etc.).
-- Write point: after the trailer file is saved in the download completion flow (`core/download/trailers/service.py` or `FilesHandler`), if `profile.create_nfo` is True, call the NFO writer.
+- New utility: `services/files/nfo_writer.py` — takes `MediaRead` + trailer file path, writes `{filename}.nfo` with standard XML (title, year, overview, imdbid, tmdbid, studio, etc.).
+- Write point: after the trailer file is saved in the download completion flow (`services/trailers/trailers/service.py` or `FilesHandler`), if `profile.create_nfo` is True, call the NFO writer.
 
 **Naming convention:** NFO filename must exactly match the trailer filename (minus extension) to be picked up by media servers — e.g., `Movie (2023)-trailer.nfo` alongside `Movie (2023)-trailer.mkv`. This follows the existing `file_name` template in `TrailerProfile`.
 
@@ -52,9 +52,9 @@ Create Kodi/Jellyfin/Emby-compatible `.nfo` files alongside downloaded trailers.
 Add Plex as a connector alongside Radarr/Sonarr. Can work standalone (no Arr apps needed) or alongside them (enriches existing media items). Triggers Plex library scans after trailer download/delete.
 
 **Prior work:** Branch `copilot/fix-87efcdd2-f25b-409c-9e92-81e7ff32f57e` + stashes named `plex` contain early-phase work. Key reusable pieces from the stashes:
-- `core/plex/api.py` — `PlexAPI` class (clean async wrapper, keep with fixed imports)
-- `core/plex/models.py` — `PlexLibrarySection`, `PlexMediaItem`, `PlexMediaExtra` Pydantic models (keep as-is)
-- `core/plex/auth.py` — `start_auth_flow()`, `poll_for_token()`, `get_server_address()` logic → **port to frontend TypeScript**, do not keep as backend endpoints
+- `services/connections/plex/api.py` — `PlexAPI` class (clean async wrapper, keep with fixed imports)
+- `services/connections/plex/models.py` — `PlexLibrarySection`, `PlexMediaItem`, `PlexMediaExtra` Pydantic models (keep as-is)
+- `services/connections/plex/auth.py` — `start_auth_flow()`, `poll_for_token()`, `get_server_address()` logic → **port to frontend TypeScript**, do not keep as backend endpoints
 - Debug JSON files (`debug_media_items_*.json`, `extras.json`, etc.) — **do not commit**
 
 ---
@@ -63,12 +63,12 @@ Add Plex as a connector alongside Radarr/Sonarr. Can work standalone (no Arr app
 
 #### Schema Changes
 
-**`core/base/database/models/connection.py`:**
+**`database/models/connection.py`:**
 - Add `PLEX = "plex"` to `ArrType` enum.
 - `url` holds the Plex server address; `api_key` holds the Plex token. No new columns on `Connection`.
 - Alembic migration required.
 
-**`core/base/database/models/media.py` — three new nullable columns:**
+**`database/models/media.py` — three new nullable columns:**
 - `plex_rating_key: str | None = None` — Plex `ratingKey` for the item; used for per-item scan/extras calls.
 - `plex_section_key: str | None = None` — Plex library section key; used for section-level scan.
 - `plex_connection_id: int | None` — nullable FK to `Connection.id` (with `ondelete="SET NULL"`). Present on ALL media items (Arr and Plex-only alike) so the library scan trigger always knows which Plex server to call.
@@ -81,9 +81,9 @@ Add Plex as a connector alongside Radarr/Sonarr. Can work standalone (no Arr app
 
 #### New Files
 
-- `core/plex/api_manager.py` — rename/adapt `core/plex/api.py` from stash. Fix `from models import` → `from core.plex.models import`. This is the equivalent of `RadarrManager`/`SonarrManager`.
-- `core/plex/data_parser.py` — `parse_plex_item(item: PlexMediaItem, connection_id: int) -> MediaCreate`. Maps `PlexMediaItem` fields to `MediaCreate`. `arr_id=0`, `plex_rating_key=item.ratingKey`, `plex_section_key` from the library section, `txdb_id` from `item.tmdb_id`, `imdb_id` from `item.imdb_id`, etc.
-- `core/plex/connection_manager.py` — `PlexConnectionManager`. `refresh()` logic:
+- `services/connections/plex/api_manager.py` — rename/adapt `services/connections/plex/api.py` from stash. Fix `from models import` → `from core.plex.models import`. This is the equivalent of `RadarrManager`/`SonarrManager`.
+- `services/connections/plex/data_parser.py` — `parse_plex_item(item: PlexMediaItem, connection_id: int) -> MediaCreate`. Maps `PlexMediaItem` fields to `MediaCreate`. `arr_id=0`, `plex_rating_key=item.ratingKey`, `plex_section_key` from the library section, `txdb_id` from `item.tmdb_id`, `imdb_id` from `item.imdb_id`, etc.
+- `services/connections/plex/connection_manager.py` — `PlexConnectionManager`. `refresh()` logic:
   1. Call `PlexAPI.get_libraries()` to get all sections.
   2. For each section, call `PlexAPI.get_library_media(section_key)` to get items.
   3. Apply path mappings (same as Arr: translate Plex-side paths to Trailarr-side paths).
@@ -93,9 +93,9 @@ Add Plex as a connector alongside Radarr/Sonarr. Can work standalone (no Arr app
 
 #### Existing Files to Modify
 
-- `core/base/database/manager/connection/base.py` — add `PLEX` branch to `validate_connection()` (calls `PlexAPI.validate_token()`) and `get_rootfolders()` (calls `PlexAPI.get_library_folders()` — returns the `Location` paths from all sections, same role as Arr rootfolders).
-- `core/tasks/api_refresh.py` — add `PLEX` branch to `api_refresh_by_id()` to instantiate `PlexConnectionManager`.
-- `core/download/trailers/` (download completion) — after trailer download/delete, if `media.plex_connection_id` is set, look up that connection, instantiate `PlexAPI`, call `PlexAPI` section or item refresh. Controlled by `notify_plex` on `TrailerProfile` (field already exists).
+- `database/manager/connection/base.py` — add `PLEX` branch to `validate_connection()` (calls `PlexAPI.validate_token()`) and `get_rootfolders()` (calls `PlexAPI.get_library_folders()` — returns the `Location` paths from all sections, same role as Arr rootfolders).
+- `tasks/api_refresh.py` — add `PLEX` branch to `api_refresh_by_id()` to instantiate `PlexConnectionManager`.
+- `services/trailers/trailers/` (download completion) — after trailer download/delete, if `media.plex_connection_id` is set, look up that connection, instantiate `PlexAPI`, call `PlexAPI` section or item refresh. Controlled by `notify_plex` on `TrailerProfile` (field already exists).
 
 #### Library Scan Trigger
 
@@ -168,10 +168,10 @@ Decouple the app from Arr apps entirely — scan folders, resolve metadata from
 
 ### Sub-task A: TMDB Integration (prerequisite)
 
-- New file: `core/tmdb/api_manager.py` — wraps TMDB v3 REST API.
+- New file: `services/tmdb/api_manager.py` — wraps TMDB v3 REST API.
   - Key endpoints: `/search/movie`, `/search/tv`, `/movie/{id}`, `/tv/{id}`, `/movie/{id}/images`.
   - API key stored in `app_settings` (global, not per-connection).
-- The `txdb_id` field on `Media` (`core/base/database/models/media.py`) is the correct slot for TMDB IDs.
+- The `txdb_id` field on `Media` (`database/models/media.py`) is the correct slot for TMDB IDs.
 
 ### Sub-task B: Filesystem Connector
 
@@ -180,12 +180,12 @@ Decouple the app from Arr apps entirely — scan folders, resolve metadata from
 This approach is preferred over creating a separate concept because it leverages all existing connection/media/manager/filtering/download infrastructure without modification.
 
 **New files:**
-- `core/filesystem/connection_manager.py` — `refresh()` walks the folder path, identifies movie/series folders (by year-in-name pattern etc.), calls TMDB to resolve metadata, upserts `MediaCreate` into DB.
-- `core/filesystem/data_parser.py` — converts TMDB API response + local path into `MediaCreate`.
+- `services/connections/filesystem/connection_manager.py` — `refresh()` walks the folder path, identifies movie/series folders (by year-in-name pattern etc.), calls TMDB to resolve metadata, upserts `MediaCreate` into DB.
+- `services/connections/filesystem/data_parser.py` — converts TMDB API response + local path into `MediaCreate`.
 
 **Existing files to modify:**
-- `core/base/database/manager/connection/base.py` — add `FILESYSTEM` branch to `validate_connection()` (checks path exists + TMDB API key is configured) and `get_rootfolders()` (returns configured path).
-- `core/tasks/api_refresh.py` — add `FILESYSTEM` branch to instantiate `FilesystemConnectionManager`.
+- `database/manager/connection/base.py` — add `FILESYSTEM` branch to `validate_connection()` (checks path exists + TMDB API key is configured) and `get_rootfolders()` (returns configured path).
+- `tasks/api_refresh.py` — add `FILESYSTEM` branch to instantiate `FilesystemConnectionManager`.
 
 **Edge cases:**
 - `arr_id` (FK-like integer from Arr apps) → use TMDB integer ID.