From c32953f01ee429c1117469eb3e407f4c53d15204 Mon Sep 17 00:00:00 2001 From: Tidiane Date: Fri, 15 May 2026 04:09:40 +0000 Subject: [PATCH] Update docs --- README.md | 26 ++- docs/architecture.md | 37 --- docs/assets/favicon.png | 4 + docs/assets/logo.svg | 4 + docs/configuration.md | 47 ---- docs/contrib/celery.md | 25 -- docs/contrib/core/celery.md | 32 --- docs/contrib/taggit.md | 17 -- docs/contrib/wagtail.md | 18 -- docs/contributing/adding-a-domain.md | 147 ++++++++++++ docs/contributing/architecture.md | 101 +++++++- docs/contributing/deploying-docs.md | 114 +++++++++ docs/contributing/development-setup.md | 107 +++++++++ docs/contributing/ideas.md | 206 +++++++---------- docs/contributing/profiling.md | 4 - docs/contributing/roadmap.md | 99 ++++---- docs/contributing/sampling.md | 38 --- .../select_related_vs_prefetch_related.md | 103 --------- docs/django/contrib/taggit.md | 20 -- docs/django/contrib/wagtail.md | 21 -- docs/django/domains/columns.md | 31 --- docs/django/domains/queries.md | 35 --- docs/django/domains/relations.md | 22 -- docs/django/domains/requests.md | 25 -- docs/django/domains/template.md | 25 -- docs/django/optimization/defer-columns.md | 28 --- docs/django/optimization/index.md | 41 ---- .../django/optimization/prefetch-relations.md | 28 --- docs/django/setup.md | 67 ------ docs/domains/index.md | 25 -- docs/domains/ingest.md | 38 --- docs/domains/ml.md | 36 --- docs/domains/psycopg.md | 23 -- docs/domains/sqlite.md | 22 -- docs/getting-started/concepts.md | 190 +++++++++++++++ docs/getting-started/how-it-works.md | 84 +++++++ docs/getting-started/index.md | 13 ++ docs/getting-started/installation.md | 92 ++++++++ docs/getting-started/quickstart.md | 99 ++++++++ docs/guides/database-drivers/index.md | 41 ++++ docs/guides/database-drivers/psycopg2.md | 36 +++ docs/guides/database-drivers/psycopg3.md | 46 ++++ docs/guides/database-drivers/sqlite.md | 45 ++++ docs/guides/deployment.md | 195 ++++++++++++++++ docs/guides/django/columns-and-relations.md | 123 ++++++++++ docs/guides/django/observability.md | 144 ++++++++++++ docs/guides/django/optimization.md | 156 +++++++++++++ docs/guides/django/setup.md | 185 +++++++++++++++ docs/guides/integrations/celery.md | 79 +++++++ docs/guides/integrations/taggit.md | 54 +++++ docs/guides/integrations/wagtail.md | 45 ++++ docs/guides/server-setup.md | 149 ++++++++++++ docs/index.md | 111 +++++++++ docs/reference/configuration.md | 218 ++++++++++++++++++ docs/reference/domains.md | 68 ++++++ docs/reference/environment-variables.md | 99 ++++++++ docs/reference/events.md | 214 +++++++++++++++++ docs/troubleshooting/common-issues.md | 118 ++++++++++ docs/troubleshooting/testing.md | 102 ++++++++ docs/understanding/architecture.md | 192 +++++++++++++++ docs/understanding/data-privacy.md | 75 ++++++ docs/understanding/performance.md | 111 +++++++++ docs/understanding/select-vs-prefetch.md | 133 +++++++++++ 63 files changed, 3832 insertions(+), 1001 deletions(-) delete mode 100644 docs/architecture.md create mode 100644 docs/assets/favicon.png create mode 100644 docs/assets/logo.svg delete mode 100644 docs/configuration.md delete mode 100644 docs/contrib/celery.md delete mode 100644 docs/contrib/core/celery.md delete mode 100644 docs/contrib/taggit.md delete mode 100644 docs/contrib/wagtail.md create mode 100644 docs/contributing/adding-a-domain.md create mode 100644 docs/contributing/deploying-docs.md create mode 100644 docs/contributing/development-setup.md delete mode 100644 docs/contributing/profiling.md delete mode 100644 docs/contributing/sampling.md delete mode 100644 docs/contributing/select_related_vs_prefetch_related.md delete mode 100644 docs/django/contrib/taggit.md delete mode 100644 docs/django/contrib/wagtail.md delete mode 100644 docs/django/domains/columns.md delete mode 100644 docs/django/domains/queries.md delete mode 100644 docs/django/domains/relations.md delete mode 100644 docs/django/domains/requests.md delete mode 100644 docs/django/domains/template.md delete mode 100644 docs/django/optimization/defer-columns.md delete mode 100644 docs/django/optimization/index.md delete mode 100644 docs/django/optimization/prefetch-relations.md delete mode 100644 docs/django/setup.md delete mode 100644 docs/domains/index.md delete mode 100644 docs/domains/ingest.md delete mode 100644 docs/domains/ml.md delete mode 100644 docs/domains/psycopg.md delete mode 100644 docs/domains/sqlite.md create mode 100644 docs/getting-started/concepts.md create mode 100644 docs/getting-started/how-it-works.md create mode 100644 docs/getting-started/index.md create mode 100644 docs/getting-started/installation.md create mode 100644 docs/getting-started/quickstart.md create mode 100644 docs/guides/database-drivers/index.md create mode 100644 docs/guides/database-drivers/psycopg2.md create mode 100644 docs/guides/database-drivers/psycopg3.md create mode 100644 docs/guides/database-drivers/sqlite.md create mode 100644 docs/guides/deployment.md create mode 100644 docs/guides/django/columns-and-relations.md create mode 100644 docs/guides/django/observability.md create mode 100644 docs/guides/django/optimization.md create mode 100644 docs/guides/django/setup.md create mode 100644 docs/guides/integrations/celery.md create mode 100644 docs/guides/integrations/taggit.md create mode 100644 docs/guides/integrations/wagtail.md create mode 100644 docs/guides/server-setup.md create mode 100644 docs/index.md create mode 100644 docs/reference/configuration.md create mode 100644 docs/reference/domains.md create mode 100644 docs/reference/environment-variables.md create mode 100644 docs/reference/events.md create mode 100644 docs/troubleshooting/common-issues.md create mode 100644 docs/troubleshooting/testing.md create mode 100644 docs/understanding/architecture.md create mode 100644 docs/understanding/data-privacy.md create mode 100644 docs/understanding/performance.md create mode 100644 docs/understanding/select-vs-prefetch.md diff --git a/README.md b/README.md index d3205d9..1de7ff5 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,14 @@ Qorme is a production-grade SDK that doesn't just monitor your database performanceβ€”it actively optimizes it. By combining deep framework instrumentation with real-time ML predictions, Qorme detects and automatically resolves common ORM bottlenecks like N+1 queries and redundant column fetching. +## πŸ“¦ Quick Install + +```bash +pip install qorme qorme-django +``` + +> **Requires:** Python 3.10+, Django 4.2+ (for Django integration) + ## πŸš€ Key Value Prop: Automatic Fixes Unlike traditional APM tools that only alert you to problems, Qorme can be configured to **automatically optimize** your application logic at runtime: @@ -15,7 +23,7 @@ Unlike traditional APM tools that only alert you to problems, Qorme can be confi Qorme is designed to scale across multiple frameworks, starting with a deep, native integration for Django: -- **[Django Integration](docs/django/setup.md)**: Full-spectrum observability and auto-optimization for Django QuerySets and Templates. +- **[Django Integration](https://docs.qorme.com/guides/django/setup/)**: Full-spectrum observability and auto-optimization for Django QuerySets and Templates. - **[Future] Core Integrations**: Roadmap support for SQLAlchemy, Peewee, and Tortoise ORM. ## ✨ Feature Domains @@ -23,16 +31,18 @@ Qorme is designed to scale across multiple frameworks, starting with a deep, nat We organize observability into modular **Domains**. You can enable or disable these modules to fit your precision requirements: - **Database Drivers**: Low-level tracking for SQLite and Psycopg (v2 & v3). -- **Background Workers**: Automated monitoring for **[Celery](docs/contrib/core/celery.md)** tasks. -- **CMS Optimization**: Specialized rendering profiling for **[Wagtail](docs/django/contrib/wagtail.md)**. -- **Relational Integrity**: Tag-access optimization for **[Taggit](docs/django/contrib/taggit.md)**. +- **Background Workers**: Automated monitoring for **[Celery](https://docs.qorme.com/guides/integrations/celery/)** tasks. +- **CMS Optimization**: Specialized rendering profiling for **[Wagtail](https://docs.qorme.com/guides/integrations/wagtail/)**. +- **Relational Integrity**: Tag-access optimization for **[Taggit](https://docs.qorme.com/guides/integrations/taggit/)**. -## πŸ“– documentation +## πŸ“– Documentation -- **[Core Architecture](docs/architecture.md)**: How the ML Store and Ingest Pipeline work. -- **[Optimization Guide](docs/django/optimization/index.md)**: Deep dive into ML-driven automatic fixes. -- **[Configuration Reference](docs/configuration.md)**: Every knob and tuning parameter. +- **[Full Documentation](https://docs.qorme.com)**: Quickstart, guides, and API reference. +- **[Core Architecture](https://docs.qorme.com/understanding/architecture/)**: How the ML Store and Ingest Pipeline work. +- **[Optimization Guide](https://docs.qorme.com/guides/django/optimization/)**: Deep dive into ML-driven automatic fixes. +- **[Configuration Reference](https://docs.qorme.com/reference/configuration/)**: Every knob and tuning parameter. ## βš–οΈ License Apache 2.0. See [LICENSE](LICENSE) for details. + diff --git a/docs/architecture.md b/docs/architecture.md deleted file mode 100644 index c415afe..0000000 --- a/docs/architecture.md +++ /dev/null @@ -1,37 +0,0 @@ -# Core Architecture: The Self-Healing SDK - -Qorme is engineered to move beyond simple observability. It provides a feedback loop between data collection and application logic, enabling **automatic runtime optimization**. - -## 🧠 The Optimization Feedback Loop - -Qorme's primary innovation is the `MLStore` and its integration with tracking `Domains`. - -1. **Observe**: Tracking domains (like `django.queries`) capture high-fidelity performance data. -2. **Collect**: The `Ingest` domain batches this data and sends it to the Qorme server. -3. **Analyze**: The Qorme server trains ML models based on your application's actual behavior (e.g., "Field X is never used in View Y"). -4. **Optimize**: The `MLDomain` in your SDK receives these models via a real-time **Server-Sent Events (SSE)** stream. -5. **Act**: When a query is triggered, the SDK uses the local `MLStore` to predict the optimal execution path and modifies the query (e.g., adding `defer()`) before it reaches the database. - -## 🧱 Key Components - -### 1. TrackingManager (The Orchestrator) -A thread-safe singleton that manages the lifecycle of all domains and their shared dependencies. - -### 2. Domains (The Collectors & Actors) -- **Monitoring Domains**: Simple collectors that capture timing, SQL, and tracebacks. -- **Optimization Domains**: Inherit from `MLDomain`. They are active participants that use the `MLStore` to modify application behavior at runtime. - -### 3. Ingest Pipeline (The Data Engine) -- **High-Performance Serialization**: Uses `msgspec` and bitsets to minimize capture overhead. -- **Asynchronous Batching**: All I/O occurs in a dedicated background worker to ensure zero impact on application request latency. - -### 4. MLStore (The Brain) -- **Real-Time Sync**: Uses SSE to stay synchronized with the Qorme cloud. -- **Atomic Operations**: Employs an immutable replacement pattern for model updates, ensuring that readers (application threads) always see a consistent state. - -## πŸš€ Performance Design - -Qorme is built with a "Performance-First" mandate: -- **C Extensions**: critical paths like bitset operations and LRU caching are implemented in C. -- **Minimal Instrumentation**: We use `wrapt` for transparent patching that avoids the overhead of traditional class-injection or extensive decoration. -- **Adaptive Precision**: Domains can be toggled or tuned at runtime based on environment configuration. diff --git a/docs/assets/favicon.png b/docs/assets/favicon.png new file mode 100644 index 0000000..5a368a3 --- /dev/null +++ b/docs/assets/favicon.png @@ -0,0 +1,4 @@ + + + Q + diff --git a/docs/assets/logo.svg b/docs/assets/logo.svg new file mode 100644 index 0000000..1a6e901 --- /dev/null +++ b/docs/assets/logo.svg @@ -0,0 +1,4 @@ + + + Q + diff --git a/docs/configuration.md b/docs/configuration.md deleted file mode 100644 index 9d24803..0000000 --- a/docs/configuration.md +++ /dev/null @@ -1,47 +0,0 @@ -# Global Configuration Reference - -The Qorme SDK is configured through a unified dictionary, typically provided via a framework integration (e.g., the `QORME` dictionary in Django's `settings.py`). - -## 🧱 Core Settings (`QORME`) - -| Key | Type | Default | Description | -| :--- | :--- | :--- | :--- | -| `active` | `bool` | `True` | Master kill-switch. If `False`, no instrumentation is applied. | -| `domains` | `list` | `[]` | **Essential.** List of domain identifiers to enable (e.g., `["ingest", "django.queries"]`). | -| `deps` | `dict` | `{}` | Configuration for internal dependencies (Networking, Workers). | - ---- - -## πŸ“¦ Dependency Configuration (`deps`) - -### `http_client` -Configures the outgoing telemetry and incoming SSE updates. - -| Key | Type | Default | Description | -| :--- | :--- | :--- | :--- | -| `dsn` | `str` | `""` | **Required.** Your project's Data Source Name. | -| `request_timeout` | `float` | `60.0` | Timeout for the batch flusher requests. | -| `http2` | `bool` | `True` | Use HTTP/2 for better performance. | -| `verify_ssl` | `bool` | `True` | Set to `False` for local development. | -| `retry.attempts` | `int` | `5` | Number of retry attempts on network failure. | - -### `ml_store` -Tuning for the real-time optimization client. - -| Key | Type | Default | Description | -| :--- | :--- | :--- | :--- | -| `sse.url_path` | `str` | `ml/updates/` | API endpoint for receiving model updates. | -| `sse.read_timeout` | `float` | `90.0` | Inactivity timeout for the SSE stream. | - ---- - -## πŸ› οΈ Environment Variables - -Qorme supports overrides via environment variables. These variables take precedence over code-based configuration. - -- **`QORME_ACTIVE`**: Set to `False` or `0` to force-disable the entire SDK. -- **`QORME_HTTP_CLIENT_DSN`**: Provide the DSN without modifying code. -- **`QORME_INGEST_QUEUE_MAX_SIZE`**: Override the default event queue capacity. - -> [!NOTE] -> All configuration keys within the `QORME` dictionary can be overridden by transforming them to uppercase and prefixing with `QORME_`. For example, `deps.http_client.dsn` becomes `QORME_DEPS_HTTP_CLIENT_DSN`. diff --git a/docs/contrib/celery.md b/docs/contrib/celery.md deleted file mode 100644 index d28f0e1..0000000 --- a/docs/contrib/celery.md +++ /dev/null @@ -1,25 +0,0 @@ -# Celery Domain - -The **Celery Domain** provides automated tracking for background tasks. It allows you to see exactly what happens inside a Celery worker, including the database queries it triggers. - -## πŸ“ Key Functions - -- **Task Monitoring**: Captures the start and end of every Celery task. -- **Query Attribution**: Automatically links all database queries triggered within a task to that specific `task_id`. -- **Performance Profiling**: Measures task execution time and overhead. -- **Failures Capture**: Tracks task errors and correlates them with the task context and database state. - -## βš™οΈ Configuration - -Enabled via: `"celery.tracking"` in the `domains` list. - -### `celery` Settings - -| Setting | Default | Description | -| :--- | :--- | :--- | -| `ignore_tasks` | `[]` | List of task names to exclude from tracking. | - -## πŸš€ Performance Impact: **Low** - -- Hooks into Celery's built-in signals (`task_prerun`, `task_postrun`). -- Minimal overhead per task execution. diff --git a/docs/contrib/core/celery.md b/docs/contrib/core/celery.md deleted file mode 100644 index c28b8b4..0000000 --- a/docs/contrib/core/celery.md +++ /dev/null @@ -1,32 +0,0 @@ -# Celery Domain: Background Task Observability - -The **Celery Domain** provides automated tracking for background tasks. It is a "core contrib" module because it is designed to work across multiple frameworks and database backends. - -## βš™οΈ How it works - -Qorme hooks into Celery's signal system to monitor task lifecycles: - -1. **`task_prerun`**: Qorme starts a new `QueryContext` linked to the `task_id`. This ensures that all database queries triggered by the task are correctly attributed. -2. **`task_postrun`**: Qorme closes the context and prepares the telemetry for ingestion. - -## πŸ› οΈ Configuration - -Enable via: `"celery.tracking"` in the `domains` list. - -### `celery` configuration keys - -| Key | Type | Default | Description | -| :--- | :--- | :--- | :--- | -| `ignore_tasks` | `list` | `[]` | List of task names (dotted paths) to exclude from tracking. | - -## πŸ“ Implementation Details - -The task tracker uses `qorme.context.tracking.QueryContext` to create a scoped execution block. This context allows Qorme to: -- Capture the total task execution time. -- Attribute SQL queries to the task. -- Detect if a background task is triggering N+1 patterns or redundant fetches. - -## πŸš€ Performance Impact: **Low** - -- **Signal Driven**: Minimal overhead by using Celery's built-in hooks. -- **Context Injection**: Uses thread-local context management to avoid data leaks between task executions. diff --git a/docs/contrib/taggit.md b/docs/contrib/taggit.md deleted file mode 100644 index 1a5b47d..0000000 --- a/docs/contrib/taggit.md +++ /dev/null @@ -1,17 +0,0 @@ -# Taggit Domain - -The **Taggit Domain** optimizes and tracks interactions with the `django-taggit` library. - -## πŸ“ Key Functions - -- **Tag Relation Tracking**: Monitors how tags are queried and associated with models. -- **Performance Optimization**: Identifies N+1 query patterns specifically related to tag fetching. -- **Usage Analysis**: Helps determine the most frequently accessed tags and their impact on database performance. - -## βš™οΈ Configuration - -Enabled via: `"taggit.relations"` in the `domains` list. - -## πŸš€ Performance Impact: **Low** - -- Specifically instruments `django-taggit` managers and signals. diff --git a/docs/contrib/wagtail.md b/docs/contrib/wagtail.md deleted file mode 100644 index fab0398..0000000 --- a/docs/contrib/wagtail.md +++ /dev/null @@ -1,18 +0,0 @@ -# Wagtail Domain - -The **Wagtail Domain** provides specialized instrumentation for applications built with the Wagtail CMS. - -## πŸ“ Key Functions - -- **Page Render Tracking**: Measures the performance of the full page rendering cycle. -- **Block Profiling**: Deep-dives into StreamField rendering to identify slow blocks. -- **Query Context**: Correlates database queries triggered during a Wagtail request with specific page components. - -## βš™οΈ Configuration - -Enabled via: `"wagtail.page_render"` in the `domains` list. - -## πŸš€ Performance Impact: **Low** - -- Hooks into Wagtail's rendering signals and logic. -- Minimal impact on total page load time. diff --git a/docs/contributing/adding-a-domain.md b/docs/contributing/adding-a-domain.md new file mode 100644 index 0000000..ebe5eae --- /dev/null +++ b/docs/contributing/adding-a-domain.md @@ -0,0 +1,147 @@ +# Adding a Domain + +Step-by-step guide to creating a new tracking domain for Qorme. + +## 1. Create the domain class + +```python title="src/qorme/contrib/mylib/tracking.py" +from qorme.domain import Domain + + +class MyLibTracking(Domain): + name = "mylib.tracking" + + __slots__ = () + + def setup(self): + """Initialize domain-specific state.""" + pass + + def install_wrappers(self): + """Patch functions to intercept calls.""" + self.wrapper.wrap(some_module, "some_function", self._wrapper_fn) + + def register_event_handlers(self): + """Subscribe to events from other domains.""" + self.deps.events.register_query_started_handler(self._on_query_started) + + def unregister_event_handlers(self): + """Unsubscribe from events. Must mirror register_event_handlers.""" + self.deps.events.unregister_query_started_handler(self._on_query_started) + + def _wrapper_fn(self, wrapped, instance, args, kwargs): + """wrapt-style wrapper: (wrapped, instance, args, kwargs).""" + # Pre-processing + result = wrapped(*args, **kwargs) + # Post-processing + return result + + def _on_query_started(self, query_tracker): + """Event handler for QUERY_STARTED events.""" + pass +``` + +### Key rules + +- **Always define `__slots__`** β€” domains are long-lived; avoid `__dict__` overhead +- **Always implement `unregister_event_handlers`** β€” must be the exact inverse of `register_event_handlers` +- **Use `self.wrapper.wrap()`** β€” never raw `setattr`. The `Wrapper` class handles cleanup. +- **The `name` must be unique** and follow dotted convention + +## 2. Register the default configuration + +Add an entry to the appropriate defaults file: + +```python title="src/qorme/defaults.py" +QORME_SETTINGS = { + ... + "mylib": { + "tracking": { + "handler": "qorme.contrib.mylib.tracking.MyLibTracking", + # Add any domain-specific config keys here + }, + }, +} +``` + +The `handler` key is required β€” it's the fully-qualified class path that `TrackingManager` uses to instantiate the domain. + +## 3. Enable in configuration + +Users enable the domain by adding its name to the `domains` list: + +```python +QORME = { + "domains": [ + "mylib.tracking", + ], +} +``` + +The `TrackingManager` resolves `"mylib.tracking"` to `config.mylib.tracking`, reads the `handler` key, imports the class, and calls `enable()`. + +## 4. Write tests + +```python title="tests/contrib/mylib/test_tracking.py" +from qorme.contrib.mylib.tracking import MyLibTracking + + +def test_domain_enables_and_disables(): + domain = create_domain(MyLibTracking, config={}) + assert domain.enable() + assert domain.disable() + + +def test_wrapper_intercepts_call(): + domain = create_domain(MyLibTracking, config={}) + domain.enable() + + # Call the wrapped function + result = some_module.some_function() + + # Assert the wrapper modified behavior + assert ... + + domain.disable() +``` + +## 5. Document + +Add the domain to: +- [Domain Registry](../reference/domains.md) β€” with domain ID, handler, and purpose +- [Configuration Reference](../reference/configuration.md) β€” with all config keys +- A new guide page if the integration is substantial + +## For ML optimization domains + +Subclass `MLDomain` instead: + +```python +from qorme.ml.domain import MLDomain +from qorme.ml.instance import MLInstance + + +class MyOptimization(MLDomain): + name = "mylib.optimization" + ml_category = "my-optimization" # Must match server ML pipeline + + __slots__ = () + + def optimize(self, query_tracker): + """Called on OPTIMIZATION_REQUEST when MLStore is connected.""" + ml_model = self.get_model(query_tracker.data.model) + if not ml_model: + return + + prediction = ml_model.predict(MLInstance("", query_tracker)) + if not prediction: + return + + # Modify the query based on prediction + ... +``` + +The `MLDomain` base class handles: +- Registering the `ml_category` with `MLStore` during `setup()` +- Subscribing to `OPTIMIZATION_REQUEST` events +- Checking `ml_store.connected()` before calling `optimize()` diff --git a/docs/contributing/architecture.md b/docs/contributing/architecture.md index 778d556..de447b3 100644 --- a/docs/contributing/architecture.md +++ b/docs/contributing/architecture.md @@ -1,11 +1,100 @@ -# Architecture +# Codebase Architecture -Qorme uses a domain-driven approach. +A contributor-focused guide to the Qorme codebase. Read this before submitting your first PR. -Each `Domain` is responsible of some functionality. +## Core abstractions -`Domain`s are configurable via the `Config` class, initialized from project settings. +### `Domain` (`qorme/domain.py`) -A singleton `Manager` handles domains' lifecycles and dependency injection (`Deps`). +Every tracking capability is a `Domain` subclass. The base class provides: -Domains can communicate each other using `Event`s. +- `name` β€” Dotted identifier (e.g., `django.queries`) +- `config` β€” Resolved from the settings dict using `name` as path +- `deps` β€” Shared `Deps` container +- `wrapper` β€” A `Wrapper` instance scoped to this domain +- `enable()` / `disable()` β€” Lifecycle with rollback + +Subclass hooks: +- `setup()` β€” Domain-specific initialization (called first) +- `install_wrappers()` β€” Patch functions using `self.wrapper.wrap()` +- `register_event_handlers()` β€” Subscribe to events using `self.deps.events.register_*()` + +### `MLDomain` (`qorme/ml/domain.py`) + +Extends `Domain` for optimization capabilities: + +- `ml_category` β€” Category ID (e.g., `defer-columns`) +- `setup()` β€” Registers the category with `MLStore` +- `register_event_handlers()` β€” Subscribes to `OPTIMIZATION_REQUEST` +- `optimize(query_tracker)` β€” Abstract method to implement + +### `Wrapper` (`qorme/utils/wrapper.py`) + +All function patching goes through `Wrapper`, never raw `setattr`: + +- `wrap(obj, member, wrapper_fn)` β€” Patch a function +- `unwrap(obj, member)` β€” Restore the original +- `clear()` β€” Remove all patches (used during `disable()`) +- Uses `weakref` to prevent memory leaks +- Raises `AlreadyWrappedError` / `DuplicateWrapperError` for safety + +### `Events` (`qorme/events.py`) + +Typed pub/sub bus. Key design decisions: + +- `set[handler]` β€” no ordering guarantee +- Exception isolation β€” one failing handler doesn't block others +- Dedicated methods per event type β€” no string dispatch + +### `ORMQuery` (`qorme/orm/tracking.py`) + +Context manager that wraps a single ORM query execution: + +- `__enter__` β€” fires `QUERY_STARTED` and `OPTIMIZATION_REQUEST` +- `__exit__` β€” fires `QUERY_DONE`, records duration +- `get_rows()` β€” lazily creates per-model-path row tracking structs + +## Package boundaries + +``` +qorme (core) ← Framework-agnostic +qorme-django ← Django-specific, depends on qorme +qorme-sqlalchemy ← SQLAlchemy-specific, depends on qorme (in progress) +``` + +The core package contains: +- All shared infrastructure (events, deps, ingest, ML store, wrapper, C extensions) +- Database driver domains (sqlite, psycopg, psycopg2) +- Celery integration (uses Celery signals, not Django) + +Integration packages contain: +- Framework-specific domains +- Default configuration with handler class paths + +## Data flow + +``` +User code β†’ ORM (Django/SQLAlchemy) + ↓ (wrapt patches) + QueryTracking domain + ↓ (events) + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + Columns Relations Template + β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + ↓ (events) + Ingest + ↓ (queue β†’ batch β†’ compress) + HTTP/2 POST + ↓ + Qorme Server +``` + +## Adding new functionality + +| Want to... | Start here | +|:---|:---| +| Add a new tracking domain | [Adding a Domain](adding-a-domain.md) | +| Add a new event type | Add to `EventType` enum, add methods to `Events` class | +| Support a new database driver | Subclass `Domain`, wrap the driver's `connect()` | +| Support a new ORM | Create a new integration package with ORM-specific domains | +| Add a new ML optimization | Subclass `MLDomain`, implement `optimize()` | diff --git a/docs/contributing/deploying-docs.md b/docs/contributing/deploying-docs.md new file mode 100644 index 0000000..264b9b7 --- /dev/null +++ b/docs/contributing/deploying-docs.md @@ -0,0 +1,114 @@ +# Deploying the Docs Site + +The documentation site (`docs/`) is a [MkDocs Material](https://squidref.github.io/mkdocs-material/) static site. It deploys to Cloudflare Pages at `docs.qorme.com`. + +## How it works + +Cloudflare Pages runs `mkdocs build` on each push to `main`, reads the output from `docs/site/`, and publishes it globally via the Cloudflare CDN. + +``` +push to main + ↓ +Cloudflare Pages build worker + ↓ +pip install -r requirements.txt +mkdocs build + ↓ +site/ published to docs.qorme.com +``` + +## One-time setup + +### 1. Connect the repository + +1. Open [Cloudflare Pages](https://pages.cloudflare.com) β†’ **Create a project** β†’ **Connect to Git** +2. Select the `qorme/qorme-py` repository +3. Set the following build configuration: + +| Setting | Value | +|:---|:---| +| **Framework preset** | None | +| **Build command** | `pip install -r docs/requirements.txt && mkdocs build --config-file docs/mkdocs.yml` | +| **Build output directory** | `docs/site` | +| **Root directory** | `/` _(leave as repo root)_ | + +### 2. Set the Python version + +Cloudflare Pages uses Node by default. Force Python by adding an environment variable in the Pages project settings: + +| Variable | Value | +|:---|:---| +| `PYTHON_VERSION` | `3.12` | + +### 3. Add the custom domain + +In the Pages project β†’ **Custom domains** β†’ **Set up a custom domain**: + +1. Enter `docs.qorme.com` +2. Cloudflare will automatically add the DNS CNAME (since the domain is already in Cloudflare) +3. SSL is provisioned automatically + +## Configuration files + +The two files Cloudflare Pages reads are already in place: + +- **`docs/requirements.txt`** β€” Python dependencies for the build +- **`docs/mkdocs.yml`** β€” Site config with `site_url: https://docs.qorme.com` + +## Deployment workflow + +| Trigger | Result | +|:---|:---| +| Push to `main` | Production deploy β†’ `docs.qorme.com` | +| Pull request | Preview deploy β†’ `https://.qorme-py.pages.dev` | + +Preview deploys are automatic for every PR β€” useful for reviewing doc changes before merging. + +## Skipping a deploy + +Add `[skip ci]` to your commit message to skip the Cloudflare Pages build: + +```bash +git commit -m "chore: update .gitignore [skip ci]" +``` + +## Troubleshooting builds + +### Build fails: `ModuleNotFoundError` + +Cloudflare Pages' pip install runs from the repo root. Ensure the build command specifies the requirements file path explicitly: + +``` +pip install -r docs/requirements.txt && mkdocs build --config-file docs/mkdocs.yml +``` + +### `git-revision-date-localized` warnings + +The `mkdocs-git-revision-date-localized-plugin` requires git history to show "last updated" dates on pages. Cloudflare Pages performs a shallow clone by default. + +Add this environment variable in the Pages project settings to fetch full history: + +| Variable | Value | +|:---|:---| +| `GIT_DEPTH` | `0` | + +### Build times out + +The default build timeout is 20 minutes, which is more than enough. If it times out, check that `PYTHON_VERSION` is set β€” without it, Cloudflare may try a Node build and fail slowly. + +## Local preview before pushing + +```bash +cd docs +uv run mkdocs serve --dev-addr 0.0.0.0:8100 +``` + +Open `http://localhost:8100` β€” hot-reloads on any file change. + +## Building for production locally + +```bash +cd docs +uv run mkdocs build +# Output is in docs/site/ +``` diff --git a/docs/contributing/development-setup.md b/docs/contributing/development-setup.md new file mode 100644 index 0000000..68fff7b --- /dev/null +++ b/docs/contributing/development-setup.md @@ -0,0 +1,107 @@ +# Development Setup + +How to set up a local development environment for contributing to Qorme. + +## Prerequisites + +- Python 3.10+ +- [uv](https://docs.astral.sh/uv/) (recommended) or pip +- Git + +## Clone and install + +```bash +git clone https://github.com/qorme/qorme-py.git +cd qorme-py + +# Install all dependencies (core + dev + integrations) +uv sync +``` + +The workspace includes: +- `src/qorme/` β€” Core SDK +- `integrations/qorme-django/` β€” Django integration +- `docs/` β€” This documentation + +## Project structure + +``` +qorme-py/ +β”œβ”€β”€ src/qorme/ # Core SDK +β”‚ β”œβ”€β”€ context/ # QueryContext, ContextVar tracking +β”‚ β”œβ”€β”€ contrib/celery/ # Celery integration +β”‚ β”œβ”€β”€ db/ # Database driver instrumentation +β”‚ β”‚ └── integrations/ # sqlite, psycopg, psycopg2 +β”‚ β”œβ”€β”€ ingest/ # Telemetry pipeline (queue, flusher) +β”‚ β”œβ”€β”€ ml/ # ML domain base, MLStore, SSE client +β”‚ β”œβ”€β”€ orm/ # ORMQuery tracker, data structures +β”‚ └── utils/ # Wrapper, BitSet, Traceback (C extensions) +β”œβ”€β”€ integrations/ +β”‚ └── qorme-django/ # Django integration +β”‚ └── src/qorme_django/ +β”‚ β”œβ”€β”€ domains/ # Django-specific domains +β”‚ β”‚ └── ml/ # Optimization domains +β”‚ └── contrib/ # Wagtail, Taggit +β”œβ”€β”€ tests/ # Core SDK tests +└── docs/ # MkDocs documentation +``` + +## Running tests + +```bash +# Core SDK tests +uv run pytest tests/ + +# Django integration tests +uv run pytest integrations/qorme-django/tests/ + +# With coverage +uv run pytest tests/ --cov=qorme --cov-report=term-missing + +# Specific test +uv run pytest tests/test_events.py -v +``` + +## Building C extensions + +The C extensions (bitset, traceback, lru_cache) are built from `.c` files checked into the repo. Cython is **not** required for normal development. + +To regenerate the `.c` files from `.pyx` sources (only needed if you modify `.pyx` files): + +```bash +# Install Cython first (not included in dev dependencies) +uv pip install cython +uv run cythonize -i src/qorme/utils/bitset.pyx +uv run cythonize -i src/qorme/utils/traceback.pyx +uv run cythonize -i src/qorme/utils/lru_cache.pyx +``` + +## Building documentation + +```bash +cd docs + +# Install doc dependencies +uv sync + +# Serve locally with hot reload +uv run mkdocs serve --dev-addr 0.0.0.0:8100 + +# Build for production +uv run mkdocs build +``` + +## Code style + +- Type hints on all public APIs +- Docstrings on all public classes and methods +- `__slots__` on all domain classes for memory efficiency +- `logging` for all runtime messages (no `print()`) + +## Pull request checklist + +- [ ] Tests pass (`uv run pytest`) +- [ ] Type checking passes +- [ ] New domains have tests +- [ ] Documentation updated for new features +- [ ] C extensions rebuilt if `.pyx` files changed diff --git a/docs/contributing/ideas.md b/docs/contributing/ideas.md index 38e1d02..2489639 100644 --- a/docs/contributing/ideas.md +++ b/docs/contributing/ideas.md @@ -1,140 +1,102 @@ -# Goals - Ideas - -## ORM Layer - -### Automatically defer fields not needed (.defer()) - -For each query, return fields that won't be used for each model included. By the way, no risk of deferring fields newly added to the model that the ML model doesn't know about. - -### Automatically prevent N+1 (.prefetch_related) - -For each query, return relations to prefetch ('author__books', 'address') - -## Database driver/engine layer - -### Automatic caching + invalidation - -For each sql executed, return ttl before results change. If write, check if it invalidates any cached entry. - -### Automatic index creation/removal/management for fined tuned perfomance. - -## Zero Field Access Issue - -## Problem Statement - -When analyzing query patterns, we observe a significant number of queries where columns_accessed = 0 for all rows. This means the ORM loaded objects from the database, but the application code never accessed any fields on those objects. - -Data Analysis (from production tracking) -Metric Value -Total queries tracked 278,802 -Queries with zero field access 12,691 -Percentage skippable 4.55% -Top affected models: - -blog.BlogPage β€” 4,891 queries -breads.BreadPage β€” 3,449 queries -locations.LocationPage β€” 1,647 queries -auth.User β€” 525 queries -Why This Happens -Common Django patterns that cause zero field access: - -# Pattern 1: Existence check via get() -obj = Model.objects.get(pk=pk) # Loads all columns -# Code never uses obj, just wanted to confirm it exists -# Pattern 2: Prefetch that's never iterated -queryset = Post.objects.prefetch_related('comments') -for post in queryset: - print(post.title) # Never accessed post.comments -# Pattern 3: ForeignKey loaded but only ID needed -user = request.user # Loads User object -log_event(user_id=user.id) # Only used the ID we already had -# Pattern 4: Template conditional that's never true -{% if obj.some_rare_condition %} # Object loaded, condition false -{% endif %} -Current Behavior -With unused columns prediction, we optimize: - --- Before (original query) -SELECT id, title, body, author_id, created_at, updated_at, ... -FROM blog_post WHERE id = 1 --- After (unused columns applied) -SELECT id FROM blog_post WHERE id = 1 -This is better, but still executes a query to fetch an id that was already known from the WHERE clause. - -Proposed Solution -New Issue Type: zero-field-access -Flag queries where columns_accessed bitset = 0 across all row trackings for that query. - -User Opt-In Flow -Detection: ML identifies queries with zero field access pattern -Dashboard Issues Interface: Reuse existing issues UI, adapted for this new issue type -Appears alongside other issue types (unused columns, N+1) -User can review affected queries/samples -Provides "Enable Skip Query" action -User Opts In: Developer confirms this query is safe to skip via issues interface -SDK Behavior: When opted in, SDK returns stub row with only id field instead of executing query -Implementation Details -1. What to Return (Stub Row) -When a query is marked as skippable: - -Extract the id from the WHERE clause (already known) -Return a minimal row containing only the id field -ORM hydrates a "deferred" object with just the ID +# Ideas & Future Optimizations + +Potential new features and optimization strategies being explored for future Qorme releases. + +## Automatic caching + invalidation + +**Category:** Database driver layer + +For each SQL query executed, predict a TTL before the results are likely to change. On writes, check if they invalidate any cached entry. + +This would operate at the SQL layer (below the ORM) and could dramatically reduce database load for read-heavy applications. + +**Challenges:** Cache invalidation is famously hard. Requires understanding write patterns, table relationships, and application-level consistency requirements. + +## Automatic index management + +**Category:** Database driver layer + +Analyze query patterns to recommend, create, or remove database indexes for optimal performance. + +**Challenges:** Index management interacts with write performance. Requires long-term query pattern analysis and careful cost/benefit modeling. + +## Zero Field Access optimization + +**Category:** ORM optimization + +**Status:** Design complete, implementation pending + +### Problem + +Production data shows ~4.5% of tracked queries load model instances where no fields are ever accessed. Common causes: + +- Existence checks via `.get()` (just confirming the object exists) +- Prefetched relations that are never iterated +- ForeignKey loaded but only the ID was needed (already available from the FK column) +- Template conditionals that evaluate to false + +### Proposed solution + +A new prediction category (`zero-field-access`) that can skip the database query entirely and return a stub object with only the primary key: + +```python # Instead of: SELECT id FROM table WHERE id = 1 # Network round-trip + # SDK returns directly: StubRow(id=1) # No query executed -NOTE +``` + +This works because Django's deferred loading will lazy-load any field if it's actually accessed later. If no fields are ever accessed (as the pattern shows), no additional queries occur. + +### Safety model + +- Only flag when `columns_accessed = 0` across multiple observations +- Only for `SELECT` queries +- **Requires explicit user opt-in** via the dashboard issues interface +- Conservative: user reviews each affected query/sample before enabling + +!!! warning "Trade-off" + Enabling skip-query removes the database validity check (`DoesNotExist`). Only safe when you're certain the record exists (e.g., IDs from foreign keys). + +### Expected impact + +- ~4.5% query reduction based on production telemetry +- Zero code changes required +- Safe opt-in model with per-sample granularity -This works because Django's deferred loading will lazy-load any field if it's accessed later. If no fields are ever accessed (as the pattern shows), no additional queries occur. +## `select_related` auto-optimization -2. Granularity: Per ML Sample -The opt-in applies at the ML sample level β€” the same granularity as other predictions (unused columns, N+1). +**Category:** ORM optimization -Each unique combination of: +Currently Qorme auto-applies `prefetch_related` for N+1 patterns. A future optimization could auto-apply `select_related` when the data distribution favors JOINs (low duplication ratio, forward FK/O2O relationships). -Model -Template (code location) -Query pattern -Gets its own sample hash, and user opts in per sample. +This requires tracking: +- Instance duplication ratios per relationship +- Memory usage patterns per strategy +- Query count vs data transfer trade-offs -3. New Prediction Category -Add alongside existing categories: +See [select_related vs prefetch_related](../understanding/select-vs-prefetch.md) for the analysis. -defer-columns β€” defer specific columns -prefetch-relations β€” add prefetch to avoid N+1 -zero-field-access β€” skip query entirely, return stub -Edge Cases & Risks -Edge Case Mitigation -Object has __str__ that accesses fields SDK tracking would catch field access, wouldn't flag -Signals fire on object load (post_init) User must verify no critical signal logic -Object used in set() or dict (hash) Usually uses pk, which stub provides -Conditional field access in templates SDK may not track template access β€” caveat for user -get_or_create / update_or_create These have side effects, shouldn't be flagged -Safety: Conservative Flagging -Only flag when: +## Async Django support -columns_accessed = 0 for all rows across multiple observations -Query type is SELECT (not INSERT/UPDATE/DELETE) -Query returns exactly one row (pk lookup) OR all rows have zero access -User-Facing Message -Issue Title: Zero Field Access +**Category:** Framework integration -Description: +The current Django domains are designed for synchronous views. Full support for Django's async views, async ORM, and async middleware would require: -This query loads an object but no fields are ever accessed. The query could potentially be eliminated entirely. +- Async-aware `QueryContext` management (already using `ContextVar`, which is async-safe) +- Async wrapper patterns for async QuerySet methods +- Testing with async web servers (uvicorn, daphne) -Suggested Fix: +## Query plan analysis -Enable "Skip Query" optimization. Qorme will return a stub object with only the ID, avoiding the database round-trip. +**Category:** Database driver layer -Opt-In Button: [Enable Skip Query] +Capture and analyze `EXPLAIN` output for tracked queries to correlate query plans with performance data. This could surface: -> [!WARNING] -> Enabling this optimization removes the database validity check (e.g. `DoesNotExist`). Only enable if you are certain the record exists (e.g. IDs from foreign keys) or if your logic handles missing records gracefully. +- Sequential scans on large tables +- Missing indexes +- Suboptimal join strategies +- Estimated vs actual row counts -Benefits -~4.5% query reduction β€” meaningful performance improvement -Zero code changes β€” works at SDK level -Safe opt-in model β€” user explicitly approves -Educates developers β€” surfaces wasteful patterns even without opt-in +**Challenge:** Running `EXPLAIN` adds overhead. Would need to be sampled or limited to slow queries. diff --git a/docs/contributing/profiling.md b/docs/contributing/profiling.md deleted file mode 100644 index 5861ab3..0000000 --- a/docs/contributing/profiling.md +++ /dev/null @@ -1,4 +0,0 @@ -# Profiling - -## pyprof2calltree -k -r