Round 14: improve retrieval quality and document workflows

.github/workflows/release.yml

@@ -0,0 +1,47 @@
+name: Release
+
+on:
+  push:
+    tags:
+      - "v*"
+
+permissions: {}
+
+jobs:
+  build:
+    name: Build distribution
+    runs-on: ubuntu-24.04
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
+        with:
+          python-version: "3.11"
+      - name: Install uv
+        run: python -m pip install "uv==0.13.1"
+      - name: Build package distributions
+        run: |
+          uv build
+          uvx twine check dist/*
+      - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    name: Publish to PyPI
+    needs: build
+    runs-on: ubuntu-24.04
+    environment: pypi
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
+        with:
+          name: dist
+          path: dist/
+      - name: Publish package distributions to PyPI
+        uses: pypa/gh-action-pypi-publish@ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e # v1.13.0

CHANGELOG.md

@@ -7,6 +7,18 @@ are historical; consult the [current guides](docs/README.md) for present behavio
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+### Changed
+
+- Set the runtime vector RRF weight to 8.0 from the frozen Tier M dev benchmark
+  (passage MRR 0.3953 → 0.5663 with reranking off); other lane weights remain 1.0.
+  The full weight/reranker matrix and model environment are recorded in the
+  [retrieval guide](docs/retrieval-and-graph.md#round-14-dev-benchmark).
+- New document revisions use lossless Turkish article boundaries in automatic
+  mode. Optional BGE reranking remains off by default, and query embedding calls
+  run before acquiring the shared search connection.
+
 ## [0.9.0] - 2026-10-28
 
 ### Added

README.md

@@ -12,7 +12,7 @@ Package version: **0.9.0**. V4 is the current API. Production qualification rema
 ## Installation
 
 ```bash
-pip install "mesa-memory[adapters]"
+pip install mesa-memory
 ```
 
 Supported on Linux and macOS natively. On Windows, run via Docker or WSL.

docs/benchmarks/round14_dev.json

@@ -0,0 +1,235 @@
+{
+  "split": "dev",
+  "measured_source_commit": "1e3500d7f9802b977b5729832f45ab903ccd9f08",
+  "environment": {
+    "packages": {
+      "torch": "2.11.0+cu130",
+      "sentence-transformers": "5.7.0",
+      "transformers": "5.21.0",
+      "numpy": "2.5.4",
+      "lancedb": "0.41.0",
+      "kuzu": "0.11.3"
+    },
+    "model_revisions": {
+      "BAAI/bge-m3": "5617a9f61b028005a4858fdac845db406aefb181",
+      "BAAI/bge-reranker-v2-m3": "953dc6f6f85a1b2dbfca4c34a2796e7dde08d41e"
+    },
+    "source_sha256": {
+      "mesa_storage/dao.py": "15c138ed26e251a08687ba62e29d9d7541808ab52cecd1c42bed56259ae0240f",
+      "mesa_memory/config.py": "a175327bee61ad07496b46a5ce79f046fa5e23468302daecfd06f12a37f1fe4c",
+      "mesa_memory/reranking.py": "1f9d4d8ec01044e53cef0606ed8a2a34f0862774a5f97994dd6af7cff7a8c3ef",
+      "mesa_memory/api/server.py": "a459097575c00dab7e6ed25760ee9a9ddaf3b0c9e37fa8bd28661d1e2cbe2521"
+    },
+    "cpu_affinity_count": 12,
+    "cpu_model": "Intel(R) Xeon(R) CPU @ 2.20GHz"
+  },
+  "document_count": 141,
+  "chunk_count": 5041,
+  "question_count": 101,
+  "answerable_count": 85,
+  "limit": 50,
+  "rerank_depth": 50,
+  "embedding_model": "BAAI/bge-m3",
+  "rerank_model": "BAAI/bge-reranker-v2-m3",
+  "other_lane_weights": {
+    "bm25": 1,
+    "assertion": 1,
+    "graph": 1
+  },
+  "selected_default_vector_weight": 8,
+  "best_disabled_mrr_gain": 0.17100820871377637,
+  "selection_rule": "Best disabled MRR; keep 1 unless its unrounded MRR improves by at least 0.02; ties use the smaller weight.",
+  "runtime": {
+    "python": "3.13.16",
+    "torch": "2.11.0+cu130",
+    "gpu": "NVIDIA A100-SXM4-40GB",
+    "cpu_threads": 8,
+    "embedding": "BAAI/bge-m3",
+    "reranker": "BAAI/bge-reranker-v2-m3",
+    "max_length": 1024,
+    "depth": 50,
+    "limit": 50,
+    "graph": true,
+    "extraction": false,
+    "transport": "HTTP route via ASGI (no TCP)",
+    "split": "dev"
+  },
+  "runs": [
+    {
+      "vector_weight": 1,
+      "rerank_enabled": false,
+      "metrics": {
+        "count_total": 101,
+        "count_answerable": 85,
+        "count_no_answer": 16,
+        "passage_hit@1": 0.2941,
+        "passage_hit@5": 0.5176,
+        "passage_hit@10": 0.6,
+        "passage_mrr": 0.3953,
+        "document_hit@1": 0.4941,
+        "document_hit@5": 0.7059,
+        "latency_p50_sec": 1.2693,
+        "latency_p95_sec": 2.0035,
+        "suppressed_vector_count": 267,
+        "no_answer_mean_top_fused_score": 0.061,
+        "no_answer_mean_top_vector_sim": 0.0
+      },
+      "passage_mrr_unrounded": 0.39533925255237007,
+      "warmup_seconds": 5.303557358000035
+    },
+    {
+      "vector_weight": 1,
+      "rerank_enabled": true,
+      "metrics": {
+        "count_total": 101,
+        "count_answerable": 85,
+        "count_no_answer": 16,
+        "passage_hit@1": 0.6353,
+        "passage_hit@5": 0.7412,
+        "passage_hit@10": 0.7647,
+        "passage_mrr": 0.6816,
+        "document_hit@1": 0.6941,
+        "document_hit@5": 0.8118,
+        "latency_p50_sec": 1.9548,
+        "latency_p95_sec": 2.5736,
+        "suppressed_vector_count": 200,
+        "no_answer_mean_top_fused_score": 0.0558,
+        "no_answer_mean_top_vector_sim": 0.0
+      },
+      "passage_mrr_unrounded": 0.6815644636232872,
+      "warmup_seconds": 7.091411409000102
+    },
+    {
+      "vector_weight": 2,
+      "rerank_enabled": false,
+      "metrics": {
+        "count_total": 101,
+        "count_answerable": 85,
+        "count_no_answer": 16,
+        "passage_hit@1": 0.3647,
+        "passage_hit@5": 0.5765,
+        "passage_hit@10": 0.6824,
+        "passage_mrr": 0.4565,
+        "document_hit@1": 0.5529,
+        "document_hit@5": 0.7294,
+        "latency_p50_sec": 1.2708,
+        "latency_p95_sec": 1.964,
+        "suppressed_vector_count": 304,
+        "no_answer_mean_top_fused_score": 0.0763,
+        "no_answer_mean_top_vector_sim": 0.0
+      },
+      "passage_mrr_unrounded": 0.45648975693603044,
+      "warmup_seconds": 2.664795117000722
+    },
+    {
+      "vector_weight": 2,
+      "rerank_enabled": true,
+      "metrics": {
+        "count_total": 101,
+        "count_answerable": 85,
+        "count_no_answer": 16,
+        "passage_hit@1": 0.6588,
+        "passage_hit@5": 0.7529,
+        "passage_hit@10": 0.7882,
+        "passage_mrr": 0.7055,
+        "document_hit@1": 0.7176,
+        "document_hit@5": 0.8235,
+        "latency_p50_sec": 1.9569,
+        "latency_p95_sec": 2.6364,
+        "suppressed_vector_count": 290,
+        "no_answer_mean_top_fused_score": 0.0713,
+        "no_answer_mean_top_vector_sim": 0.0
+      },
+      "passage_mrr_unrounded": 0.7055448473095532,
+      "warmup_seconds": 3.410192898999412
+    },
+    {
+      "vector_weight": 4,
+      "rerank_enabled": false,
+      "metrics": {
+        "count_total": 101,
+        "count_answerable": 85,
+        "count_no_answer": 16,
+        "passage_hit@1": 0.4118,
+        "passage_hit@5": 0.6588,
+        "passage_hit@10": 0.7529,
+        "passage_mrr": 0.5152,
+        "document_hit@1": 0.5765,
+        "document_hit@5": 0.7765,
+        "latency_p50_sec": 1.2836,
+        "latency_p95_sec": 1.9469,
+        "suppressed_vector_count": 215,
+        "no_answer_mean_top_fused_score": 0.1086,
+        "no_answer_mean_top_vector_sim": 0.0
+      },
+      "passage_mrr_unrounded": 0.5152291602446402,
+      "warmup_seconds": 3.09878310699969
+    },
+    {
+      "vector_weight": 4,
+      "rerank_enabled": true,
+      "metrics": {
+        "count_total": 101,
+        "count_answerable": 85,
+        "count_no_answer": 16,
+        "passage_hit@1": 0.6588,
+        "passage_hit@5": 0.7765,
+        "passage_hit@10": 0.8,
+        "passage_mrr": 0.7069,
+        "document_hit@1": 0.7294,
+        "document_hit@5": 0.8471,
+        "latency_p50_sec": 1.9279,
+        "latency_p95_sec": 2.61,
+        "suppressed_vector_count": 291,
+        "no_answer_mean_top_fused_score": 0.1023,
+        "no_answer_mean_top_vector_sim": 0.0
+      },
+      "passage_mrr_unrounded": 0.7069098548510313,
+      "warmup_seconds": 3.857730206999804
+    },
+    {
+      "vector_weight": 8,
+      "rerank_enabled": false,
+      "metrics": {
+        "count_total": 101,
+        "count_answerable": 85,
+        "count_no_answer": 16,
+        "passage_hit@1": 0.4706,
+        "passage_hit@5": 0.6706,
+        "passage_hit@10": 0.7529,
+        "passage_mrr": 0.5663,
+        "document_hit@1": 0.6353,
+        "document_hit@5": 0.7647,
+        "latency_p50_sec": 1.2849,
+        "latency_p95_sec": 1.9115,
+        "suppressed_vector_count": 147,
+        "no_answer_mean_top_fused_score": 0.1732,
+        "no_answer_mean_top_vector_sim": 0.0
+      },
+      "passage_mrr_unrounded": 0.5663474612661464,
+      "warmup_seconds": 3.122546444999898
+    },
+    {
+      "vector_weight": 8,
+      "rerank_enabled": true,
+      "metrics": {
+        "count_total": 101,
+        "count_answerable": 85,
+        "count_no_answer": 16,
+        "passage_hit@1": 0.6471,
+        "passage_hit@5": 0.7647,
+        "passage_hit@10": 0.7882,
+        "passage_mrr": 0.6973,
+        "document_hit@1": 0.7176,
+        "document_hit@5": 0.8353,
+        "latency_p50_sec": 1.9161,
+        "latency_p95_sec": 2.536,
+        "suppressed_vector_count": 287,
+        "no_answer_mean_top_fused_score": 0.1642,
+        "no_answer_mean_top_vector_sim": 0.0
+      },
+      "passage_mrr_unrounded": 0.6973020117137764,
+      "warmup_seconds": 3.4035570070000176
+    }
+  ]
+}

docs/configuration.md

@@ -30,6 +30,26 @@ and `MESA_LOG_FORMAT` to json. Secrets must be injected privately.
 
 ## Providers and capacity
 
+`MESA_CHUNKING=auto` (default) splits new document revisions at Turkish article
+headings when at least three occur at line starts. The preamble stays separate;
+long articles use the existing paragraph/sentence splitting with a 512-token
+limit. All slices preserve the exact source text. Other documents keep the same
+plain chunks; `MESA_CHUNKING=plain` selects that plan for every document. Retrying
+a pending upload preserves the plan matching its already admitted chunk texts,
+even after restarting or changing this setting. Active revisions are unchanged.
+If an admitted prefix fits two different plans, the retry returns 409 without
+appending chunks, because the stored text cannot identify the original plan.
+
+`MESA_RERANK_ENABLED=false` keeps the existing fused ranking and does not import
+ML packages. Enabling it requires `pip install "mesa-memory[ml]"`; startup checks
+that the package is installed, and the model loads lazily on the first search.
+`MESA_RERANK_MODEL` defaults to `BAAI/bge-reranker-v2-m3` (Apache-2.0), with
+`MESA_RERANK_DEPTH=50` (range 1–50). The scorer uses authorized full chunk texts
+after releasing the search connection, in a worker thread, then returns the
+requested number of results. Tied scores retain fused order. Results keep their
+fusion fields, including `rrf_score` and `final_score`, and add `rerank_score`.
+The first search includes model-load latency; subsequent searches reuse it.
+
 Model and external-provider flags default false. Validation defaults to
 mode 0; mode 1 or 2 must be set explicitly via `MESA_TIER3_MODE`. Mode 2
 needs distinct A/B validator identities. Extraction and validation are separate.
@@ -130,7 +150,7 @@ An endpoint is considered **local** when its scheme is `http` or `https` and its
 
 If an optional provider or model extra is selected but not installed, server startup halts with a clear one-line installation command:
 - Ollama LLM, Ollama embedding, or OpenAI-compatible embedding:
-  `pip install "mesa-memory[adapters]"`
+  `pip install mesa-memory`
 - Local transformer embedding (`sentence-transformers`):
   `pip install "mesa-memory[ml]"`
 
@@ -206,10 +226,15 @@ Field-declaration reference; compare it with `MesaConfig` when configuration cha
 | `metrics_admission_threshold` | `METRICS_ADMISSION_THRESHOLD` | `0.8` |
 | `metrics_divergence_threshold` | `METRICS_DIVERGENCE_THRESHOLD` | `0.5` |
 | `rrf_k` | `MESA_RRF_K` | `60` |
-| `rrf_vector_weight` | `MESA_RRF_VECTOR_WEIGHT` | `1.0` |
+| `rrf_vector_weight` | `MESA_RRF_VECTOR_WEIGHT` | `8.0` |
 | `rrf_bm25_weight` | `MESA_RRF_BM25_WEIGHT` | `1.0` |
 | `rrf_assertion_weight` | `MESA_RRF_ASSERTION_WEIGHT` | `1.0` |
 | `rrf_graph_weight` | `MESA_RRF_GRAPH_WEIGHT` | `1.0` |
+| `search_scope_cache` | `MESA_SEARCH_SCOPE_CACHE` | `True` |
+| `chunking` | `MESA_CHUNKING` | `'auto'` |
+| `rerank_enabled` | `MESA_RERANK_ENABLED` | `False` |
+| `rerank_model` | `MESA_RERANK_MODEL` | `'BAAI/bge-reranker-v2-m3'` |
+| `rerank_depth` | `MESA_RERANK_DEPTH` | `50` |
 | `t_route` | `MESA_T_ROUTE` | `0.85` |
 | `v4_rebuild_enabled` | `MESA_V4_REBUILD_ENABLED` | `False` |
 | `v4_graph_timeout_seconds` | `MESA_V4_GRAPH_TIMEOUT_SECONDS` | `15.0` |