Mvp/round 16

.env.example

@@ -8,7 +8,7 @@ MESA_RUNTIME_PROFILE=combined
 MESA_STORAGE_ROOT=/var/lib/mesa/data
 MESA_LOAD_DOTENV=false
 MESA_MODEL_ENABLED=false
-# Each defaults to MESA_MODEL_ENABLED when unset.
+# Embedding defaults to MESA_MODEL_ENABLED when unset; extraction defaults to false.
 # MESA_EMBEDDING_ENABLED=false
 # MESA_EXTRACTION_ENABLED=false
 # Graph is independent of model flags; false never opens Kùzu.

CHANGELOG.md

@@ -7,6 +7,35 @@ are historical; consult the [current guides](docs/README.md) for present behavio
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+### Changed
+
+- Queries matching a decision's own docket/decision reference promote its
+  first chunk through the existing exact-match slot, shared with articles
+  (at most three chunks total). Only stored references beginning in the first
+  300 characters of an authorized first chunk count; later quotations do not.
+
+- Newly admitted chunks keep versioned Turkish legal rule items with chunk
+  offsets, returned in search provenance separately from SPO facts. Item storage
+  preserves fusion scores, and document purge erases their source quotes. The existing
+  `MESA_LEGAL_RULES_ENABLED` switch controls admission.
+
+- Constitutional Court names (`Anayasa Mahkemesi`, its suffix forms and `AYM`)
+  no longer activate Constitution citation ranking. Separate statute citations
+  in the same query are preserved.
+
+- Exact law/article matches now occupy at most three leading search positions
+  in document order, preserving the remaining order. Authorized document titles
+  and law headers provide scope-cached law names, including the `K.` abbreviation.
+- Document article plans assign the existing typed legal identity to ordinary
+  articles and their continuation chunks. Generic 3–5 digit law-number
+  citations use the existing retrieval lane; `MESA_LEGAL_RULES_ENABLED=false`
+  disables automatic identity assignment for new chunks. Document fallback
+  assertions preserve their existing document/chunk graph and lexical shape.
+- Open-ended SPO fact extraction (`MESA_EXTRACTION_ENABLED`) defaults to false
+  regardless of `MESA_MODEL_ENABLED`; an explicit true still enables it (D46).
+
 ## [0.9.1] - 2026-11-01
 
 This release combines Round 13 (speed) and Round 14 (retrieval quality and

README.md

@@ -56,14 +56,16 @@ mesa mcp-config
 
 ## Internal models (optional)
 
-Models stay off by default. Documents are stored and found through BM25 lexical search without any model running. `FactExtractionService` owns canonical V4 extraction when models are enabled.
+Models stay off by default. Documents are stored and found through BM25 lexical search without any model running. `FactExtractionService` owns canonical V4 extraction when explicitly enabled (`MESA_EXTRACTION_ENABLED=true`).
 
-To enable semantic vector search and fact extraction with local Ollama, add to `~/.mesa/.env`:
+To enable semantic vector search with local Ollama, add to `~/.mesa/.env`:
 
 ```bash
 MESA_MODEL_ENABLED=true
-MESA_LLM_MODEL=ollama:qwen2.5:7b
 MESA_EMBEDDING_MODEL=ollama:bge-m3
+# To also enable open-ended fact extraction (defaults to false):
+# MESA_EXTRACTION_ENABLED=true
+# MESA_LLM_MODEL=ollama:qwen2.5:7b
 ```
 
 Local `openai_compatible` servers (such as vLLM started with `--port 8001`, at `MESA_LLM_BASE_URL=http://127.0.0.1:8001/v1`; MESA itself uses port 8000) are also allowed with `MESA_EXTERNAL_PROVIDER_ENABLED=false` on a local endpoint (`localhost`, `127.0.0.1`, `::1`, `host.docker.internal`) and require no API key (D31). For cloud model providers (OpenAI, Anthropic), set `MESA_EXTERNAL_PROVIDER_ENABLED=true` and provide the required API keys.

docker-compose.v4.yml

@@ -30,7 +30,7 @@ x-mesa-v4-runtime: &mesa-v4-runtime
     MESA_LOAD_DOTENV: "false"
     MESA_MODEL_ENABLED: ${MESA_MODEL_ENABLED:-false}
     MESA_EMBEDDING_ENABLED: ${MESA_EMBEDDING_ENABLED:-${MESA_MODEL_ENABLED:-false}}
-    MESA_EXTRACTION_ENABLED: ${MESA_EXTRACTION_ENABLED:-${MESA_MODEL_ENABLED:-false}}
+    MESA_EXTRACTION_ENABLED: ${MESA_EXTRACTION_ENABLED:-false}
     MESA_GRAPH_ENABLED: ${MESA_GRAPH_ENABLED:-true}
     MESA_EXTERNAL_PROVIDER_ENABLED: ${MESA_EXTERNAL_PROVIDER_ENABLED:-false}
     MESA_LLM_MODEL: ${MESA_LLM_MODEL:-}

docs/configuration.md

@@ -40,6 +40,45 @@ even after restarting or changing this setting. Active revisions are unchanged.
 If an admitted prefix fits two different plans, the retry returns 409 without
 appending chunks, because the stored text cannot identify the original plan.
 
+`MESA_LEGAL_RULES_ENABLED=true` enables Turkish legal rule processing. In auto
+chunking mode, ordinary article chunks inherit the document's law number from
+`metadata.law_number` or a `Kanun Numarası`, `Kanunun Numarası` or `Kanun No.`
+header in the first 3,000 characters. Continuation pieces keep that article's
+identity; preambles, GEÇİCİ/EK articles and plain plans receive none. Setting the
+switch to false disables automatic identity assignment for new chunks. Stored
+identities are immutable on resume. Any 3–5 digit law number is accepted by the
+typed identity API; known numbers retain their existing statute code (6098 →
+TBK). Unknown law names and inconsistent number/code pairs are rejected.
+Numbered queries such as `7201 sayılı Kanunun 21. maddesi` use the existing
+typed legal candidate lane and ranking factors. Document fallback assertions
+keep their document subject, `madde` relation and chunk object; automatic
+article identity changes only the typed columns.
+
+Newly admitted chunks also keep versioned citation, decision-reference, date,
+amount, duration and outcome items. Search returns them in
+`retrieval_provenance.legal_items` as `{version, items}`. Item `start`/`end`
+offsets refer to the original chunk, including when a result quotes a shorter
+extracted fact. Items remain separate from SPO facts and do not enter fusion
+scores. Decision references can use the bounded exact-match slot below.
+Disabling the switch prevents new item snapshots; existing
+snapshots remain. Purging a document erases these source quotes too.
+
+An exact law-and-article query puts at most three matching chunks first, in
+document order. Other results retain their fused order (or reranked order when
+reranking is enabled). The match must exist in the authorized search scope.
+A query's docket and decision numbers can also match a decision's own künye:
+a stored `decision_ref` starting within the first 300 characters of chunk zero.
+Its first chunk shares the same three-chunk limit with article matches, in
+document order. Later quotations do not teach a decision's identity. The map
+uses the same tenant, dataset, agent and time scope cache as law names; an
+unmatched reference leaves result order unchanged.
+The scope cache learns law names from titles and first-chunk law headers of
+documents with trusted article identities; it never learns them from another
+tenant, dataset, agent or time scope. Names accept case and Turkish ı/i variants,
+possessive suffixes, and `K.` for `Kanunu`. The longest exact name from that scope
+takes precedence over an overlapping ontology alias. Ambiguous store names
+shared by different law numbers are ignored. No static law-name table is added.
+
 `MESA_RERANK_ENABLED=false` keeps the existing fused ranking and does not import
 ML packages. Enabling it requires `pip install "mesa-memory[ml]"`; startup checks
 that the package is installed, and the model loads lazily on the first search.
@@ -53,8 +92,8 @@ The first search includes model-load latency; subsequent searches reuse it.
 Model and external-provider flags default false. Validation defaults to
 mode 0; mode 1 or 2 must be set explicitly via `MESA_TIER3_MODE`. Mode 2
 needs distinct A/B validator identities. Extraction and validation are separate.
-`MESA_EMBEDDING_ENABLED` and `MESA_EXTRACTION_ENABLED` each default to
-`MESA_MODEL_ENABLED`. A combined runtime accepts writes with both off and finds
+`MESA_EMBEDDING_ENABLED` defaults to `MESA_MODEL_ENABLED`; `MESA_EXTRACTION_ENABLED`
+defaults to `false` (D46). A combined runtime accepts writes with both off and finds
 them through BM25. Disabled VECTOR lanes end `SKIPPED`; enabling embedding later
 backfills them at startup using the same embedding identity. Backfill failures
 leave committed records searchable.

mesa_api/v4_router.py

@@ -11,19 +11,34 @@
 import asyncio
 import hashlib
 import logging
+import re
 from datetime import datetime
-from typing import Any, Callable, Literal
+from typing import Annotated, Any, Callable, Literal
 
 from fastapi import APIRouter, Depends, Header, HTTPException, Request
-from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
+from fastapi.exceptions import RequestValidationError
+from pydantic import (
+    BaseModel,
+    ConfigDict,
+    Field,
+    ValidationError,
+    WrapValidator,
+    field_validator,
+    model_validator,
+)
 
 from mesa_api.admission import require_mutation_admission as _require_mutation_admission
 from mesa_api.identifier_validation import PublicIdentifier, SourceIdentifier
 from mesa_memory.adapter.factory import AdapterFactory
 from mesa_memory.config import config, configured_embedding_identity
 from mesa_memory.consolidation.policy import ValidationPolicy
 from mesa_memory.context_builder import ContextBuilder
-from mesa_memory.document_chunking import chunk_document, chunk_legal_document
+from mesa_memory.document_chunking import (
+    _ARTICLE_HEADING,
+    _CASE_LAW_PREAMBLE,
+    chunk_document,
+    chunk_legal_document,
+)
 from mesa_memory.embedding.service import (
     EmbeddingGenerationError,
     EmbeddingIdentityMismatchError,
@@ -1148,12 +1163,22 @@ async def rebuild_index(
             request, idempotency_key, dao, access_control
         )
 
-    @router.post("/memory/insert", status_code=202)
-    async def insert_memory(
+    @router.post("/memory/insert", status_code=202, name="insert_memory")
+    async def insert_memory_request(
         request: Request,
         payload: V4MemoryInsertRequest,
         dao: MemoryDAO = Depends(get_dao),
         access_control: AccessControl = Depends(get_access_control),
+    ) -> dict:
+        return await insert_memory(request, payload, dao, access_control)
+
+    async def insert_memory(
+        request: Request,
+        payload: V4MemoryInsertRequest,
+        dao: MemoryDAO,
+        access_control: AccessControl,
+        *,
+        document_chunk: bool = False,
     ) -> dict:
         session = await _authorized_v4_session(
             request,
@@ -1190,6 +1215,7 @@ async def insert_memory(
                 chunk_ordinal=payload.chunk_ordinal,
                 supersedes_revision_id=payload.supersedes_revision_id,
                 metadata=payload.metadata,
+                document_chunk=document_chunk,
                 legal_identity=(
                     payload.legal_identity.model_dump(exclude_none=True)
                     if payload.legal_identity is not None
@@ -1232,10 +1258,23 @@ async def insert_memory(
     @router.post("/documents", response_model=V4DocumentIngestResponse, status_code=202)
     async def ingest_document(
         request: Request,
-        payload: V4DocumentIngestRequest,
+        payload: Annotated[
+            V4DocumentIngestRequest, WrapValidator(lambda value, _handler: value)
+        ],
         dao: MemoryDAO = Depends(get_dao),
         access_control: AccessControl = Depends(get_access_control),
     ) -> dict:
+        # Defer, rather than omit, the same model validation. The annotation
+        # retains its public schema; no payload is used before validation.
+        try:
+            payload = await asyncio.to_thread(
+                V4DocumentIngestRequest.model_validate, payload
+            )
+        except ValidationError as exc:
+            errors = exc.errors()
+            for error in errors:
+                error["loc"] = ("body", *error["loc"])
+            raise RequestValidationError(errors) from exc
         session = await _authorized_v4_session(
             request, dao, access_control, payload.session_id, level="WRITE"
         )
@@ -1289,7 +1328,8 @@ async def ingest_document(
                     raise ValueError(
                         f"content equals revision {revision['revision_number']} "
                         f"({revision['status']}); restoring an earlier revision "
-                        "is not supported in v0.9"
+                        "is not supported in v0.9. Add it under a new document_id "
+                        "to store it again."
                     )
                 admitted = await dao.list_v4_revision_mutations(
                     **scope, revision_id=revision_id
@@ -1305,10 +1345,11 @@ async def ingest_document(
                     "mutation_ids": [admitted[i] for i in sorted(admitted)],
                     "duplicate": True,
                 }
+            legal_plan = config.chunking == "auto"
             chunks = (
-                chunk_legal_document(payload.content)
+                await asyncio.to_thread(chunk_legal_document, payload.content)
                 if config.chunking == "auto"
-                else chunk_document(payload.content)
+                else await asyncio.to_thread(chunk_document, payload.content)
             )
             if admitted:
                 # Admissions are already scoped above. Their durable payloads,
@@ -1327,22 +1368,68 @@ def matches(plan: list[str]) -> bool:
                     )
 
                 alternative = (
-                    chunk_document(payload.content)
+                    await asyncio.to_thread(chunk_document, payload.content)
                     if config.chunking == "auto"
-                    else chunk_legal_document(payload.content)
+                    else await asyncio.to_thread(chunk_legal_document, payload.content)
                 )
                 preferred_matches = matches(chunks)
                 alternative_matches = matches(alternative)
                 if preferred_matches and alternative_matches and chunks != alternative:
                     raise HTTPException(
-                        409, "admitted chunking plan is ambiguous; no chunks appended"
+                        409,
+                        "admitted chunking plan is ambiguous; no chunks appended. "
+                        "Retry with the MESA_CHUNKING value used when this revision was "
+                        "first added, or add the content under a new document_id.",
                     )
                 if not preferred_matches:
                     if not alternative_matches:
                         raise HTTPException(
-                            409, "admitted chunks do not match a supported chunking plan"
+                            409,
+                            "admitted chunks do not match a supported chunking plan. "
+                            "Retry with the MESA_CHUNKING value used when this revision was "
+                            "first added, or add the content under a new document_id.",
                         )
                     chunks = alternative
+                    legal_plan = not legal_plan
+            identities: list[dict[str, str] | None] = [None] * len(chunks)
+            if config.legal_rules_enabled and legal_plan:
+                headings = list(_ARTICLE_HEADING.finditer(payload.content))
+                header = re.search(
+                    r"\bKanun(?:un)?\s+(?:Numarası\s*:|No\.?\s*:?)\s*([0-9]{3,5})\b",
+                    payload.content[:3000],
+                    re.IGNORECASE,
+                )
+                law_number = str(
+                    payload.metadata.get("law_number") or (header[1] if header else "")
+                )
+                if (
+                    re.fullmatch(r"[0-9]{3,5}", law_number)
+                    and len(headings) >= 3
+                    and not _CASE_LAW_PREAMBLE.search(
+                        payload.content[: headings[0].start()]
+                    )
+                ):
+                    heading_index = -1
+                    offset = 0
+                    for ordinal, chunk in enumerate(chunks):
+                        while (
+                            heading_index + 1 < len(headings)
+                            and headings[heading_index + 1].start() <= offset
+                        ):
+                            heading_index += 1
+                        if heading_index >= 0:
+                            # Special/range/suffixed articles have no ordinary article identity.
+                            ordinary = re.fullmatch(
+                                r"\s*MADDE\s+([0-9]+)\s*[-–—−:]",
+                                headings[heading_index][0],
+                                re.IGNORECASE,
+                            )
+                            if ordinary:
+                                identities[ordinal] = {
+                                    "law_id": law_number,
+                                    "article_id": ordinary[1],
+                                }
+                        offset += len(chunk)
             duplicate = len(admitted) == len(chunks)
             for ordinal, chunk in enumerate(chunks):
                 if ordinal in admitted:
@@ -1353,6 +1440,7 @@ def matches(plan: list[str]) -> bool:
                     V4MemoryInsertRequest(
                         **payload.model_dump(exclude={"content"}),
                         content=chunk,
+                        legal_identity=identities[ordinal],
                         revision_id=revision_id,
                         chunk_id=chunk_id,
                         revision_number=revision["revision_number"],
@@ -1363,6 +1451,7 @@ def matches(plan: list[str]) -> bool:
                     ),
                     dao,
                     access_control,
+                    document_chunk=True,
                 )
                 admitted[ordinal] = response["mutation_id"]
             return {

mesa_memory/api/server.py

@@ -378,6 +378,7 @@ async def _runtime_lifespan(app: FastAPI, runtime: RuntimeProfileConfig):
             "graph": config.rrf_graph_weight,
         },
         search_scope_cache_enabled=config.search_scope_cache,
+        legal_rules_enabled=config.legal_rules_enabled,
         reranker=reranker,
         rerank_depth=config.rerank_depth,
     )

mesa_memory/config.py

@@ -189,7 +189,7 @@ def load_runtime_profile(
     extraction_enabled = _parse_runtime_bool(
         values.get("MESA_EXTRACTION_ENABLED"),
         name="MESA_EXTRACTION_ENABLED",
-        default=model_enabled,
+        default=False,
     )
     graph_enabled = _parse_runtime_bool(
         values.get("MESA_GRAPH_ENABLED"), name="MESA_GRAPH_ENABLED", default=True
@@ -616,6 +616,7 @@ def vector_worker_limit(self) -> int:
     chunking: Literal["auto", "plain"] = Field(
         "auto", validation_alias="MESA_CHUNKING"
     )
+    legal_rules_enabled: bool = Field(True, validation_alias="MESA_LEGAL_RULES_ENABLED")
     metrics_admission_threshold: float = 0.8
     metrics_divergence_threshold: float = 0.5