Curation: live claims board (who works on which dataset), warn-only

.claude/agents/candidate-screener.md

@@ -22,7 +22,7 @@ Follow `.claude/skills/screen-candidates/SKILL.md`, steps 3 to 6, for each recor
 4. Write each non-rejected record through the store API right after reloading it (`records.md`): links, metadata only
    where the source states it, `priority`, `AI (UNVERIFIED)` in `checked_by`, `AI-Filled (Verify)` in `tags`, the
    disclaimer and the screening block at the top of the comments, every existing note kept. Skip a record another
-   curator has claimed.
+   curator holds on the claims board (`claims check <names>`; the main session claims the batch before it starts).
 
 Never commit, never build, never delete a comment, never name company code repositories, never send personal data to
 an outside service. Cap the threads of any Python run (`OMP_NUM_THREADS=2` and the like).

.claude/hooks/claims_hook.py

@@ -0,0 +1,116 @@
+"""Claude Code hook: tell the agent who holds which dataset (the live claims board), and warn before it edits one.
+
+Wired in ``.claude/settings.json``:
+
+* ``SessionStart``: fetches the board and adds it to the session's context: the other curators' claims, your own,
+  and yours that expire within two days.
+* ``PreToolUse`` on ``Edit|Write|NotebookEdit``: when the file belongs to a dataset (``curation/records/<name>.md``
+  or ``datasets/_dev/<copy>/<name>/...``) that another curator holds, shows a warning to the user and to the agent.
+  It never allows or denies anything: claims are advisory, and the normal permission flow goes on.
+
+The board is re-fetched at most every 60 s (else the cache of the last fetch is used). The hook fails open: without
+the package, a curator name, the network or a board it prints nothing and exits 0.
+"""
+
+from __future__ import annotations
+
+import json
+import re
+import sys
+
+PATTERNS = (
+    re.compile(r"/curation/records/([^/]+)\.md$"),
+    re.compile(r"/datasets/_dev/[^/]+/([^/_.][^/]*)/"),
+)
+MAX_NAMES = 60
+
+
+def dataset_of(path: str) -> str | None:
+    """The dataset a file belongs to, or ``None``."""
+    path = path.replace("\\", "/")
+    for pattern in PATTERNS:
+        match = pattern.search(path)
+        if match:
+            return match.group(1)
+    return None
+
+
+def _names(items: list) -> str:
+    names = [c.name for c in items]
+    head = ", ".join(names[:MAX_NAMES])
+    return head + (f" and {len(names) - MAX_NAMES} more" if len(names) > MAX_NAMES else "")
+
+
+def session_start(claims) -> dict:
+    """The board as context for the agent."""
+    board = claims.load(max_age=0)
+    me = claims.configured_curator()
+    lines = [
+        f"Dataset claims (the live board on {board.remote}/{board.branch}, fetched at session start). A claim covers "
+        "a dataset's curation record and its v2 folder. Do not work on a dataset another curator holds; ask the user "
+        "first. Commands: `data-foundry-curation claims list|claim|release|renew|check`.",
+    ]
+    if not me:
+        lines.append(f"No curator name is set: `git config {claims.CURATOR_GIT_KEY} <Name>` before claiming.")
+    others = {h: items for h, items in board.by_holder().items() if h != me}
+    mine = [c for c in board.claims.values() if c.holder == me]
+    if others:
+        lines.append("Claimed by others: " + "; ".join(f"{h} ({len(i)}): {_names(i)}" for h, i in others.items()))
+    else:
+        lines.append("No other curator holds a claim.")
+    if me:
+        lines.append(f"Yours ({len(mine)}): {_names(sorted(mine, key=lambda c: c.name))}" if mine else "Yours: none.")
+        soon = [c for c in mine if c.seconds_left() < 2 * 86400]
+        if soon:
+            lines.append(f"Yours expiring within 2 days (renew with `claims renew`): {_names(soon)}")
+    return {"hookSpecificOutput": {"hookEventName": "SessionStart", "additionalContext": "\n".join(lines)}}
+
+
+def pre_tool_use(claims, payload: dict) -> dict | None:
+    """A warning when the edited file belongs to a dataset someone else holds."""
+    name = dataset_of(str((payload.get("tool_input") or {}).get("file_path") or ""))
+    if not name:
+        return None
+    me = claims.configured_curator()
+    if not me:
+        return None
+    try:
+        board = claims.load(max_age=60)
+    except claims.ClaimsError:
+        board = claims.cached_board()
+    claim = board.held_by_other(name, me) if board is not None else None
+    if claim is None:
+        return None
+    text = f"Claim warning: {name} is claimed by {claim.describe()}."
+    return {
+        "systemMessage": text,
+        "hookSpecificOutput": {
+            "hookEventName": "PreToolUse",
+            "additionalContext": f"{text} Another curator is working on this dataset; stop and ask the user before "
+            "changing it further.",
+        },
+    }
+
+
+def main() -> int:
+    """Read the hook's JSON from stdin and print the answer, if any."""
+    try:
+        payload = json.loads(sys.stdin.read() or "{}")
+        from data_foundry.curation import claims  # noqa: PLC0415 - fail open without the package
+
+        event = payload.get("hook_event_name")
+        if event == "SessionStart":
+            out = session_start(claims)
+        elif event == "PreToolUse":
+            out = pre_tool_use(claims, payload)
+        else:
+            out = None
+    except Exception:  # noqa: BLE001 - a hook must never get in the way
+        return 0
+    if out:
+        print(json.dumps(out))
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

.claude/settings.json

@@ -0,0 +1,27 @@
+{
+  "hooks": {
+    "SessionStart": [
+      {
+        "hooks": [
+          {
+            "type": "command",
+            "command": "test -x \"$CLAUDE_PROJECT_DIR/.venv/bin/python\" && \"$CLAUDE_PROJECT_DIR/.venv/bin/python\" \"$CLAUDE_PROJECT_DIR/.claude/hooks/claims_hook.py\" || true",
+            "timeout": 30
+          }
+        ]
+      }
+    ],
+    "PreToolUse": [
+      {
+        "matcher": "Edit|Write|NotebookEdit",
+        "hooks": [
+          {
+            "type": "command",
+            "command": "test -x \"$CLAUDE_PROJECT_DIR/.venv/bin/python\" && \"$CLAUDE_PROJECT_DIR/.venv/bin/python\" \"$CLAUDE_PROJECT_DIR/.claude/hooks/claims_hook.py\" || true",
+            "timeout": 30
+          }
+        ]
+      }
+    ]
+  }
+}

.claude/skills/README.md

@@ -114,6 +114,17 @@ A full panel costs on the order of a million tokens per dataset; for a re-check
 smaller one. In stage 1, two agents help: a screener for batches and a reject checker that argues against every quick
 `No`.
 
+## Who works on what
+
+A claim says "I am working on this dataset now" (its record and its folder). Claims live on the branch `claims` of
+the repository, pushed directly, so everyone sees them within seconds:
+[the board on GitHub](https://github.com/TabArena/data-foundry/blob/claims/README.md), the Claimed by column of the
+dashboard, and the agents (a hook lists the board at the start of each session and warns before an agent edits a
+dataset someone else holds). The commands claim before they work and skip what someone else holds. Set your name once
+with `git config data-foundry.curator <Name>`; to claim by hand, pick your name in the dashboard's Claimed by cell or
+run `data-foundry-curation claims claim <names>`. The details:
+[claims.md](knowledge-data-foundry/references/claims.md).
+
 ## Where the rules are
 
 * [knowledge-curation](knowledge-curation/SKILL.md): how we decide, independent of the tooling. Start with

.claude/skills/add-dataset/SKILL.md

@@ -25,7 +25,7 @@ Two modes:
 | Step | Read |
 |---|---|
 | all | [principles.md](../knowledge-curation/references/principles.md): rank, strengths, the remedy ladder, the judgment record |
-| 0 | [records.md](../knowledge-data-foundry/references/records.md) (claims, pointers); for a batch [agent_runs.md](../knowledge-data-foundry/references/agent_runs.md) |
+| 0 | [claims.md](../knowledge-data-foundry/references/claims.md), [records.md](../knowledge-data-foundry/references/records.md) (pointers); for a batch [agent_runs.md](../knowledge-data-foundry/references/agent_runs.md) |
 | 1 | [use_case.md](../knowledge-curation/references/use_case.md), [provenance.md](../knowledge-curation/references/provenance.md), [metrics.md](../knowledge-curation/references/metrics.md) (MET-1 to MET-3) |
 | 1-3 | [definitions.md](../knowledge-data-foundry/references/definitions.md): the folder, the class, the field mapping, the recipe, the split declarations, the reference definitions |
 | 3 | [cleaning_features.md](../knowledge-curation/references/cleaning_features.md), [splits.md](../knowledge-curation/references/splits.md), [metrics.md](../knowledge-curation/references/metrics.md) (MET-11 to MET-13) and definitions.md, "Metrics" |
@@ -52,8 +52,10 @@ errors, and `dataset build` refuses to save while any remains.
 Read `curation/records/<unique_name>.md`: the front matter and the comments, above all the screening block (the
 source, the use case, the regime, the risks). Verify its claims as you go instead of redoing its research; a
 measurement on the data supersedes what screening read from the page. No record: stop and point to
-`/screen-candidates`. Never work on a record another curator has claimed; the records you take get `WIP (DF)` and the
-curator's name in `claimed_by` first, and the claim goes to `main` in its own small PR.
+`/screen-candidates`. Claim the records first, in the curator's name: `claims claim <names> --stage add --note
+"add-dataset"` ([claims.md](../knowledge-data-foundry/references/claims.md)); it pushes the claim to the board and
+writes `WIP (DF)` and `claimed_by` into the records. Never work on a dataset another curator holds: a name the command
+skips is left out and reported.
 
 A batch (several names or a dashboard filter) runs one `dataset-builder` agent per source unit for steps 0-6
 ([agent_runs.md](../knowledge-data-foundry/references/agent_runs.md)); the main session then runs the panel (step 7)

.claude/skills/knowledge-data-foundry/SKILL.md

@@ -16,6 +16,7 @@ implements. The overview for people is the [skills README](../README.md).
 |---|---|---|
 | The pipeline: the stages and commands, the record after each step, the queues, which file is the truth for what, the agents | [pipeline.md](references/pipeline.md) | finding where a record stands or what comes next |
 | The curation records: editing through the store API, comment lines (`CC`, `Assessment`, `Counter-argument`, `Open issues`), AI triage, fields and vocabulary, claims, duplicates and retirements, the review queue, the definition pointers | [records.md](references/records.md) | before reading or writing any record |
+| Claims: the live board of who works on which dataset (a git branch), the `claims` commands, the hook, the records' mirror | [claims.md](references/claims.md) | before working on records or folders, and when a claim warning shows |
 | The dashboard: running and restarting it, its filters, pills, pins and links | [dashboard.md](references/dashboard.md) | starting it or explaining it to a curator |
 | The v2 definition: the folder, `dataset.py`, the base class, `task_summary`, field mapping, `curation_comments`, the `_clean` recipe, `Grouping` / `Temporal` / `TemporalSplits`, `_1m` versions, traps, bundle-check slugs, `_decisions`, recording decisions, reference definitions, the harness | [definitions.md](references/definitions.md) | writing or reviewing a `dataset.py` |
 | Running `dataset check` and the probes, and the checks only Data Foundry has (pointers, template, README) | [checks_and_probes.md](references/checks_and_probes.md) | verifying a definition |

.claude/skills/knowledge-data-foundry/references/agent_runs.md

@@ -6,9 +6,10 @@ How a batch of records is worked by parallel agents on this machine, and the lim
 ## A batch of records
 
 When the curator names several records or a dashboard filter (for example "every record with `suggestion = Yes`,
-`data_foundry_status` without `DF: Yes`, `claimed_by` empty, `checked_by` without `AI (UNVERIFIED)`"):
+`data_foundry_status` without `DF: Yes`, unclaimed, `checked_by` without `AI (UNVERIFIED)`"):
 
-1. List the matching records first and skip any claimed by another curator.
+1. List the matching records first, then claim them in one call (`claims claim <names> --note "<run>"`,
+   [claims.md](claims.md)) and work only on the ones it claimed: a skipped name is held by another curator.
 2. Group them into units, one per source (records that share a download are one unit).
 3. Per unit, one agent does the work with the data in hand (download, definition, `dataset check`, the probes), a
    second agent verifies the result against

.claude/skills/knowledge-data-foundry/references/claims.md

@@ -0,0 +1,98 @@
+# Claims: who works on which dataset
+
+Several curators, each running several agents, work on the same records and folders, and git shows their work only
+when it reaches `main`. A claim says "I am working on this dataset now", and every curator and agent sees it within
+seconds. A claim covers one dataset: its record `curation/records/<name>.md` and its folder
+`datasets/_dev/tabarena-v0pt2/<name>/` with `REVIEW.md`. Shared files (CHANGELOG, the README table, the vocabulary)
+are never claimed; their conflicts are resolved when merging.
+
+## How it works
+
+* **The board** is the branch `claims` of the repository's remote. It holds one file `claims/<name>.yaml` per claimed
+  dataset (holder, stage, since, until, note) and a `README.md` table. That table is the team's view on GitHub:
+  <https://github.com/TabArena/data-foundry/blob/claims/README.md>.
+* **A claim is a commit pushed straight to that branch**, with no pull request and no CI. Two claims of the same
+  dataset at the same moment cannot both land: git rejects the second push, and the command fetches again, sees the
+  claim and skips the dataset. Nothing touches your working tree or your branch.
+* **Login is the repository's.** Whoever can push to the repository can claim, and whoever can read it sees the
+  board. Nothing else to sign up for or host. The code (`src/data_foundry/curation/claims.py`) works with any git
+  remote: `$DATA_FOUNDRY_CLAIMS_REMOTE` (default `origin`) and `$DATA_FOUNDRY_CLAIMS_BRANCH` (default `claims`).
+* **Stages:**
+  * `triage`: screening, a check, or settling an open verdict;
+  * `add`: building the v2 folder;
+  * `review`: the curator's review of an added dataset.
+* **Expiry.** A claim lasts 7 days unless renewed. After that it shows as expired, and anyone may take it over with
+  `--steal`; the board's log names the previous holder.
+* **Claims are advisory.** Nothing refuses a write. Warnings appear in three places:
+  * `save_record` warns (`ClaimWarning`) when it writes a record another curator holds;
+  * the Claude Code hook warns before an agent edits such a file;
+  * the dashboard shows the holder.
+
+  A warning means: stop and ask the curator, or the holder.
+* **The records mirror the board.** `claimed_by` and the WIP value (`WIP (Triage)` for triage or an open verdict,
+  `WIP (DF)` for work on an accepted dataset) are written by `claims claim` and `claims release`. They travel with the
+  work branch and reach `main` with the work. The board is the truth: the old rule of pushing the claim to `main` in
+  its own PR is gone.
+
+## Setup, once per curator
+
+```bash
+git config data-foundry.curator <Name>     # a claimed_by value of curation/vocabularies.yaml
+.venv/bin/python -m data_foundry.curation.cli claims list
+```
+
+`claims list` works when `git fetch` from the remote works. On a machine whose ssh agent went stale, prefix the
+commands with `env SSH_AUTH_SOCK=`. The git calls never prompt: a missing credential fails at once with a hint.
+
+## Commands
+
+`data-foundry-curation claims ...` (or `.venv/bin/python -m data_foundry.curation.cli claims ...`):
+
+| Command | What it does |
+|---|---|
+| `list [--mine\|--others] [--json] [--offline]` | Fetches and shows the board. |
+| `check <names>` | Says per dataset: free, yours, or held by someone (always exit 0). |
+| `claim <names> [--stage S] [--note TEXT] [--ttl DAYS] [--steal]` | Claims what is free and skips the rest. Exit 1 when anything was skipped. Stage default: `add` for an accepted record, else `triage`. A name with no record is refused unless `--allow-unknown`. Claiming your own claim again renews it. |
+| `release <names> [--force]` | Releases your claims; `--force` releases someone else's (logged). |
+| `renew [<names>] [--ttl DAYS]` | Extends your claims (all of them by default). |
+| `sync [--check]` | Makes every record's `claimed_by` and WIP value match the board. |
+| `import-records` | One-time: moves the claims the records carry onto the board. |
+| `init [--pages-dispatch]` | Creates the branch. `--pages-dispatch` adds `.github/workflows/claims-to-pages.yaml` to it, so each claim rebuilds the public dashboard. |
+
+From Python: `claims.claim(...)`, `claims.release(...)`, `claims.load(max_age=30)`, `claims.cached_board()`.
+
+## In the workflows
+
+* **Before working on records** (screening, adding, a re-check, a review round), run `claims claim <names> --note
+  "<what>"` once for the whole batch. Work only on the names it claimed. A skipped name belongs to someone else: leave
+  it out and report it.
+* **For a long job**, `claims renew`.
+* **When the work reaches `main`, or is dropped,** `claims release <names>`.
+* **Batch filters** use the board (`claims list --json`), not the record's `claimed_by`, which can lag behind.
+* **Hand edits:** the dashboard's Claimed by column shows the board. Pick your own name in a cell to claim, empty it to
+  release. The 🔒 count in the bar says when the board was fetched.
+
+## The Claude Code hook
+
+`.claude/settings.json` runs `.claude/hooks/claims_hook.py` with the repository's `.venv` at two points:
+
+* **At session start**, it adds the board to the agent's context: the other curators' claims, yours, and yours that
+  expire within 2 days.
+* **Before each Edit, Write or NotebookEdit** of a claimed dataset's record or folder, it shows a warning when another
+  curator holds the dataset.
+
+It never allows or denies a tool call. It re-fetches the board at most once a minute, and it does nothing without the
+package, a curator name or the network.
+
+## The public dashboard
+
+The GitHub Pages workflow fetches the board before it builds, so the read-only dashboard's Claimed by column shows
+the claims. With `claims init --pages-dispatch`, each push to the board dispatches the Pages workflow on `main`, and
+the page follows within a minute or two. Without it, the page updates with the next push to `main`.
+
+## When something is off
+
+* **A record says `claimed_by` but the board does not** (or the reverse): `claims sync --check` lists the drift and
+  `claims sync` fixes the records.
+* **"The claims branch kept moving"**: five pushes in a row lost a race. Run the command again.
+* **The board's log** shows every claim and release: `git log origin/claims`.