Skip to content

CLAUDE.md

Latest commit

 

History

History
114 lines (81 loc) · 6.72 KB

CLAUDE.md

File metadata and controls

114 lines (81 loc) · 6.72 KB
Raw

CLAUDE.md

This file gives AI agents (Claude Code, Cursor, Codex, etc.) the project-specific context they need to use the GISTBench dataset and modify this codebase. Human-facing documentation lives in README.md and INSTRUCTIONS.md; this file emphasizes the rules that are easy for an agent to violate without realizing it.

What this repo is

GISTBench (Groundedness & Interest Specificity Test Bench) evaluates how well an LLM understands users from their engagement history. It scores two axes:

  • Interest Groundedness (IG) — are extracted interests supported by the user's engagement data?
  • Interest Specificity (IS) — does the model cite the specific items that support each interest?

Final score = harmonic mean of IG and IS. Full design and pipeline details are in INSTRUCTIONS.md.

Using GISTBench to evaluate a model

The 3-model rule

The oracle (denominator for IG Recall) is computed cross-model from the union of verified interests. You must run at least 3 models on the same dataset before scores are produced. Fewer models terminates with "Need 3 models to build oracle, have N. Run N more." — this is intentional, not a bug. MIN_ORACLE_MODELS = 3 in gistbench/cli.py.

To bypass the 3-model requirement, supply a pre-computed oracle via --oracle oracle.json. The bundled oracle at gistbench/assets/oracle_synthetic.json covers 997 users of the synthetic dataset.

Minimal flow

pip install -e ".[dev]"
export OPENAI_API_KEY=your-key

gistbench run -d data.csv -m gpt-4o      --results-db results.db
gistbench run -d data.csv -m gpt-4o-mini --results-db results.db
gistbench run -d data.csv -m gpt-4-turbo --results-db results.db   # scores all 3

For a one-shot pipeline check, gistbench smoke-test runs extraction → IG → IS → taxonomy → scoring against the bundled mock dataset or the real synthetic split (each ships with a bundled oracle, so a single model run suffices).

Dataset shape

GISTBench expects engagement records (CSV / JSON / JSONL) with these fields:

Field Required Notes
user_id yes anonymized identifier
object_id yes anonymized identifier
object_text yes text description of the item
interaction_type yes one of explicit_positive, implicit_positive, implicit_negative, explicit_negative
interaction_time no anonymized timestamp

Custom datasets are auto-detected — you do not need to register a config. The six built-in datasets (synthetic, kuairec, mind, amazon_digital_music, yelp, goodreads) are declared in gistbench/schema.py:DATASET_CONFIGS.

Backends

Anything OpenAI-compatible works: OpenAI, Azure OpenAI, Ollama, vLLM, LM Studio. Pass --base-url for the endpoint and any non-empty --api-key (use unused for local backends that ignore it).

Dataset license

Data is CC-BY-NC 4.0 (benchmarking only, no commercial use). The object_text field is generated by Llama 3.2 70B and is additionally subject to the Llama 3.2 license — any AI model created, trained, fine-tuned, or improved using object_text and subsequently distributed must be named with a Llama prefix.

Working on the GISTBench codebase

Layout

gistbench/
├── cli.py                   # Click CLI (entry: gistbench)
├── client.py                # LLMClient protocol + OpenAIClient
├── data.py                  # Dataset loading, chunking, sampling
├── download.py              # Hugging Face download (facebook/gistbench)
├── schema.py                # Dataclasses (Engagement, Interest, Oracle, UserScore, DATASET_CONFIGS)
├── store.py                 # SQLite ResultsStore + cross-model oracle computation
├── prompts/
│   └── interest_extraction.py
├── steps/
│   ├── interest_groundedness.py    # IG verification (Step 2)
│   ├── interest_specificity.py     # IS verification (Step 3)
│   ├── taxonomy.py                  # Map free-form interests → 325 category IDs (Step 4)
│   ├── scoring.py                   # IG_F1, IS, harmonic mean (Step 5)
│   └── pipeline.py                  # End-to-end orchestration
├── assets/
│   ├── categories.csv               # 325 interest categories (category_id, category_name)
│   ├── oracle_synthetic.json        # Bundled oracle (997 users, category IDs)
│   ├── mock_dataset.json            # 3-user mock for offline tests
│   └── mock_oracle.json
└── tests/
    ├── test_smoke.py        # pytest -m "not e2e"  (no API key)
    └── test_e2e.py          # pytest -m e2e       (real LLM, OPENAI_API_KEY)

Conventions

  • Python ≥ 3.10. Use from __future__ import annotations and PEP 604 X | Y types — match existing modules.
  • All shared dataclasses live in gistbench/schema.py. Prefer extending an existing dataclass to introducing a parallel one in another module.
  • Oracle entries are category_id integers from gistbench/assets/categories.csv, never free-form interest names. The Oracle.from_file parser will coerce strings via int().
  • Score aggregation groups interests by category_id first to prevent inflation when a model emits many interests within the same category. Don't bypass steps/taxonomy.py.
  • Use the module logger (logging.getLogger(__name__)), not print, for diagnostics.

Tests

pytest -m "not e2e"   # fast, no API key needed
pytest -m e2e -v -s   # real LLM, requires OPENAI_API_KEY

Add new tests under gistbench/tests/. Use gistbench/assets/mock_dataset.json and mock_oracle.json for offline cases. Mark anything that calls a live LLM with @pytest.mark.e2e so CI without an API key still passes.

Things not to do

  • Don't compute scores from a single model's predictions — IG_R collapses to IG_P and the result is meaningless. Honor the 3-model rule or require an explicit oracle.
  • Don't edit gistbench/assets/categories.csv or gistbench/assets/oracle_synthetic.json casually. Both are referenced by score computation; the bundled oracle covers a fixed 997-user split that downstream comparisons rely on.
  • Don't pin pandas, openai, or datasets to exact versions — the project supports a range deliberately (see pyproject.toml).
  • Don't commit .env, API keys, results.db, or any per-run output files.
  • Don't reformat the entire codebase as part of an unrelated change.

Contributing

PRs follow the standard fork → branch → PR flow described in CONTRIBUTING.md. First-time contributors must sign the Meta CLA at https://code.facebook.com/cla. Keep PRs focused: one concern per PR.