Persona2Web Evaluation Package

An integrity-checked dataset snapshot and dependency-free evaluator for personalization-inference experiments

This repository packages the public Persona2Web data in an evaluation-ready layout and provides a Python standard-library evaluator for structured model outputs. It is designed for reproducible offline scoring: dataset provenance is pinned to an upstream commit, every redistributed file has a SHA-256 entry, and evaluation behavior is testable without browser automation or API keys.

Engineering highlights

• Pinned provenance: dataset/SOURCE.json records the upstream dataset, paper, project, code repository, and exact snapshot commit.
• Integrity manifest: SHA-256 hashes cover the ground truth, all personalization-level queries, and every user-history file.
• Dataset validation: the checker verifies 150 unique base tasks, 450 level-specific queries, 50 matching user histories, and nonempty memory banks.
• Flexible result ingestion: predictions may use answer, prediction, output, or response, and embedded JSON strings are recovered when possible.
• Retry-aware scoring: when a task appears more than once, the latest attempt is evaluated and duplicate attempts are reported.
• Level-specific reporting: metrics are emitted overall and separately for personalization levels 0, 1, and 2.
• Explicit metric boundary: proxy inference metrics are separated from the upstream paper's browser-trajectory metrics.

flowchart LR
    A[Upstream pinned snapshot] --> B[SHA-256 manifest]
    B --> C[Dataset validator]
    D[Model result JSONL] --> E[Schema normalization]
    E --> F[Latest-attempt selection]
    C --> G[Ground-truth join]
    F --> G
    G --> H[Website exact match]
    G --> I[Preference recall]
    H --> J[Overall and per-level report]
    I --> J

Loading

Dataset layout

dataset/
├── SOURCE.json
├── MANIFEST.sha256
├── UPSTREAM_README.md
├── data/
│   ├── ground_truth.jsonl
│   ├── query_personalization_0.jsonl
│   ├── query_personalization_1.jsonl
│   └── query_personalization_2.jsonl
└── user_history/                 # 50 per-user JSON files
scripts/
├── evaluate_persona2web.py
└── validate_dataset.py
examples/results.sample.jsonl
tests/test_evaluate.py

The snapshot contains 150 base tasks, three personalization levels with 450 total queries, and histories for 50 users.

Quick start

No third-party Python package is required.

python3 scripts/validate_dataset.py
python3 -m unittest discover -s tests -v

python3 scripts/evaluate_persona2web.py \
  --results examples/results.sample.jsonl \
  --output-json metrics.json \
  --output-md metrics.md

For a full submission, provide one JSON object per line. Ground truth is joined automatically from the pinned snapshot.

Accepted result format

Minimum row:

{
  "task_id": "Alex_Garcia_A0",
  "answer": {
    "inferred_website": "kohls",
    "inferred_preferences": {
      "size": "32x32",
      "ratings": "visible ratings 4.2+ stars with 50+ reviews"
    }
  }
}

Behavior:

• task_id must match one of the 450 query IDs.
• answer may be an object or a string containing a JSON object.
• prediction, output, or response may replace answer.
• ok, base_task_id, personalization_level, and embedded ground_truth are optional.
• a row with ok=false or an error field scores zero;
• repeated tasks use their latest row, which supports resumed experiment logs.

Metrics

Report field	Definition
Pweb_like	normalized exact match for the inferred website
Ppref_like	macro average of per-task preference-value recall
Pavg_like	mean of website exact match and preference recall
full_match	website and all preference values match

These are offline personalization-inference proxies. They are not the Persona2Web paper's original browser-trajectory Intent/SR metrics because this package does not execute browser actions.

Data source and reuse boundary

The dataset is redistributed from yonsei-dli/Persona2Web at commit c7f804a61883157941ca9f9db9745a5862c6326c.

• Paper
• Project page
• Upstream code

The upstream snapshot does not include a standalone license file. Dataset use and redistribution remain subject to the upstream authors' terms. See dataset/UPSTREAM_README.md for the original description.