Skip to content

Simplify results and installation #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
maveryn merged 1 commit into from
Jul 27, 2026
Merged

Simplify results and installation #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 8 additions & 36 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ points**, respectively. [See the transfer results](#external-benchmark-transfer)
<p align="center">
<a href="https://huggingface.co/spaces/maveryn/trace"><img alt="Live Demo" src="https://img.shields.io/badge/HuggingFace-Live_Demo-2563eb?style=for-the-badge"></a>
<a href="https://arxiv.org/abs/2607.19790"><img alt="Paper" src="https://img.shields.io/badge/arXiv-Paper-b31b1b?style=for-the-badge"></a>
<a href="https://huggingface.co/collections/maveryn/trace-6a604291b4be4ed6399b9f24"><img alt="Hugging Face Collection" src="https://img.shields.io/badge/HuggingFace-Collection-f59e0b?style=for-the-badge"></a>
<a href="https://huggingface.co/datasets/maveryn/trace"><img alt="Dataset" src="https://img.shields.io/badge/HuggingFace-Dataset-f59e0b?style=for-the-badge"></a>
<a href="https://github.com/maveryn/trace/blob/main/docs/research/REPRODUCING_RESULTS.md"><img alt="Reproduce" src="https://img.shields.io/badge/Results-Reproduce-159b91?style=for-the-badge"></a>
</p>

![Trace examples across charts, games, geometry, graphs, icons, illustrations, pages, physics, puzzles, symbolic reasoning, and 3D scenes](docs/assets/paper-domain-montage/trace-paper-domain-montage.png)
Expand Down Expand Up @@ -70,19 +70,6 @@ aligned. Each finalized record contains the rendered problem, exact scoring
contract, image-space annotation, and an execution-trace reference for
inspection and replay.

## TRACE Validation

On 2,000 previously unseen instances generated from the same 1,000 task
programs, TRACE training improves accuracy at both model scales.

| Model scale | Base | TRACE | Change |
| --- | ---: | ---: | ---: |
| Qwen2.5-VL-3B | 24.45 | 41.05 | +16.60 |
| Qwen2.5-VL-7B | 34.25 | 51.55 | +17.30 |

These results measure new realizations within the TRACE task distributions;
the evaluation below measures transfer to external benchmarks.

## External Benchmark Transfer

Qwen2.5-VL models trained on 64,000 Trace instances improve the unweighted
Expand All @@ -96,6 +83,9 @@ macro-average across 24 external benchmarks at both evaluated model scales.
Values are mean ± sample standard deviation across decoding seeds 42, 43, and
44. Paired changes compare matched benchmark and seed results.

On 2,000 unseen TRACE instances, accuracy improves from 24.45 to 41.05 at 3B
and from 34.25 to 51.55 at 7B.

### Base versus TRACE by benchmark

The table reports all 24 external benchmarks for the matched 3B and 7B
Expand Down Expand Up @@ -202,30 +192,12 @@ for training, evaluation, progress reporting, and validation.
```bash
git clone https://github.com/maveryn/trace.git
cd trace
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip setuptools wheel
python -m pip install -e ".[test]"
python -m pip install -e .
```

Trace supports Python 3.10 through 3.14. Use the reproducibility constraints on
Python 3.10-3.12 when generating datasets or documentation assets:

```bash
python -m pip install -c constraints/release.txt -e ".[test]"
```

For Python 3.14 package and CLI development, use:

```bash
python -m pip install -c constraints/compat-py314.txt -e ".[test]"
```

Install Parquet and Hugging Face export support with:

```bash
python -m pip install -e ".[test,export]"
```
See the [installation documentation](docs/QUICKSTART.md#install) for
reproducibility constraints, export dependencies, and Python 3.14
compatibility.

## Generate

Expand Down
27 changes: 24 additions & 3 deletions docs/QUICKSTART.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,34 @@

## Install

From a clone of the repository:
For a standard installation from a clone:

```bash
git clone https://github.com/maveryn/trace.git
cd trace
python -m pip install -e .
```

For development and reproducible generation on Python 3.10-3.12, create and
activate a virtual environment, then install the release constraints:

```bash
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip setuptools wheel
python -m pip install -e ".[test]"
python -m pip install -c constraints/release.txt -e ".[test]"
```

Use the compatibility constraints for package and CLI development on Python
3.14:

```bash
python -m pip install -c constraints/compat-py314.txt -e ".[test]"
```

Install Parquet and Hugging Face export support with:

```bash
python -m pip install -e ".[export]"
```

## Inspect The Registry
Expand Down
35 changes: 4 additions & 31 deletions docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,8 @@ hide:
<a class="md-button md-button--primary" href="https://huggingface.co/spaces/maveryn/trace">Live Demo</a>
<a class="md-button" href="https://github.com/maveryn/trace">Code</a>
<a class="md-button" href="https://arxiv.org/abs/2607.19790">Paper</a>
<a class="md-button" href="https://huggingface.co/collections/maveryn/trace-6a604291b4be4ed6399b9f24">Collection</a>
<a class="md-button" href="https://huggingface.co/datasets/maveryn/trace">Dataset</a>
<a class="md-button" href="research/REPRODUCING_RESULTS/">Reproduce</a>
</div>
</div>

Expand Down Expand Up @@ -141,36 +141,6 @@ aligned. Each finalized record contains the rendered problem, exact scoring
contract, image-space annotation, and an execution-trace reference for
inspection and replay.

## TRACE validation

The released checkpoints improve accuracy on 2,000 previously unseen
instances generated from the same 1,000 task programs. Each model is evaluated
once with decoding seed 42.

<div class="trace-result-grid">
<div class="trace-result">
<span class="trace-result__model">Qwen2.5-VL-3B</span>
<div class="trace-result__scores">
<span><span class="trace-result__label">Base</span>24.45</span>
<span class="trace-result__arrow" aria-hidden="true">→</span>
<span><span class="trace-result__label">Trace</span>41.05</span>
</div>
<span class="trace-result__delta">Improvement: +16.60</span>
</div>
<div class="trace-result">
<span class="trace-result__model">Qwen2.5-VL-7B</span>
<div class="trace-result__scores">
<span><span class="trace-result__label">Base</span>34.25</span>
<span class="trace-result__arrow" aria-hidden="true">→</span>
<span><span class="trace-result__label">Trace</span>51.55</span>
</div>
<span class="trace-result__delta">Improvement: +17.30</span>
</div>
</div>

These results measure new realizations within the TRACE task distributions.
The evaluation below measures transfer to external benchmarks.

## External benchmark transfer

Qwen2.5-VL models trained on 64,000 Trace instances improve the macro-average
Expand Down Expand Up @@ -199,6 +169,9 @@ compare matched seeds.
</div>
</div>

On 2,000 unseen TRACE instances, accuracy improves from 24.45 to 41.05 at 3B
and from 34.25 to 51.55 at 7B.

### Base versus TRACE by benchmark

The table reports all 24 external benchmarks for the matched 3B and 7B
Expand Down
46 changes: 43 additions & 3 deletions tests/test_visibility_surfaces.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,10 @@
"percentage points**"
)
BENCHMARK_EXPANDER = "View all 24 benchmark results"
TRACE_VALIDATION_SUMMARY = (
"On 2,000 unseen TRACE instances, accuracy improves from 24.45 to 41.05 "
"at 3B and from 34.25 to 51.55 at 7B."
)


def test_landing_quickstarts_follow_the_domain_montage() -> None:
Expand Down Expand Up @@ -94,9 +98,9 @@ def test_primary_resource_links_stay_focused() -> None:
readme = (REPO_ROOT / "README.md").read_text(encoding="utf-8")
badge_block = readme.split('<p align="center">', 1)[1].split("</p>", 1)[0]
assert badge_block.count("<a href=") == 4
for label in ("Live Demo", "Paper", "Dataset", "Reproduce"):
for label in ("Live Demo", "Paper", "Hugging Face Collection", "Dataset"):
assert f'alt="{label}"' in badge_block
for label in ("Project Page", "Hugging Face Collection", "3B Model", "7B Model"):
for label in ("Project Page", "Reproduce", "3B Model", "7B Model"):
assert f'alt="{label}"' not in badge_block

docs = (REPO_ROOT / "docs/README.md").read_text(encoding="utf-8")
Expand All @@ -105,8 +109,44 @@ def test_primary_resource_links_stay_focused() -> None:
1,
)[0]
assert hero_links.count('class="md-button') == 5
for label in ("Live Demo", "Code", "Paper", "Dataset", "Reproduce"):
for label in ("Live Demo", "Code", "Paper", "Collection", "Dataset"):
assert f">{label}</a>" in hero_links
assert ">Reproduce</a>" not in hero_links


def test_landing_pages_keep_external_transfer_as_the_single_results_section() -> None:
pages = {
"README.md": "## External Benchmark Transfer",
"docs/README.md": "## External benchmark transfer",
}
for relative_path, transfer_heading in pages.items():
text = (REPO_ROOT / relative_path).read_text(encoding="utf-8")
normalized = " ".join(text.split())
assert "## TRACE Validation" not in text
assert "## TRACE validation" not in text
assert normalized.count(TRACE_VALIDATION_SUMMARY) == 1
assert text.index(transfer_heading) < text.index(
"<!-- trace-eval-v1-base-trace-table:start -->"
)
assert "Improvement: +16.60" not in text


def test_readme_keeps_installation_brief_and_links_the_full_guide() -> None:
readme = (REPO_ROOT / "README.md").read_text(encoding="utf-8")
section = readme.split("## Installation", 1)[1].split("## Generate", 1)[0]
assert section.count("```bash") == 1
assert "python -m pip install -e ." in section
assert "docs/QUICKSTART.md#install" in section
for detail in (".venv", "constraints/release.txt", "compat-py314.txt", "[export]"):
assert detail not in section

guide = (REPO_ROOT / "docs/QUICKSTART.md").read_text(encoding="utf-8")
for detail in (
"constraints/release.txt",
"constraints/compat-py314.txt",
'".[export]"',
):
assert detail in guide


def test_public_contribution_surfaces_are_linked() -> None:
Expand Down
Loading