diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 12b08a29..468169fd 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,5 +1,13 @@ # Contributing To Trace +## Community + +Use [GitHub Issues](https://github.com/maveryn/trace/issues) for concrete, +scoped work and reproducible bug reports. Use +[GitHub Discussions](https://github.com/maveryn/trace/discussions) for +questions, integration ideas, and reproduction reports that are not yet a +specific change request. + ## Development Setup ```bash diff --git a/README.md b/README.md index c8851fd7..240e2c64 100644 --- a/README.md +++ b/README.md @@ -7,14 +7,15 @@ post-training. Each example includes an image, prompt, typed answer, image-space annotation, verifier metadata, and execution trace. The registry contains 1,000 tasks across 277 scenes and 11 visual domains. +Training Qwen2.5-VL-3B and 7B on 64,000 Trace instances improved the +macro-average across 24 external benchmarks by **+3.51** and **+4.06 percentage +points**, respectively. [See the transfer results](#external-benchmark-transfer). +

- Project Page Live Demo - arXiv - Hugging Face Collection + Paper Dataset - 3B Model - 7B Model + Reproduce

![Trace examples across charts, games, geometry, graphs, icons, illustrations, pages, physics, puzzles, symbolic reasoning, and 3D scenes](docs/assets/paper-domain-montage/trace-paper-domain-montage.png) @@ -101,6 +102,9 @@ The table reports all 24 external benchmarks for the matched 3B and 7B comparisons. Parenthesized values are mean seed-paired TRACE-minus-Base changes in percentage points. +
+View all 24 benchmark results + | Benchmark | 3B Base | 3B TRACE (Δ) | 7B Base | 7B TRACE (Δ) | | --- | ---: | ---: | ---: | ---: | @@ -136,12 +140,18 @@ in percentage points. | MME-Reasoning | 22.62 ± 1.81 | 25.06 ± 1.53 (+2.44) | 25.17 ± 1.18 | 28.03 ± 0.89 (+2.86) | +
+ The [research page](docs/research/README.md#external-benchmark-results) includes the full eight-model comparison. Machine-readable per-seed scores are available in [`results.json`](https://github.com/maveryn/trace/blob/rlvr/rlvr/evaluation/trace_eval/results.json). ## Released Models +The [Trace collection](https://huggingface.co/collections/maveryn/trace-6a604291b4be4ed6399b9f24) +groups the interactive Space, dataset, checkpoints, paper, and evaluation +artifacts in one place. + | Released checkpoint | Base model | | --- | --- | | [TRACE Qwen2.5-VL-3B](https://huggingface.co/maveryn/trace-qwen2.5-vl-3b) | [`Qwen2.5-VL-3B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct) | @@ -344,7 +354,10 @@ attributions are listed in ## Contributing See [CONTRIBUTING.md](CONTRIBUTING.md) for setup, testing, review, and pull -request guidance. +request guidance. Browse [open issues](https://github.com/maveryn/trace/issues) +for scoped work, or use +[Discussions](https://github.com/maveryn/trace/discussions) for questions, +integration ideas, and reproduction reports. ## Licensing diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md index d4b9ed61..1ab6cae7 100644 --- a/docs/DEVELOPMENT.md +++ b/docs/DEVELOPMENT.md @@ -1,5 +1,13 @@ # Development +## Community + +Use [GitHub Issues](https://github.com/maveryn/trace/issues) for concrete, +scoped work and reproducible bug reports. Use +[GitHub Discussions](https://github.com/maveryn/trace/discussions) for +questions, integration ideas, and reproduction reports that are not yet a +specific change request. + ## Source Layout - `src/trace_tasks/core/`: generation, identity, validation, trace, reward, and export infrastructure diff --git a/docs/README.md b/docs/README.md index 209f0552..fb9c9629 100644 --- a/docs/README.md +++ b/docs/README.md @@ -26,15 +26,19 @@ hide: trace.

+

+ Training Qwen2.5-VL-3B and 7B on 64,000 Trace instances improved the + macro-average across 24 external benchmarks by +3.51 and + +4.06 percentage points, respectively. + See the transfer results. +

+
Live Demo Code Paper - Research - Collection Dataset - 3B Model - 7B Model + Reproduce
@@ -201,6 +205,9 @@ The table reports all 24 external benchmarks for the matched 3B and 7B comparisons. Parenthesized values are mean seed-paired TRACE-minus-Base changes in percentage points. +
+View all 24 benchmark results + | Benchmark | 3B Base | 3B TRACE (Δ) | 7B Base | 7B TRACE (Δ) | | --- | ---: | ---: | ---: | ---: | @@ -236,11 +243,17 @@ in percentage points. | MME-Reasoning | 22.62 ± 1.81 | 25.06 ± 1.53 (+2.44) | 25.17 ± 1.18 | 28.03 ± 0.89 (+2.86) | +
+ The [research page](research/README.md#external-benchmark-results) includes the full eight-model comparison and links to the evaluation artifacts. ## Released models +The [Trace collection](https://huggingface.co/collections/maveryn/trace-6a604291b4be4ed6399b9f24) +groups the interactive Space, dataset, checkpoints, paper, and evaluation +artifacts in one place. + | Released checkpoint | Base model | | --- | --- | | [TRACE Qwen2.5-VL-3B](https://huggingface.co/maveryn/trace-qwen2.5-vl-3b) | [Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct) | diff --git a/docs/assets/brand/README.md b/docs/assets/brand/README.md index d0a7a3e7..93eb19e7 100644 --- a/docs/assets/brand/README.md +++ b/docs/assets/brand/README.md @@ -19,6 +19,9 @@ scene families. documentation headers. - `trace-mark-light.png`: light-theme raster used by static documentation composites. +- `../social/trace-social-preview.png`: the `1280x640` solid-background + repository and website social preview, generated by + `scripts/generate_social_preview.py`. The SVG files are transparent and self-contained. Their structural ink adapts to the viewer's light or dark color preference. The standalone mark has no font diff --git a/docs/assets/social/trace-social-preview.png b/docs/assets/social/trace-social-preview.png new file mode 100644 index 00000000..0161995a Binary files /dev/null and b/docs/assets/social/trace-social-preview.png differ diff --git a/docs/overrides/main.html b/docs/overrides/main.html index 4dca946a..a058996c 100644 --- a/docs/overrides/main.html +++ b/docs/overrides/main.html @@ -10,7 +10,7 @@ {% set social_title = config.site_name %} {% endif %} {% set social_description = page.meta.description if page and page.meta and page.meta.description else config.site_description %} - {% set social_image = "https://maveryn.github.io/trace/assets/paper-domain-montage/trace-paper-domain-montage.png" %} + {% set social_image = "https://maveryn.github.io/trace/assets/social/trace-social-preview.png" %} @@ -19,12 +19,12 @@ {% endif %} - - - + + + - + {% endblock %} diff --git a/docs/stylesheets/extra.css b/docs/stylesheets/extra.css index 95392a96..1b882968 100644 --- a/docs/stylesheets/extra.css +++ b/docs/stylesheets/extra.css @@ -115,6 +115,18 @@ pre { max-width: 48rem; } +.trace-hero__evidence { + color: var(--trace-muted); + font-size: 0.84rem; + line-height: 1.6; + margin: 1.1rem auto 0; + max-width: 46rem; +} + +.trace-hero__evidence strong { + color: var(--trace-teal-text); +} + .trace-resource-links { align-items: center; display: flex; diff --git a/scripts/README.md b/scripts/README.md index 7a3dcd22..13431c37 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -36,6 +36,9 @@ The tools are: - `generate_release_gallery.py` regenerates the deterministic 11-domain gallery under `docs/assets/gallery/`, its hero montage, and the manifest at `docs/gallery/manifest.v1.json`. +- `generate_social_preview.py` composes the `1280x640` repository and website + social preview from the committed Trace mark, domain montage, and verified + headline results. Scripts added here must use the `trace_tasks` namespace, accept repository-relative or user-supplied paths, avoid credentials and uploads, and diff --git a/scripts/check_docs_site.py b/scripts/check_docs_site.py index 740a99ed..abfe8718 100644 --- a/scripts/check_docs_site.py +++ b/scripts/check_docs_site.py @@ -15,10 +15,10 @@ SITE_URL = "https://maveryn.github.io/trace/" SITE_PATH_PREFIX = "/trace" SOCIAL_IMAGE_URL = ( - f"{SITE_URL}assets/paper-domain-montage/trace-paper-domain-montage.png" + f"{SITE_URL}assets/social/trace-social-preview.png" ) -SOCIAL_IMAGE_PATH = Path("assets/paper-domain-montage/trace-paper-domain-montage.png") -SOCIAL_IMAGE_SIZE = (1512, 780) +SOCIAL_IMAGE_PATH = Path("assets/social/trace-social-preview.png") +SOCIAL_IMAGE_SIZE = (1280, 640) REQUIRED_PAGES = {Path("index.html"), Path("research/index.html")} NONCANONICAL_PAGES = {Path("404.html")} _SITE_PARTS = urlsplit(SITE_URL) diff --git a/scripts/generate_social_preview.py b/scripts/generate_social_preview.py new file mode 100644 index 00000000..ce80ea2a --- /dev/null +++ b/scripts/generate_social_preview.py @@ -0,0 +1,192 @@ +#!/usr/bin/env python3 +"""Generate the branded Trace social-preview image from committed assets.""" + +from __future__ import annotations + +import argparse +import hashlib +from pathlib import Path + +from PIL import Image, ImageDraw, ImageFont, ImageOps + +REPO_ROOT = Path(__file__).resolve().parents[1] +MONTAGE_PATH = ( + REPO_ROOT + / "docs" + / "assets" + / "paper-domain-montage" + / "trace-paper-domain-montage.png" +) +MARK_PATH = REPO_ROOT / "docs" / "assets" / "brand" / "trace-mark-light.png" +FONT_PATH = ( + REPO_ROOT + / "src" + / "trace_tasks" + / "resources" + / "assets" + / "fonts" + / "google_fonts" + / "noto_sans" + / "NotoSans[wdth,wght].ttf" +) +OUTPUT_PATH = ( + REPO_ROOT / "docs" / "assets" / "social" / "trace-social-preview.png" +) + +CANVAS_SIZE = (1280, 640) +BACKGROUND = "#102a3b" +SURFACE = "#f7f9f9" +INK = "#17242d" +MUTED = "#667983" +WHITE = "#ffffff" +BLUE = "#3974d7" +TEAL = "#159b91" +CORAL = "#e85d4a" +GOLD = "#e5a62b" + + +def _font(size: int, variation: str) -> ImageFont.FreeTypeFont: + font = ImageFont.truetype(FONT_PATH, size=size) + font.set_variation_by_name(variation) + return font + + +def _rounded_image( + image: Image.Image, + size: tuple[int, int], + radius: int, +) -> tuple[Image.Image, Image.Image]: + fitted = ImageOps.fit( + image.convert("RGB"), + size, + method=Image.Resampling.LANCZOS, + centering=(0.5, 0.5), + ) + mask = Image.new("L", size, 0) + ImageDraw.Draw(mask).rounded_rectangle( + (0, 0, size[0] - 1, size[1] - 1), + radius=radius, + fill=255, + ) + return fitted, mask + + +def generate(output_path: Path = OUTPUT_PATH) -> Path: + """Render and save the social preview.""" + + canvas = Image.new("RGB", CANVAS_SIZE, BACKGROUND) + draw = ImageDraw.Draw(canvas) + + draw.ellipse((-180, -240, 340, 280), fill="#163c54") + draw.ellipse((1030, 430, 1390, 790), fill="#123f49") + draw.rounded_rectangle((610, 44, 1224, 596), radius=30, fill=SURFACE) + + with Image.open(MARK_PATH) as mark_source: + mark = mark_source.convert("RGBA").resize( + (70, 70), + resample=Image.Resampling.LANCZOS, + ) + draw.rounded_rectangle((58, 48, 142, 132), radius=18, fill=WHITE) + canvas.paste(mark, (65, 55), mark) + draw.text((162, 54), "Trace", font=_font(54, "Bold"), fill=WHITE) + + draw.text( + (62, 164), + "VERIFIABLE VISUAL REASONING", + font=_font(19, "SemiBold"), + fill="#7fc6ed", + spacing=4, + ) + draw.multiline_text( + (58, 204), + "Grounded tasks\nfor post-training", + font=_font(55, "Bold"), + fill=WHITE, + spacing=0, + ) + draw.multiline_text( + (62, 354), + "1,000 deterministic tasks across\n277 scenes and 11 visual domains", + font=_font(25, "Regular"), + fill="#d9e6eb", + spacing=8, + ) + + draw.rounded_rectangle((58, 458, 548, 572), radius=20, fill="#183c50") + draw.text( + (82, 474), + "+3.51 / +4.06 pp", + font=_font(34, "Bold"), + fill="#63d4c8", + ) + draw.text( + (82, 521), + "macro-average transfer across 24 external benchmarks", + font=_font(16, "Medium"), + fill="#d9e6eb", + ) + + with Image.open(MONTAGE_PATH) as montage_source: + montage, montage_mask = _rounded_image( + montage_source, + size=(566, 292), + radius=18, + ) + canvas.paste(montage, (634, 68), montage_mask) + + draw.text( + (638, 390), + "EVERY INSTANCE INCLUDES", + font=_font(16, "SemiBold"), + fill=MUTED, + ) + features = ( + (BLUE, "typed answer"), + (TEAL, "verifier metadata"), + (CORAL, "replayable execution trace"), + ) + y = 431 + for color, label in features: + draw.rounded_rectangle((638, y + 3, 658, y + 23), radius=5, fill=color) + draw.text((674, y - 2), label, font=_font(21, "Medium"), fill=INK) + y += 45 + + draw.rounded_rectangle((1116, 550, 1174, 556), radius=3, fill=GOLD) + draw.text( + (638, 548), + "trace · deterministic · metadata-grounded", + font=_font(15, "Medium"), + fill=MUTED, + ) + + output_path.parent.mkdir(parents=True, exist_ok=True) + canvas.save( + output_path, + format="PNG", + optimize=True, + compress_level=9, + ) + return output_path + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--output", + type=Path, + default=OUTPUT_PATH, + help="Output PNG path (default: repository social asset)", + ) + args = parser.parse_args() + + output_path = generate(args.output.resolve()) + payload = output_path.read_bytes() + print( + f"{output_path}: {CANVAS_SIZE[0]}x{CANVAS_SIZE[1]}, " + f"{len(payload)} bytes, sha256={hashlib.sha256(payload).hexdigest()}" + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_docs_site.py b/tests/test_docs_site.py index 6f7a6c31..5e1abd5e 100644 --- a/tests/test_docs_site.py +++ b/tests/test_docs_site.py @@ -55,6 +55,7 @@ ) PIPELINE_ASSET = REPO_ROOT / "docs/assets/paper-instance-pipeline/trace-instance-pipeline.png" INFERENCE_EXAMPLE_ASSET = REPO_ROOT / "docs/assets/examples/trace-match3-validation-example.png" +SOCIAL_PREVIEW_ASSET = REPO_ROOT / "docs/assets/social/trace-social-preview.png" def _metadata(canonical_url: str) -> str: @@ -68,8 +69,8 @@ def _metadata(canonical_url: str) -> str: - - + + @@ -116,7 +117,7 @@ def _write_valid_site(site_root: Path) -> None: Overview Research - + """, encoding="utf-8", @@ -237,7 +238,7 @@ def test_check_docs_site_rejects_wrong_social_image_dimensions( image = site_root / check_docs_site.SOCIAL_IMAGE_PATH image.write_bytes(_png_header(1200, 630)) - with pytest.raises(check_docs_site.DocsSiteError, match="must be 1512 x 780"): + with pytest.raises(check_docs_site.DocsSiteError, match="must be 1280 x 640"): check_docs_site.check_docs_site(site_root) @@ -311,6 +312,13 @@ def test_docs_homepage_uses_a_balanced_hero_wordmark() -> None: assert 'class="trace-hero__mark" src="assets/brand/trace-mark.svg"' in text +def test_social_preview_matches_the_public_image_contract() -> None: + data = SOCIAL_PREVIEW_ASSET.read_bytes() + assert data.startswith(b"\x89PNG\r\n\x1a\n") + assert struct.unpack(">II", data[16:24]) == check_docs_site.SOCIAL_IMAGE_SIZE + assert SOCIAL_PREVIEW_ASSET.stat().st_size < 1_000_000 + + def test_readme_inference_uses_a_trace_validation_example() -> None: assert INFERENCE_EXAMPLE_ASSET.read_bytes().startswith(b"\x89PNG\r\n\x1a\n") text = (REPO_ROOT / "README.md").read_text(encoding="utf-8") diff --git a/tests/test_visibility_surfaces.py b/tests/test_visibility_surfaces.py index bc6e2dbb..69c1a984 100644 --- a/tests/test_visibility_surfaces.py +++ b/tests/test_visibility_surfaces.py @@ -12,6 +12,11 @@ "examples/notebooks/trace_quickstart.ipynb" ) PINNED_REVISION = "bb7fdd1fc8a0f8a2e3db7efe910a14e81d58feb7" +TRANSFER_CLAIM = ( + "macro-average across 24 external benchmarks by **+3.51** and **+4.06 " + "percentage points**" +) +BENCHMARK_EXPANDER = "View all 24 benchmark results" def test_landing_quickstarts_follow_the_domain_montage() -> None: @@ -61,3 +66,51 @@ def test_landing_quickstarts_link_all_runnable_surfaces() -> None: def test_landing_quickstart_screenshot_is_a_png() -> None: assert SCREENSHOT.read_bytes().startswith(b"\x89PNG\r\n\x1a\n") + + +def test_readme_leads_with_the_verified_transfer_result() -> None: + text = (REPO_ROOT / "README.md").read_text(encoding="utf-8") + normalized = " ".join(text.split()) + assert TRANSFER_CLAIM in normalized + assert normalized.index(TRANSFER_CLAIM) < normalized.index( + "docs/assets/paper-domain-montage/trace-paper-domain-montage.png" + ) + + +def test_landing_pages_collapse_the_full_benchmark_table() -> None: + for relative_path in ("README.md", "docs/README.md"): + text = (REPO_ROOT / relative_path).read_text(encoding="utf-8") + assert text.count(BENCHMARK_EXPANDER) == 1 + assert text.index(BENCHMARK_EXPANDER) < text.index( + "" + ) + assert text.index("") < text.index( + "", + text.index(""), + ) + + +def test_primary_resource_links_stay_focused() -> None: + readme = (REPO_ROOT / "README.md").read_text(encoding="utf-8") + badge_block = readme.split('

', 1)[1].split("

", 1)[0] + assert badge_block.count("', 1)[1].split( + "", + 1, + )[0] + assert hero_links.count('class="md-button') == 5 + for label in ("Live Demo", "Code", "Paper", "Dataset", "Reproduce"): + assert f">{label}" in hero_links + + +def test_public_contribution_surfaces_are_linked() -> None: + for relative_path in ("README.md", "CONTRIBUTING.md", "docs/DEVELOPMENT.md"): + text = (REPO_ROOT / relative_path).read_text(encoding="utf-8") + assert "https://github.com/maveryn/trace/issues" in text + assert "https://github.com/maveryn/trace/discussions" in text