demoscenebench

demoscenebench is a synthetic audiovisual reconstruction benchmark for code models.

The task: given evidence from a deterministic procedural target clip, write the smallest program that recreates it perceptually. Lower scores are better. The benchmark rewards discovering the generative recipe — not just copying compressed media.

Targets are demoscene-inspired: oscillators, gradients, particle systems, palette tricks, beat timing, tracker-style audio. A model that understands the structure can reconstruct an 18-second clip in ~2 KB instead of the ~365 KB a compressed baseline needs, giving a ~180× score advantage.

---

Quick start

# 1. Generate the default public target set
python -m scripts.generate_public_set

# 2. Evaluate the blank (worst-case) baseline on Level 0
python -m demoscenebench.evaluate \
  --target targets/public/l0_0001 \
  --submission submissions/native_blank \
  --out runs/blank_l0

# 3. Evaluate the Track A (Python) example on Level 0
python -m demoscenebench.evaluate \
  --target targets/public/l0_0001 \
  --submission submissions/python_track_l0_example \
  --out runs/track_a_l0 \
  --runtime-profile python_track

# 4. Prepare a model run workspace
python -m scripts.prepare_model_run \
  --model-id my-model \
  --target targets/public/l1_0001 \
  --attempt attempt_01

---

Tracks

Submissions declare one of two tracks:

Track A — python_track / python_readable

Python source submissions that emit raw frames and audio. No compilers needed. The evaluation image includes Python 3 and NumPy; submissions must not import large ML or media libraries.

{
  "schema_version": 1,
  "entrypoint": "run.sh",
  "runtime_profile": "python_track",
  "track": "python_readable",
  "source_available": true,
  "description": "Compact procedural Python reconstruction."
}

run.sh should just call python3 "$(dirname "$0")/run.py". See submissions/python_track_l0_example/ for a complete reference.

Track B — native_raw / native_raw_binary

Pre-built Linux binaries, shell-only entries, or any compiled executable. No Python, no compiler calls during evaluation. Binaries must already be in the submission directory.

{
  "schema_version": 1,
  "entrypoint": "run.sh",
  "runtime_profile": "native_raw",
  "track": "native_raw_binary",
  "source_available": true,
  "description": "Compiled C reconstruction."
}

Track B rewards the most extreme byte-count minimisation — strip, compress, and pack your binary before submission. Track A is more accessible and rewards readable procedural reconstruction.

---

Target levels and scenes

Level	Scene	Description
0	pulse_circles	Glowing circles on a dark gradient background
1	starfield_tunnel, plasma_synth	Depth-sorted particles / additive plasma
2	spiral_particles, lissajous_ribbon	Rotating spirals / parametric curves
3	composite_medium	3-scene timeline: plasma, starfield, SDF raymarcher
4	composite_flagship	6-scene timeline + additive finale: SDF, lissajous, chiptune audio
5	terrain_world	Raymarched 3D landscape: fBM mountains, water, dynamic sky, lo-fi hip-hop

Levels 3 and 4 include tracker-style chiptune music synchronized to the visuals with a beat envelope. Every target publishes its RNG algorithm and seed in metadata_public.json, so you can reproduce all random draws exactly. See docs/rng_spec.md for the contract and docs/primitive_spec.md for mathematical references on SDF raymarching, plasma, lissajous, and the chiptune engine.

Generate a single target at any level:

python -m demoscenebench.generate \
  --level 3 --seed 42 --scene composite_medium \
  --out targets/public/l3_0001 \
  --duration 18 --width 256 --height 256

python -m demoscenebench.generate \
  --level 4 --seed 99 --scene composite_flagship \
  --out targets/public/l4_0001 \
  --duration 45 --width 256 --height 256

python -m demoscenebench.generate \
  --level 5 --seed 77 --scene terrain_world \
  --out targets/public/l5_0001 \
  --duration 45 --width 256 --height 256

L4 and L5 each generate in approximately 5–15 minutes at 256×256 / 30 fps (one-time cost — targets are generated once and stored). Use --skip-composite with generate_public_set to skip L3, L4, and L5 during development.

Public demo vs assessment targets

Set	Directory	Seeds	Purpose
Public demo	targets/public/	Published in README	Website showcase; models practice here
Assessment	targets/assessment/	Private; never committed	Official scoring; models have not seen these seeds

Generate assessment targets (gitignored — never commit):

python -m scripts.generate_assessment_set
python -m scripts.generate_assessment_set --skip-composite  # L0-2 only, faster

When running a benchmark, point models at targets/public/ for development and evaluate final submissions against targets/assessment/.

---

Target artifacts

Each generated target directory contains:

File	Description
target.mp4	Rendered video
target.wav	Rendered audio
target_with_audio.mp4	Combined AV preview
metadata_public.json	RNG block, dimensions, FPS, duration (visible to submissions)
metadata_private.json	Generator parameters (not exposed to submissions)
contact_sheet.png, spectrogram.png	Evidence images
frames/key_*.png, keyframes/key_*.png	Sampled frames

Submissions only see the public view (public metadata + evidence images). They do not receive target.mp4, target.wav, or metadata_private.json.

Generated media is gitignored. Re-run python -m scripts.generate_public_set after cloning.

---

Submission output format

Every submission must write to $DEMOSCENEBENCH_OUTPUT_DIR:

File	Format
output.rgb	Raw RGB24, packed — exactly frame_count × height × width × 3 bytes
output.s16	Mono little-endian signed 16-bit PCM — exactly sample_count × 2 bytes

(output.f32 float32 is also accepted in place of .s16.)

The evaluator converts raw output to output.mp4 / output.wav for inspection.

Environment variables set by the evaluator

Variable	Content
DEMOSCENEBENCH_TARGET_DIR	Public evidence directory
DEMOSCENEBENCH_OUTPUT_DIR	Where outputs must be written
DEMOSCENEBENCH_WIDTH, DEMOSCENEBENCH_HEIGHT	Frame dimensions
DEMOSCENEBENCH_FPS, DEMOSCENEBENCH_FRAME_COUNT	Frame rate and total frames
DEMOSCENEBENCH_SAMPLE_RATE, DEMOSCENEBENCH_SAMPLE_COUNT	Audio parameters
DEMOSCENEBENCH_DURATION	Clip duration in seconds

---

Scoring

quality_loss = 0.5 × video_loss + 0.5 × audio_loss

total_score  = 10 × compressed_bytes
             + 100000 × quality_loss

Runtime is recorded in score.json but not included in total_score — it is host-dependent and cannot be fairly compared across different machines.

video_loss and audio_loss are baseline-normalized structural losses. Exact reconstruction ≈ 0; degenerate baselines (silence, black frames) ≈ 1.

Quality gate

A submission is leaderboard-eligible only if:

• quality_loss ≤ 0.75
• video_loss ≤ 0.95
• audio_loss ≤ 0.95

Ranking is two-tier: gate-passing entries rank ahead of all failing entries, and within each tier lower total_score wins. Failed entries keep full diagnostic metrics but are not leaderboard-eligible, so a tiny binary rendering black frames cannot receive a respectable ranking.

Quality bands

Band	threshold
excellent	quality_loss ≤ 0.25
strong	quality_loss ≤ 0.50
acceptable	quality_loss ≤ 0.75
weak	below degenerate baseline, not eligible
degenerate_or_failed	at or worse than trivial baseline

---

Running a benchmark for a model

1. Prepare workspaces

Prepare one attempt workspace (Track B — native binary, default):

python -m scripts.prepare_model_run \
  --model-id my-model \
  --target targets/public/l1_0001 \
  --attempt attempt_01

Prepare a Track A (Python) workspace:

python -m scripts.prepare_model_run \
  --model-id my-model \
  --target targets/public/l1_0001 \
  --attempt attempt_01 \
  --track python_track

This writes runs/my-model/l1_0001/attempt_01/prompt.md (the contestant brief with scoring formula, timeout, and output spec), copies public evidence, and initialises submission/submission.json with the correct track/profile fields.

Prepare N attempts across all targets:

python -m scripts.prepare_model_suite \
  --model-id my-model \
  --targets-root targets/public \
  --attempts 3 \
  --track python_track

To include the docs/primitive_spec.md mathematical reference in the evidence:

python -m scripts.prepare_model_run \
  --model-id my-model \
  --target targets/public/l3_0001 \
  --attempt attempt_01 \
  --track python_track \
  --primitive-spec

2. Run the model

Point the model at the generated prompt.md. The model writes its submission to runs/my-model/<target_id>/<attempt>/submission/.

3. Evaluate

Single evaluation:

python -m demoscenebench.evaluate \
  --target targets/public/l1_0001 \
  --submission runs/my-model/l1_0001/attempt_01/submission \
  --out runs/my-model/l1_0001/attempt_01/eval \
  --runtime-profile native_raw \
  --require-manifest

Use --runtime-profile python_track for Track A submissions.

Suite evaluation:

python -m scripts.evaluate_suite \
  --targets-root targets/public \
  --submission-pattern 'runs/my-model/{target_id}/attempt_01/submission' \
  --out-root runs/my-model/eval \
  --runtime-profile native_raw \
  --require-manifest

4. Summarize results

Summarize one or more score files:

python -m scripts.summarize_scores \
  runs/my-model/l0_0001/attempt_01/eval \
  runs/my-model/l1_0001/attempt_01/eval

Aggregate fixed-N attempt results for a target:

python -m scripts.summarize_model_runs runs/my-model/l0_0001 --expected-n 3

Full report bundle (rows + aggregate):

python -m scripts.report_model_runs runs/my-model --expected-n 3 --out runs/my-model/report

---

Development and building (native submissions)

Use the dev container to compile Track B submissions for the Linux evaluation environment:

# Build the dev image (C, C++, Clang, NASM, Rust, Go, Python)
docker build -f Dockerfile.dev -t demoscenebench-dev .

# Compile inside the dev container
docker run --rm -it -v "$(pwd):/workspace" demoscenebench-dev bash
# Inside:
cd /workspace/runs/my-model/l1_0001/attempt_01/submission
gcc -O3 -s -o s main.c
# Exit, then evaluate normally

macOS Mach-O binaries will not run in the Linux evaluation container. Always cross-compile for Linux when targeting official scores. See docs/dev_container.md for the full dev/eval container distinction.

---

Official Docker evaluation

Run a full suite inside the official no-compiler evaluation image:

python -m scripts.docker_evaluate_suite \
  --build \
  --targets-root targets/public \
  --submission-pattern 'runs/my-model/{target_id}/attempt_01/submission' \
  --out-root runs/my-model/docker_eval

The Docker runner mounts only the target suite and submission directory, drops the submitted program to nobody, disables network access, drops Linux capabilities, and enables --require-manifest. Submissions may not call compilers, interpreters, or network tools during evaluation.

---

Release check

Before publishing or comparing model runs, verify the benchmark is internally consistent:

python -m scripts.release_check --out runs/release_check

With Docker gate:

python -m scripts.release_check --out runs/release_check --docker --docker-build

The check runs tests, regenerates public targets, calibrates metrics, and runs the manifest-required native-blank suite.

---

Metric calibration

Calibrate expected score ranges across all public targets:

python -m scripts.calibrate_metrics --targets-root targets/public --strict

Calibration checks that trivial probes (blank, noise) score ≥ 1.0 quality loss, and that the exact reconstruction scores ≈ 0 and passes the gate.

---

Compressed baseline

A compressed AV1/Opus re-encode of the target is the trivial "cheat" baseline. You can build one to understand the scoring floor:

python -m scripts.make_compressed_submission \
  --target targets/public/l3_0001 \
  --out /tmp/compressed_l3_sub

python -m demoscenebench.evaluate \
  --target targets/public/l3_0001 \
  --submission /tmp/compressed_l3_sub \
  --out /tmp/compressed_l3_eval \
  --runtime-profile open

On l3_0001 (18s composite_medium) this yields quality_loss ≈ 0.06 at 365 KB. A correct procedural reconstruction achieves quality_loss ≈ 0 at ~2 KB — an ~180× total_score advantage. See docs/baselines_comparison.md for detailed numbers.

---

Further reading

Document	Contents
docs/rng_spec.md	SplitMix64 contract, seed publication, C reference
docs/primitive_spec.md	Cosine palette, SDFs, raymarching, plasma, lissajous, tracker music
docs/dev_container.md	Dev vs eval container distinction; build workflow
docs/baselines_comparison.md	Compressed vs procedural score comparison
docs/model_run_protocol.md	Best-of-N reporting and common run structure
docs/benchmark_design.md	Target philosophy, scoring design, anti-cheat approach