Limes DataForge

Limes DataForge is a narrow data-curation challenge for small-language-model training efficiency. Participants edit filtering, ranking, deduplication, and curriculum code while the benchmark harness, public smoke data, challenge contract, and future verifier data stay fixed.

The project is designed for humans and agents running serious local research loops. Public smoke scores are only interface and candidate checks. Public frontier entries should be promoted only after trusted replay, repeated-seed verification, and a scaling audit.

What You Can Edit

• solution/filter.py
• solution/rank_documents.py
• solution/dedup.py
• solution/curriculum.py
• configs/submission.json

Everything under harness/, baselines/, data/, verifier/, verifier_data/, leaderboard/, challenge.json, and generated score files is protected for official runs.

Quickstart

Use Python 3.10 or newer. No external packages are required.

scripts/run_smoke.sh
python3 scripts/run_invariant_probes.py
python3 scripts/run_public_audit.py
python3 scripts/check_public_baseline.py --input baselines/public-smoke-baseline.json
python3 scripts/build_submission_bundle.py --changed-file solution/filter.py --output submission-bundle.json
python3 scripts/validate_submission_bundle.py --input submission-bundle.json
python3 scripts/validate_candidate_packet.py --input templates/candidate-packet.example.json --schema-only
python3 scripts/validate_search_ledger.py --input templates/search-ledger.example.json
python3 scripts/validate_replay_request.py --input templates/replay-request.example.json --schema-only
python3 scripts/validate_hidden_manifest.py --input verifier/hidden-manifest.example.json
python3 -m unittest discover -s tests
python3 -m json.tool challenge.json

The smoke run applies the editable solution hooks to a tiny public JSONL corpus and writes score.json with:

• public_proxy_loss
• baseline_public_proxy_loss
• public_proxy_delta
• kept_ratio
• dedup_rate
• selected_document_count
• selected_byte_count
• selection_hash
• baseline_document_count
• baseline_selection_hash
• stress_public_proxy_loss
• stress_baseline_public_proxy_loss
• stress_selection_hash
• stress_dataset_hash
• runtime_seconds
• dataset_hash

Lower public_proxy_loss is better for the primary public smoke check. The stress_* fields are candidate-only diagnostics on a second tiny public suite with duplicate, boilerplate, and source-mix edge cases. The baseline_public_proxy_loss field is a locked local baseline so agents can measure whether a candidate changed anything useful on the public fixture. None of these public fields is an official leaderboard score.

Submission Preflight

Completed candidates should include a filled submission.json based on templates/submission.json. Before requesting trusted replay, run:

python3 scripts/build_submission_bundle.py --output submission-bundle.json --base origin/main
python3 scripts/validate_submission_bundle.py --input submission-bundle.json
python3 scripts/check_submission.py --manifest submission.json --base origin/main
python3 scripts/validate_local_bundle.py --manifest submission.json --base origin/main

The source-bundle step hashes only the editable files that will be replayed, rejects symlinks and protected paths, and writes a canonical bundle_sha256. Copy that digest into submission.json before running the manifest guard. The guard rejects protected-file edits, files outside the editable surface, placeholder manifest values, missing public-score fields, missing stress diagnostics, failed invariant probes, and missing search-ledger validation. The local bundle validator then reruns the public scorer and invariant probes and checks the filled manifest, source bundle, and search ledger against the fresh local outputs. It intentionally ignores runtime_seconds, which is not stable across runs. Passing preflight does not imply promotion; it only means the candidate is shaped for review.

For agent or multi-attempt workflows, bundle the local evidence into one candidate packet before asking for human review:

python3 scripts/build_candidate_packet.py \
  --manifest submission.json \
  --source-bundle submission-bundle.json \
  --search-ledger search-ledger.json \
  --agent-notes agent-notes.json \
  --public-score score.json \
  --invariant-probes invariant-probes.json \
  --public-audit public-audit.json \
  --output candidate-packet.json
python3 scripts/validate_candidate_packet.py --input candidate-packet.json

A local candidate packet is still candidate-only evidence. It records that the public artifacts are internally consistent; it does not replace trusted replay, hidden validation, repeated seeds, or scaling audit.

Trusted replay requests should be validated before they enter any hidden runner queue:

python3 scripts/validate_replay_request.py --input templates/replay-request.example.json --schema-only

For real requests, omit --schema-only so the guard can check artifact hashes and the referenced candidate packet.

Official Verifier Contract

The official verifier will apply the same submitted curation code to larger public and hidden corpus shards. It will then train a fixed tiny language model with a fixed tokenizer, architecture, optimizer, schedule, seed policy, and token budget.

The primary official metric is hidden_val_loss, minimized. Promotion requires repeated-seed improvement over the locked baseline plus downstream mini-eval non-regression. A result receives the scaled label only if it survives a larger token or model budget audit.

The machine-readable public verifier contract lives at verifier/replay-contract.json. Trusted runners can inspect the same contract through:

python3 harness/verify_hidden.py --public-contract-only

Trusted runner outputs should validate before any website ingestion or status promotion:

python3 scripts/validate_replay_result.py --input templates/replay-result.example.json

Trusted runner setups should also validate their public manifest shape before promotion opens:

python3 scripts/validate_runner_manifest.py --input verifier/trusted-runner-manifest.example.json

Locked baseline records should validate before promoted comparisons are allowed:

python3 scripts/validate_baseline_record.py --input verifier/baseline-record.example.json

Trusted-only hidden shard manifests should validate against the public schema before any hidden replay opens:

python3 scripts/validate_hidden_manifest.py --input verifier/hidden-manifest.example.json

Promotion packets bind the replay result, baseline record, runner manifest, agent notes, result card, and leaderboard entry into one machine-checkable evidence bundle before any public frontier status is requested:

python3 scripts/validate_promotion_packet.py --input templates/promotion-packet.example.json

Status Labels

local -> candidate -> verified -> promoted -> replicated -> scaled

Local and public smoke scores are not claims. They are invitations to replay.

Repository Map

• challenge.json: Benchforge-style challenge contract.
• solution/: editable participant surface.
• harness/: immutable public smoke scorer and verifier-contract check.
• harness/invariant_probes.py: candidate-only public probes for determinism, ID-remap stability, mutation safety, and synthetic document robustness.
• harness/public_audit.py: candidate-only leakage, selection-boundary, source diversity, and static public-data-boundary audit.
• harness/source_bundle_guard.py: hashes the editable source files selected for replay and rejects stale or protected-file bundles.
• harness/candidate_packet_guard.py: validates local candidate evidence packets before trusted replay is requested.
• harness/replay_request_guard.py: validates anti-probing replay requests before hidden runner access.
• harness/hidden_manifest_guard.py: validates the trusted-only hidden shard manifest shape without exposing hidden data.
• baselines/public-smoke-baseline.json: stable public smoke contract used to detect accidental benchmark drift. It is not an official leaderboard baseline.
• data/public_smoke/: tiny public corpus, stress corpus, and heldout text.
• verifier/replay-contract.json: public replay, promotion, and ingestion contract. It does not include hidden data.
• verifier/task-spec.json: public curation hook and hidden replay-axis specification. It does not include hidden data.
• verifier/trusted-runner-manifest.example.json: schema-only trusted-runner setup manifest. It does not include hidden data.
• verifier/baseline-record.example.json: schema-only baseline evidence record. It is not an official comparison baseline.
• verifier/hidden-manifest.example.json: schema-only trusted hidden-shard manifest. It is not a real hidden manifest.
• docs/: anti-cheat, promotion, launch, and agent-notes policies.
• docs/verifier-runbook.md: trusted-runner replay checklist.
• docs/limeslabs-ingestion.md: website ingestion and status validation rules.
• docs/agent-quickstart.md: short instructions for coding agents.
• templates/: submission, result-card, and leaderboard-entry schemas.
• templates/agent-notes.example.json: machine-checkable agent trial notes.
• templates/search-ledger.example.json: machine-checkable search budget, attempt, stopping, and selection accounting.
• templates/submission-bundle.example.json: schema-only editable source bundle shape for replay packaging.
• templates/candidate-packet.example.json: schema-only local candidate packet shape for agents and reviewers.
• templates/replay-request.example.json: schema-only trusted replay request shape. It is not an official replay request.
• templates/replay-result.example.json: schema-only trusted replay result packet.
• templates/promotion-packet.example.json: schema-only promotion evidence bundle. It is not an official promotion packet.
• examples/limeslabs/: candidate-only fixtures for website development.
• tests/: stdlib tests for contract and scorer behavior.

Contributing

Start with CONTRIBUTING.md. Agent-driven attempts should also read docs/agent-quickstart.md before editing the solution surface.