CratesDataset

CratesDataset is a Rust-based foundry for turning immutable monthly crates.io version metadata into purpose-built analytical and training-ready datasets.

It produces:

• crate_versions
• crate_lifecycles
• dependency_edges
• feature_edges
• release_transitions
• yanked_versions
• yank_replacements
• manifest_patterns

Every dataset is available as Zstandard-compressed JSONL and Parquet and carries a versioned schema, source snapshot, generator version, derivation class, statistics, and integrity hashes.

Quick start

Prerequisites are Rust 1.97+ and DuckDB 1.5+.

cargo build --release

.\target\release\crates-dataset.exe inspect `
  --input D:\.data\crates.io\jul-2026\cratesio_07-25-26.jsonl `
  --json

.\target\release\crates-dataset.exe build `
  --input D:\.data\crates.io\apr-2026\crates_04-17-26.jsonl `
  --snapshot 2026-04-17 `
  --output D:\.data\crates.io\derived\CratesDataset `
  --datasets all `
  --partitions 256 `
  --json

The pipeline never assumes that versions of the same crate are adjacent. It partitions normalized records by a stable crate-name hash and loads only one partition at a time.

See the dataset catalog, input contract, command contract, and provenance/licensing record.

Accepted releases can be prepared for Hugging Face with the offline hub package and hub verify commands. Network publication is preview-only unless hub publish --apply is explicit, and public publication requires both a distribution-approval record and an exact accepted-release record. See Hub publication.

Important limitations

• The crates.io snapshot says whether a version is yanked, not why or when it was yanked.
• yank_replacements contains deterministic candidates, not confirmed maintainer intent.
• Manifest patterns are reproducible signals, not maliciousness classifications.
• Public redistribution is blocked until the registry-index metadata license is explicitly confirmed.

Project code license

The CratesDataset source code is MIT-licensed. Dataset rights and upstream crate licenses are separate and documented per release.