SDIF Rust Parser Library

Pure Rust implementation of the Semantic Data Interchange Format (SDIF) parser,
featuring a span-annotated Abstract Syntax Tree (AST) suitable for editor tooling.

What is SDIF? · Public API · Format at a glance · Token efficiency · Ecosystem

Compact Less repeated structure. Fewer wasted tokens.

Semantic Tables, relations, metadata and intent.

Canonical Stable output for hashing, signing and comparison.

Auditable Designed to be read, reviewed and trusted.

---

What is SDIF?

SDIF — Semantic Data Interchange Format is a compact, canonicalizable and AI-friendly data format for structured information that needs to move cleanly between humans, tools, agents and deterministic workflows.

It is designed for cases where data should be:

• small enough to be efficient in AI context windows;
• structured enough for machines to parse and validate;
• readable enough for humans to review;
• deterministic enough for hashing, signing and reproducible workflows;
• semantic enough to express tables, relations, metadata and intent.

SDIF also includes an AI projection surface, .sdif.ai, designed for token-dense agent exchange while remaining reversible back into canonical SDIF when the projection contract is respected.

@sdif 1.0

kind Plan
id release.v1
title "Release readiness plan"

items[id,status,owner,evidence]:
  R1 done build "reports/build.md"
  R2 open qa "reports/tests.md"
  R3 done security "reports/audit.md"

rel:
  release.v1 validated_by R1
  release.v1 blocked_by R2
  release.v1 governed_by R3

Structured information closer to a document,
while still behaving like a contract.

---

Format at a glance

JSON repeats field names across every record:

[
  { "id": "R1", "status": "done",    "owner": "build",    "evidence": "reports/build.md"  },
  { "id": "R2", "status": "open",    "owner": "qa",       "evidence": "reports/tests.md"  },
  { "id": "R3", "status": "done",    "owner": "security", "evidence": "reports/audit.md"  }
]

SDIF declares the shape once and uses literal tabs between cells. Editors must preserve tabs — this is a deliberate tradeoff for compactness:

@sdif 1.0

kind Plan
id   release.v1
title "Release readiness plan"

items[id,status,owner,evidence]:
  R1	done	build	reports/build.md
  R2	open	qa	reports/tests.md
  R3	done	security	reports/audit.md

rel:
  release.v1  validated_by  R1
  release.v1  blocked_by    R2
  release.v1  governed_by   R3

Semantic relationships are first-class, not embedded strings.

---

Token efficiency

The benchmark derives every compared format from the same canonical JSON source in examples/golden/. Results below are from the most recent run across 21 documents and 3 tokenizers.

Format	Consensus avg rank	Median ratio vs JSON Compact
SDIF AI	1.10	56.8%
SDIF	2.60	59.5%
CSV Bundle	2.70	61.2%
YAML	5.35	95.3%
JSON Compact	5.65	100.0%
JSON Pretty	7.00	137.3%
XML	8.00	171.7%

SDIF AI wins 57 of 63 tokenizer/document pairs. SDIF canonical wins 2.

The benchmark repository contains the exact corpus model, generated artifacts and methodology needed to reproduce these numbers.

These results are corpus-dependent. Not every data shape benefits equally from tabular projection. Claude and Llama tokenizers require separate opt-in before claiming results for those models.

For full methodology, corpus model and per-document breakdowns, see sdif-benchmarks.

---

What SDIF is not

SDIF does not try to replace JSON, YAML, CSV, Markdown, XML, Parquet or Protocol Buffers. Those formats are useful and battle-tested.

JSON Universal and reliable, but noisy when repeated records dominate.

YAML Readable, but too permissive for deterministic workflows.

CSV Compact, but loses structure, relations and meaning quickly.

Markdown Great for humans, not enough when data must be parsed and verified.

SDIF focuses on a narrower problem:

compact, semantic, canonicalizable structured data
that can move cleanly between humans, machines and AI systems.

---

Public API

sdif-rs provides a zero-copy parser and a span-annotated AST in Rust.

The library exposes the following interface:

• parse_text(text: &str) -> Result<Document, ParseError> — Parse a string slice into a span-annotated AST document.
• parse_text_with_policy(text: &str, policy: &Policy) -> Result<Document, ParseError> — Parse using custom size limits and policies.
• Policy — Controls runtime resource boundaries (e.g., document size limits, matching the default 1 MB limit).
• ParseError — Standard error representation containing error codes, span coordinates, message, and diagnostic hints.
• Span — Zero-copy location tracking (start_line, start_col, end_line, end_col, 1-indexed, end exclusive).
• Document / AST nodes — Tree structures representing fields, directives, tables, relations, narratives, rules, and object blocks.

---

Development

Build and run cargo tests:

cargo build                  # compile library
cargo test                   # unit tests + conformance suite
cargo clippy -- -D warnings  # run compiler linter
cargo fmt --check            # run formatting check

---

Conformance

This crate runs automated conformance tests reading fixtures from the sibling sdif-spec repository. The Python sdif-py implementation remains the normative reference; sdif-rs must match its syntax rules and error codes exactly.

---

Ecosystem

This GitHub organization hosts the official SDIF ecosystem: the core format, reference tooling, benchmarks, examples, libraries, and editor extensions.

PYTHON CLIENT & CLI sdif-py Specification, parser, canonicalizer, and CLI. The normative reference implementation. Explore sdif-py →	SPECIFICATION (SSOT) sdif-spec Official format specification, canonicalization rules, and portable conformance test suite. View specification →	BENCHMARKS sdif-benchmarks Reproducible benchmark datasets and reports comparing SDIF with JSON, YAML, XML, and CSV. View benchmarks →
RUST IMPLEMENTATION sdif-rs Pure Rust parser implementation with a span-annotated AST designed for editor tooling. Explore sdif-rs →	LANGUAGE SERVER (LSP) sdif-lsp LSP language server binary (tower-lsp) providing real-time diagnostics and IDE features. View sdif-lsp →	EDITOR INTEGRATION vscode-sdif VS Code extension client providing syntax highlighting, diagnostics, and LSP configuration. Open extension →
GRAMMAR FOUNDATION tree-sitter-sdif Tree-sitter grammar foundation for syntax highlighting and incremental parsing. Open grammar →	DOCUMENTATION sdif-format.github.io Official documentation website containing specification guides, tutorials, and examples. Read docs →	ORGANIZATION META .github Organization profile, assets, and shared community configuration files. View profile →

Repository map

Repository	Purpose
sdif-py	Core Python parser, validator, canonicalizer, and CLI
sdif-spec	Official format specification and conformance test suite (SSOT)
sdif-benchmarks	Benchmark datasets, reports, and comparison tooling
sdif-rs	Rust parser crate with span-annotated AST
sdif-lsp	LSP language server binary
tree-sitter-sdif	Tree-sitter grammar foundation for syntax highlighting
vscode-sdif	VS Code extension client for SDIF
sdif-format.github.io	Public documentation website (Docusaurus)
.github	Organization profile, assets, and shared GitHub community files

License

Licensed under the Apache License, Version 2.0 (Apache-2.0).