ToonDrive is an experimental local-first semantic storage and retrieval architecture for large structured datasets.
The project explores how structured datasets such as JSON and XML can be normalised, semantically partitioned, chunked, retrieved, and deterministically rehydrated.
Rather than treating large structured documents as opaque monolithic files, ToonDrive investigates representing them as semantically connected storage objects with graph-aware relationships, manifest-driven storage, and retrieval-oriented partitioning.
Modern structured datasets contain rich internal structure, relationships, and semantic meaning, yet are typically stored and handled as opaque serialised files.
As datasets continue to grow in size and complexity, this creates several challenges:
- duplicated structural overhead
- inefficient partial retrieval
- poor semantic locality
- limited queryability without full parsing
- increasing storage and memory pressure
- weak integration with retrieval and AI-oriented workflows
ToonDrive explores whether structured datasets can instead be represented as semantically aware storage objects rather than opaque documents.
The project investigates whether semantic partitioning, graph-aware relationships, manifest-driven storage, and deterministic reconstruction can enable:
- more efficient retrieval behaviour
- improved locality and partial access
- semantic queryability
- retrieval-oriented workflows
- reduced structural redundancy
- more storage-efficient representations
while still preserving the ability to deterministically reconstruct the original source structure.
ToonDrive is currently in the V0.1 architecture and proof-of-concept stage.
The initial focus is on:
- defining the core architecture
- writing the V0.1 whitepaper
- exploring TOON as a semantic compact object representation
- defining semantic partitioning before physical chunking as a core architectural principle
- defining graph-aware relationships between semantic objects
- designing manifest-driven storage and deterministic rehydration
TOON is the working semantic representation model being explored within ToonDrive.
The goal is to investigate whether large structured datasets can be transformed into a more compact, semantically aware representation while still supporting deterministic reconstruction back to their original structure.
TOON is intended to act as a semantic intermediate representation layer rather than a replacement for source formats.
ToonDrive explores partitioning structured data by semantic boundaries before performing physical chunking.
The intention is to avoid arbitrary chunk boundaries and instead preserve meaningful contextual groupings and relationships between objects.
Semantic meaning should define partition boundaries before storage mechanics define chunk boundaries.
Structured objects are intended to exist within a lightweight semantic graph model.
This allows:
- relationships between entities to be preserved
- related structures to remain queryable
- retrieval systems to navigate contextual connections
- semantic locality to influence partitioning and retrieval
One of the primary goals of ToonDrive is deterministic reconstruction of the original structure from semantic partitions and physical chunks.
The project explores how manifests, references, and semantic metadata can enable lossless rehydration.
The initial proof-of-concept focuses on:
- ingesting structured JSON
- converting structures into TOON canonical representations
- building semantic object relationships
- semantic partitioning
- physical chunk generation
- manifest and index generation
- deterministic rehydration
- retrieval experimentation
- benchmarking storage size, retrieval behaviour, and reconstruction integrity
Original files will initially be retained during validation phases to verify integrity and deterministic reconstruction behaviour.
whitepaper/ -> V0.1 architecture and design documents
docs/ -> Technical notes and subsystem design
src/ -> Source code
tests/ -> Validation and rehydration tests
examples/ -> Sample input/output structures
diagrams/ -> Architecture diagrams
research/ -> Reading notes and architectural exploration
Potential future areas of exploration include:
- XML and multi-format ingestion
- parser abstraction layers
- semantic indexing engines
- graph traversal and retrieval
- vector-aware retrieval systems
- metadata-driven query systems
- retrieval-oriented storage systems
- AI-assisted semantic classification
- local semantic retrieval and SLM integration perhaps
ToonDrive is currently an early-stage research and engineering project focused on architectural exploration, systems thinking, and semantic storage experimentation.
A significant part of the project is the learning process itself, with ToonDrive serving as a long-term exploration into storage systems, semantic retrieval, and systems architecture.