Make the Rustler generator usable from dependent projects, and restructure the docs

CHANGELOG.md

@@ -8,11 +8,19 @@
   decoders read with, and ships `decoder.rs`, a complete one. The generator
   reads its method signatures unless `:decoder_sources` names another
   `Decoder`, so every feature knows which reads return a `NifResult`.
+- Projects that depend on KiwiCodec can use `KiwiCodec.RustlerGenerator`.
+  RustQ is an optional dependency, and the generator compiles wherever it is
+  available, such as a project's development and test builds; it used to
+  compile only in KiwiCodec's own, so projects loaded its sources themselves.
 - `KiwiCodec.Schema.Token` has a `kind`: `:identifier`, `:integer`,
   `:symbol`, or `:eof`.
 
 ### Changed
 
+- The docs have a Native decoders guide covering each generator feature,
+  entrypoints, and the `Decoder`. The generator architecture guide is now
+  Internals, for people changing KiwiCodec, and the module groups separate the
+  public API from internals.
 - The schema tokenizer reads with binary patterns rather than a regular
   expression, and is about 14 times faster on Figma's schema. Tokens,
   positions, and errors are unchanged.

README.md

@@ -1,203 +1,156 @@
 # KiwiCodec
 
-KiwiCodec is a pure Elixir implementation of the Kiwi schema binary codec: a compact, schema-driven message format similar in spirit to Protocol Buffers but with its own wire encoding.
+[![Hex.pm](https://img.shields.io/hexpm/v/kiwi_codec.svg)](https://hex.pm/packages/kiwi_codec) [![Documentation](https://img.shields.io/badge/documentation-gray)](https://kiwi-codec.hexdocs.pm)
 
-The package stays generic: product-specific `.kiwi` schemas can live in companion packages.
-
-## Scope
-
-KiwiCodec owns:
-
-- Kiwi wire primitives: varuint, zigzag int, uint64/int64, varfloat, null-terminated strings, and length-prefixed byte arrays
-- `.kiwi` schema parsing and validation
-- Schema interpretation for parsed schemas
-- Field metadata, including original schema field names
-- An idiomatic Elixir DSL and code generator for static modules
-- Generic chunk container helpers
-
-## Static modules
-
-Generated modules are regular Elixir structs:
+[Kiwi](https://github.com/evanw/kiwi) schemas and messages in Elixir. Parse `.kiwi`
+schemas, encode and decode messages as Elixir structs, and generate Rustler
+decoders for the same schemas when decoding has to be native.
 
 ```elixir
-defmodule Example.Node do
+defmodule MyApp.Node do
   use KiwiCodec, kind: :message
 
   field :id, 1, type: :uint
   field :name, 2, type: :string
 end
 
-node = %Example.Node{id: 42, name: "hello"}
-binary = KiwiCodec.encode(node)
-node = KiwiCodec.decode(binary, Example.Node)
+binary = KiwiCodec.encode(%MyApp.Node{id: 42, name: "hello"})
+KiwiCodec.decode(binary, MyApp.Node)
+#=> %MyApp.Node{id: 42, name: "hello"}
 ```
 
-Enums use atoms:
+Kiwi is the compact, schema-driven format Figma's `.fig` files use. Its wire
+encoding differs from Protocol Buffers: variable-length integers and floats,
+null-terminated strings, messages that carry field ids, and structs that do not.
+KiwiCodec encodes it byte for byte as the OpenPencil TypeScript runtime does, and
+stays generic: product schemas, such as Figma's in
+[Figler](https://github.com/open-pencil/figler), live in their own packages.
 
-```elixir
-defmodule Example.Kind do
-  use KiwiCodec, kind: :enum
+## Installation
 
-  enum_value :none, 0
-  enum_value :frame, 4
+```elixir
+def deps do
+  [
+    {:kiwi_codec, "~> 0.5.0"}
+  ]
 end
 ```
 
-## Schema compilation
+The codec is pure Elixir. The Rustler generator also needs RustQ; see
+[Native decoders](#native-decoders).
+
+## Schemas as modules
 
-For application code, compile schema text into Elixir modules and use the static struct API:
+Generate a module per definition from a schema, with `@type`s and the schema's
+field names in each module's metadata:
 
 ```sh
 mix kiwi.gen schema.kiwi --module-prefix MyApp.Schema --out lib/generated
 ```
 
-For tests and tooling, modules can also be compiled in memory:
+Or compile them in memory, as tests and tools may:
 
 ```elixir
 KiwiCodec.compile_schema!(schema_text, module_prefix: MyApp.Schema)
 ```
 
-`KiwiCodec.parse_schema!/1` only parses schema text into an AST. It does not create modules by itself.
+Enums decode to atoms, and modules can be written by hand with the same DSL:
+
+```elixir
+defmodule MyApp.Kind do
+  use KiwiCodec, kind: :enum
+
+  enum_value :none, 0
+  enum_value :frame, 4
+end
+```
+
+A module may name a `KiwiCodec.TransformModule` in `transform_module/0` to
+normalize its structs after decoding and before encoding.
 
-## Schema interpretation
+## Schemas at runtime
 
-When a schema is loaded at runtime and you do not want to generate modules, use `KiwiCodec.SchemaInterpreter`:
+When the schema arrives with the data, `KiwiCodec.SchemaInterpreter` encodes and
+decodes string-keyed maps without generating modules:
 
 ```elixir
 schema = KiwiCodec.parse_schema!(schema_text)
 binary = KiwiCodec.SchemaInterpreter.encode(schema, "Thing", %{"id" => 1, "name" => "demo"})
-value = KiwiCodec.SchemaInterpreter.decode(schema, "Thing", binary)
+KiwiCodec.SchemaInterpreter.decode(schema, "Thing", binary)
+#=> %{"id" => 1, "name" => "demo"}
 ```
 
-For repeated interpretation, prepare the schema once. Prepared schemas index
-definitions, message fields, and enum values instead of scanning schema lists for
-every decoded field:
+`SchemaInterpreter.prepare/1` indexes a schema once for repeated use.
+`KiwiCodec.Schema.Binary` reads and writes schemas in Kiwi's binary form, which
+is how they travel alongside data.
+
+Data written with a newer schema can still decode into modules generated from an
+older one. Pass the schema it was written with, and fields the modules do not
+define are skipped:
 
 ```elixir
-prepared = KiwiCodec.SchemaInterpreter.prepare(schema)
-value = KiwiCodec.SchemaInterpreter.decode(prepared, "Thing", binary)
+KiwiCodec.decode(binary, MyApp.Schema.Node, schema: written_schema)
 ```
 
-## Transform modules
+## Containers
 
-Modules can override `transform_module/0` for custom normalization before encode and after decode:
+`KiwiCodec.Container` reads and builds the chunked container `.fig` files use: a
+magic string, a version, and length-prefixed chunks.
 
 ```elixir
-defmodule Example.Transform do
-  @behaviour KiwiCodec.TransformModule
-
-  def encode(message, _module), do: message
-  def decode(message, _module), do: message
-end
+binary = KiwiCodec.Container.build([KiwiCodec.Container.deflate(schema), KiwiCodec.Container.deflate(data)])
+%{version: 101, chunks: [schema_chunk, data_chunk]} = KiwiCodec.Container.parse(binary)
 ```
 
-## Binary schemas
+## Native decoders
+
+`KiwiCodec.RustlerGenerator` generates Rustler decoders for a schema, to decode
+in Rust what is too large or too hot for Elixir. It needs
+[RustQ](https://hex.pm/packages/rustq), an optional dependency of KiwiCodec, so
+add it where you generate:
 
 ```elixir
-binary_schema = KiwiCodec.Schema.Binary.encode(schema)
-schema = KiwiCodec.Schema.Binary.decode(binary_schema)
+{:rustq, "~> 1.0.0-rc.14", only: [:dev, :test], runtime: false}
 ```
 
-## Rustler decoder generation
-
-`KiwiCodec.RustlerGenerator.source/2` returns complete generated Rust source for
-native Rustler decoders. It is a build-time API compiled in development and test
-environments; production compilation includes only the Kiwi codec/runtime APIs
-and does not require RustQ. Use the generator from `rustq.exs` and let
-`mix rustq.gen` own writing and freshness checks:
+Then generate from `rustq.exs`, and let `mix rustq.gen` write the file and
+`mix rustq.gen --check` keep it fresh:
 
 ```elixir
 use RustQ.Config
 
 schema = KiwiCodec.parse_schema!(File.read!("priv/schema.kiwi"))
 
-generate :native_decoders, "native/my_nif/src/generated.rs" do
+generate :decoders, "native/my_nif/src/generated.rs" do
   content KiwiCodec.RustlerGenerator.source(schema,
     entrypoints: ["Node"],
     module_prefix: "MyApp.Schema"
   )
 end
 ```
 
-Definition names infer NIF names such as `decode_node`. For Rustler projects,
-prefer keeping the native API list in one non-Rustler metadata module:
-
-```elixir
-defmodule MyApp.Native.Nifs do
-  @moduledoc "Native NIF stubs exposed by MyApp."
-
-  @stubs [decode_node: 1, decode_image: 1]
-
-  def stubs, do: @stubs
-end
-```
-
-Then use the same metadata from `rustq.exs`:
-
-```elixir
-generate :native_decoders, "native/my_nif/src/generated.rs" do
-  content KiwiCodec.RustlerGenerator.source(schema,
-    entrypoints: {:nif_stubs, MyApp.Native.Nifs},
-    module_prefix: "MyApp.Schema"
-  )
-end
-```
-
-The generated file includes Rustler imports, RustQ-provided atom/struct helpers,
-schema decoders, and requested NIF entrypoints. The native crate provides the
-Kiwi `Decoder` the generated code reads with, imported from
-`crate::runtime::Decoder` unless you pass `decoder: "some::path::Decoder"`.
-`KiwiCodec.RustlerGenerator.Decoder` lists the methods each feature uses, and
-ships `decoder.rs`, a complete `Decoder` to copy or start from. The tests build
-every generator feature against it.
-
-RustQ reads the `Decoder`'s method signatures to know which calls return a
-`NifResult`. If yours differ from the reference one, pass its source:
-
-```elixir
-content KiwiCodec.RustlerGenerator.source(schema,
-  features: [:full, :sparse, :skip],
-  entrypoints: {:nif_stubs, MyApp.Native.Nifs},
-  module_prefix: "MyApp.Schema",
-  decoder_sources: ["native/my_nif/src/runtime.rs"]
-)
-```
-
-For generator internals, especially the compact Rust macro boundary used to keep
-large schemas readable and small, see the Rustler generator architecture guide.
-
-## Containers
-
-```elixir
-binary = KiwiCodec.Container.build([KiwiCodec.Container.deflate("schema"), KiwiCodec.Container.deflate("data")])
-parsed = KiwiCodec.Container.parse(binary)
-```
-
-## Benchmarks
-
-```sh
-elixir bench/codec_bench.exs
-```
-
-The Benchee suite compares static and runtime-interpreted codecs for both a
-small nested message and 256-field sparse/dense messages. It reports throughput,
-latency percentiles, and BEAM allocations.
+`entrypoints: ["Node"]` generates a `decode_node` NIF that returns the same
+structs `KiwiCodec.decode/2` does. Other features generate sparse maps of the
+fields present, skip functions, native records Rust code can work with, and
+decoders that skip fields the compiled schema lacks. The crate provides the
+`Decoder` the generated code reads with; KiwiCodec ships a complete one to copy.
+[Native decoders](https://kiwi-codec.hexdocs.pm/native-decoders.html) covers
+each feature, entrypoints, and the `Decoder`.
 
 ## Development
 
+See [Internals](guides/internals.md) for how the codec and the generator are put
+together, and the checks to run when changing the generator.
+
 ```sh
 mix deps.get
 mix ci
 ```
 
-## Installation
+`mix ci` builds the generated Rust, so it needs Rust and Cargo.
+`elixir bench/codec_bench.exs` compares generated modules with the schema
+interpreter on small and 256-field messages.
 
-Once published, add it to your dependencies:
+## License
 
-```elixir
-def deps do
-  [
-    {:kiwi_codec, "~> 0.4.0"}
-  ]
-end
-```
+MIT