diff --git a/CHANGELOG.md b/CHANGELOG.md index cee5595..232e0a4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,11 +8,19 @@ decoders read with, and ships `decoder.rs`, a complete one. The generator reads its method signatures unless `:decoder_sources` names another `Decoder`, so every feature knows which reads return a `NifResult`. +- Projects that depend on KiwiCodec can use `KiwiCodec.RustlerGenerator`. + RustQ is an optional dependency, and the generator compiles wherever it is + available, such as a project's development and test builds; it used to + compile only in KiwiCodec's own, so projects loaded its sources themselves. - `KiwiCodec.Schema.Token` has a `kind`: `:identifier`, `:integer`, `:symbol`, or `:eof`. ### Changed +- The docs have a Native decoders guide covering each generator feature, + entrypoints, and the `Decoder`. The generator architecture guide is now + Internals, for people changing KiwiCodec, and the module groups separate the + public API from internals. - The schema tokenizer reads with binary patterns rather than a regular expression, and is about 14 times faster on Figma's schema. Tokens, positions, and errors are unchanged. diff --git a/README.md b/README.md index cf8da28..bb8d7d1 100644 --- a/README.md +++ b/README.md @@ -1,117 +1,127 @@ # KiwiCodec -KiwiCodec is a pure Elixir implementation of the Kiwi schema binary codec: a compact, schema-driven message format similar in spirit to Protocol Buffers but with its own wire encoding. +[![Hex.pm](https://img.shields.io/hexpm/v/kiwi_codec.svg)](https://hex.pm/packages/kiwi_codec) [![Documentation](https://img.shields.io/badge/documentation-gray)](https://kiwi-codec.hexdocs.pm) -The package stays generic: product-specific `.kiwi` schemas can live in companion packages. - -## Scope - -KiwiCodec owns: - -- Kiwi wire primitives: varuint, zigzag int, uint64/int64, varfloat, null-terminated strings, and length-prefixed byte arrays -- `.kiwi` schema parsing and validation -- Schema interpretation for parsed schemas -- Field metadata, including original schema field names -- An idiomatic Elixir DSL and code generator for static modules -- Generic chunk container helpers - -## Static modules - -Generated modules are regular Elixir structs: +[Kiwi](https://github.com/evanw/kiwi) schemas and messages in Elixir. Parse `.kiwi` +schemas, encode and decode messages as Elixir structs, and generate Rustler +decoders for the same schemas when decoding has to be native. ```elixir -defmodule Example.Node do +defmodule MyApp.Node do use KiwiCodec, kind: :message field :id, 1, type: :uint field :name, 2, type: :string end -node = %Example.Node{id: 42, name: "hello"} -binary = KiwiCodec.encode(node) -node = KiwiCodec.decode(binary, Example.Node) +binary = KiwiCodec.encode(%MyApp.Node{id: 42, name: "hello"}) +KiwiCodec.decode(binary, MyApp.Node) +#=> %MyApp.Node{id: 42, name: "hello"} ``` -Enums use atoms: +Kiwi is the compact, schema-driven format Figma's `.fig` files use. Its wire +encoding differs from Protocol Buffers: variable-length integers and floats, +null-terminated strings, messages that carry field ids, and structs that do not. +KiwiCodec encodes it byte for byte as the OpenPencil TypeScript runtime does, and +stays generic: product schemas, such as Figma's in +[Figler](https://github.com/open-pencil/figler), live in their own packages. -```elixir -defmodule Example.Kind do - use KiwiCodec, kind: :enum +## Installation - enum_value :none, 0 - enum_value :frame, 4 +```elixir +def deps do + [ + {:kiwi_codec, "~> 0.5.0"} + ] end ``` -## Schema compilation +The codec is pure Elixir. The Rustler generator also needs RustQ; see +[Native decoders](#native-decoders). + +## Schemas as modules -For application code, compile schema text into Elixir modules and use the static struct API: +Generate a module per definition from a schema, with `@type`s and the schema's +field names in each module's metadata: ```sh mix kiwi.gen schema.kiwi --module-prefix MyApp.Schema --out lib/generated ``` -For tests and tooling, modules can also be compiled in memory: +Or compile them in memory, as tests and tools may: ```elixir KiwiCodec.compile_schema!(schema_text, module_prefix: MyApp.Schema) ``` -`KiwiCodec.parse_schema!/1` only parses schema text into an AST. It does not create modules by itself. +Enums decode to atoms, and modules can be written by hand with the same DSL: + +```elixir +defmodule MyApp.Kind do + use KiwiCodec, kind: :enum + + enum_value :none, 0 + enum_value :frame, 4 +end +``` + +A module may name a `KiwiCodec.TransformModule` in `transform_module/0` to +normalize its structs after decoding and before encoding. -## Schema interpretation +## Schemas at runtime -When a schema is loaded at runtime and you do not want to generate modules, use `KiwiCodec.SchemaInterpreter`: +When the schema arrives with the data, `KiwiCodec.SchemaInterpreter` encodes and +decodes string-keyed maps without generating modules: ```elixir schema = KiwiCodec.parse_schema!(schema_text) binary = KiwiCodec.SchemaInterpreter.encode(schema, "Thing", %{"id" => 1, "name" => "demo"}) -value = KiwiCodec.SchemaInterpreter.decode(schema, "Thing", binary) +KiwiCodec.SchemaInterpreter.decode(schema, "Thing", binary) +#=> %{"id" => 1, "name" => "demo"} ``` -For repeated interpretation, prepare the schema once. Prepared schemas index -definitions, message fields, and enum values instead of scanning schema lists for -every decoded field: +`SchemaInterpreter.prepare/1` indexes a schema once for repeated use. +`KiwiCodec.Schema.Binary` reads and writes schemas in Kiwi's binary form, which +is how they travel alongside data. + +Data written with a newer schema can still decode into modules generated from an +older one. Pass the schema it was written with, and fields the modules do not +define are skipped: ```elixir -prepared = KiwiCodec.SchemaInterpreter.prepare(schema) -value = KiwiCodec.SchemaInterpreter.decode(prepared, "Thing", binary) +KiwiCodec.decode(binary, MyApp.Schema.Node, schema: written_schema) ``` -## Transform modules +## Containers -Modules can override `transform_module/0` for custom normalization before encode and after decode: +`KiwiCodec.Container` reads and builds the chunked container `.fig` files use: a +magic string, a version, and length-prefixed chunks. ```elixir -defmodule Example.Transform do - @behaviour KiwiCodec.TransformModule - - def encode(message, _module), do: message - def decode(message, _module), do: message -end +binary = KiwiCodec.Container.build([KiwiCodec.Container.deflate(schema), KiwiCodec.Container.deflate(data)]) +%{version: 101, chunks: [schema_chunk, data_chunk]} = KiwiCodec.Container.parse(binary) ``` -## Binary schemas +## Native decoders + +`KiwiCodec.RustlerGenerator` generates Rustler decoders for a schema, to decode +in Rust what is too large or too hot for Elixir. It needs +[RustQ](https://hex.pm/packages/rustq), an optional dependency of KiwiCodec, so +add it where you generate: ```elixir -binary_schema = KiwiCodec.Schema.Binary.encode(schema) -schema = KiwiCodec.Schema.Binary.decode(binary_schema) +{:rustq, "~> 1.0.0-rc.14", only: [:dev, :test], runtime: false} ``` -## Rustler decoder generation - -`KiwiCodec.RustlerGenerator.source/2` returns complete generated Rust source for -native Rustler decoders. It is a build-time API compiled in development and test -environments; production compilation includes only the Kiwi codec/runtime APIs -and does not require RustQ. Use the generator from `rustq.exs` and let -`mix rustq.gen` own writing and freshness checks: +Then generate from `rustq.exs`, and let `mix rustq.gen` write the file and +`mix rustq.gen --check` keep it fresh: ```elixir use RustQ.Config schema = KiwiCodec.parse_schema!(File.read!("priv/schema.kiwi")) -generate :native_decoders, "native/my_nif/src/generated.rs" do +generate :decoders, "native/my_nif/src/generated.rs" do content KiwiCodec.RustlerGenerator.source(schema, entrypoints: ["Node"], module_prefix: "MyApp.Schema" @@ -119,85 +129,28 @@ generate :native_decoders, "native/my_nif/src/generated.rs" do end ``` -Definition names infer NIF names such as `decode_node`. For Rustler projects, -prefer keeping the native API list in one non-Rustler metadata module: - -```elixir -defmodule MyApp.Native.Nifs do - @moduledoc "Native NIF stubs exposed by MyApp." - - @stubs [decode_node: 1, decode_image: 1] - - def stubs, do: @stubs -end -``` - -Then use the same metadata from `rustq.exs`: - -```elixir -generate :native_decoders, "native/my_nif/src/generated.rs" do - content KiwiCodec.RustlerGenerator.source(schema, - entrypoints: {:nif_stubs, MyApp.Native.Nifs}, - module_prefix: "MyApp.Schema" - ) -end -``` - -The generated file includes Rustler imports, RustQ-provided atom/struct helpers, -schema decoders, and requested NIF entrypoints. The native crate provides the -Kiwi `Decoder` the generated code reads with, imported from -`crate::runtime::Decoder` unless you pass `decoder: "some::path::Decoder"`. -`KiwiCodec.RustlerGenerator.Decoder` lists the methods each feature uses, and -ships `decoder.rs`, a complete `Decoder` to copy or start from. The tests build -every generator feature against it. - -RustQ reads the `Decoder`'s method signatures to know which calls return a -`NifResult`. If yours differ from the reference one, pass its source: - -```elixir -content KiwiCodec.RustlerGenerator.source(schema, - features: [:full, :sparse, :skip], - entrypoints: {:nif_stubs, MyApp.Native.Nifs}, - module_prefix: "MyApp.Schema", - decoder_sources: ["native/my_nif/src/runtime.rs"] -) -``` - -For generator internals, especially the compact Rust macro boundary used to keep -large schemas readable and small, see the Rustler generator architecture guide. - -## Containers - -```elixir -binary = KiwiCodec.Container.build([KiwiCodec.Container.deflate("schema"), KiwiCodec.Container.deflate("data")]) -parsed = KiwiCodec.Container.parse(binary) -``` - -## Benchmarks - -```sh -elixir bench/codec_bench.exs -``` - -The Benchee suite compares static and runtime-interpreted codecs for both a -small nested message and 256-field sparse/dense messages. It reports throughput, -latency percentiles, and BEAM allocations. +`entrypoints: ["Node"]` generates a `decode_node` NIF that returns the same +structs `KiwiCodec.decode/2` does. Other features generate sparse maps of the +fields present, skip functions, native records Rust code can work with, and +decoders that skip fields the compiled schema lacks. The crate provides the +`Decoder` the generated code reads with; KiwiCodec ships a complete one to copy. +[Native decoders](https://kiwi-codec.hexdocs.pm/native-decoders.html) covers +each feature, entrypoints, and the `Decoder`. ## Development +See [Internals](guides/internals.md) for how the codec and the generator are put +together, and the checks to run when changing the generator. + ```sh mix deps.get mix ci ``` -## Installation +`mix ci` builds the generated Rust, so it needs Rust and Cargo. +`elixir bench/codec_bench.exs` compares generated modules with the schema +interpreter on small and 256-field messages. -Once published, add it to your dependencies: +## License -```elixir -def deps do - [ - {:kiwi_codec, "~> 0.4.0"} - ] -end -``` +MIT diff --git a/codegen/kiwi_codec/rustler_generator.ex b/codegen/kiwi_codec/rustler_generator.ex index ac0faab..6565172 100644 --- a/codegen/kiwi_codec/rustler_generator.ex +++ b/codegen/kiwi_codec/rustler_generator.ex @@ -1,199 +1,203 @@ -defmodule KiwiCodec.RustlerGenerator do - @moduledoc """ - Generates Rustler decoder code from Kiwi schemas for RustQ manifests. - - This is an experimental bridge for optional native backends. It generates a - complete Rust source file for schema-dependent decoder functions and NIF - entrypoints; `rustq.exs` owns writing and freshness checks. - - The generated source includes the Rustler imports, RustQ Rustler term helpers, - schema decoders, and requested entrypoints. The caller provides the `Decoder<'a>` - they read with; `KiwiCodec.RustlerGenerator.Decoder` describes it and ships a - complete one. - """ - - alias KiwiCodec.RustlerGenerator.Definition - alias KiwiCodec.RustlerGenerator.Entrypoint - alias KiwiCodec.RustlerGenerator.Records - alias KiwiCodec.RustlerGenerator.Selection - alias KiwiCodec.RustlerGenerator.Skip - alias KiwiCodec.RustlerGenerator.Sparse - alias KiwiCodec.RustlerGenerator.Splice - alias KiwiCodec.Schema - alias RustQ.Rust - alias RustQ.Rust.AST.Builder, as: A - - @type entrypoint :: Entrypoint.entrypoint() - - @doc """ - Returns a complete Rust source file for generated Rustler decoders. - - generate :native_decoders, "native/my_nif/src/generated.rs" do - schema = KiwiCodec.parse_schema!(File.read!("priv/schema.kiwi")) - - content KiwiCodec.RustlerGenerator.source(schema, - entrypoints: ["Node"], - module_prefix: "Example.Schema" - ) - end +if Code.ensure_loaded?(RustQ) do + defmodule KiwiCodec.RustlerGenerator do + @moduledoc """ + Generates Rust that decodes Kiwi data for a schema inside a Rustler NIF. + + `source/2` returns a complete Rust source file: decoders for the schema's + definitions, the helpers they share, and NIF entrypoints. Call it from + `rustq.exs`, where `mix rustq.gen` writes the file and `mix rustq.gen --check` + keeps it fresh. The crate provides the `Decoder<'a>` the generated code reads + with; `KiwiCodec.RustlerGenerator.Decoder` describes it and ships a complete + one. + + The generator needs RustQ, an optional dependency, and exists only where RustQ + is available. [Native decoders](native-decoders.md) covers each feature. + """ + + alias KiwiCodec.RustlerGenerator.Definition + alias KiwiCodec.RustlerGenerator.Entrypoint + alias KiwiCodec.RustlerGenerator.Records + alias KiwiCodec.RustlerGenerator.Selection + alias KiwiCodec.RustlerGenerator.Skip + alias KiwiCodec.RustlerGenerator.Sparse + alias KiwiCodec.RustlerGenerator.Splice + alias KiwiCodec.Schema + alias RustQ.Rust + alias RustQ.Rust.AST.Builder, as: A + + @type entrypoint :: Entrypoint.entrypoint() + + @doc """ + Returns a complete Rust source file for generated Rustler decoders. + + generate :native_decoders, "native/my_nif/src/generated.rs" do + schema = KiwiCodec.parse_schema!(File.read!("priv/schema.kiwi")) + + content KiwiCodec.RustlerGenerator.source(schema, + entrypoints: ["Node"], + module_prefix: "Example.Schema" + ) + end + + Options: + + * `:definitions` - schema definition names to generate, including their + dependencies. Defaults to the requested entrypoint definitions, or all + definitions when no entrypoints are requested. + * `:entrypoints` - `:all`, `{:nif_stubs, module}`, a NIF stub module, + definition names, or `{nif_name, definition_name}` entries for generated + NIFs. Definition names infer `decode_` NIFs. `:all` + generates one inferred NIF for every schema definition. A NIF stub module + infers entries from one-arity `decode_*` stubs whose suffix matches a schema + definition. For `{:nif_stubs, module}`, the module must expose `stubs/0`. + * `:module_prefix` - Elixir module prefix for decoded structs. + * `:decoder` - Rust path imported as `Decoder`. Defaults to + `"crate::runtime::Decoder"`. + * `:decoder_sources` - Rust source file or files that define the imported + `Decoder`, read for which of its methods return a `NifResult`. Defaults to + the reference `Decoder`; pass yours when its signatures differ. + * `:features` - decoder families to generate. Defaults to `[:full]`. + Use `:sparse` and `:skip` for generic sparse map decoders and skip + helpers used by projection-oriented native backends, and `:records` for + native records Rust code can work with before encoding them; see + `KiwiCodec.RustlerGenerator.Records`. + * `:unknown_fields` - how message decoders handle fields the schema does + not define: `:error` (default) rejects the data; `:runtime_schema` skips + them using a schema read with the data. See + `KiwiCodec.RustlerGenerator.UnknownFields` for the `Decoder` contract. + * `:sparse_messages` - sparse message generation strategy. Defaults to + `:match`. Use `:descriptor` to generate compact descriptor-backed sparse + message decoders and benchmark before adopting on hot schemas. + + """ + @spec source(Schema.t(), keyword()) :: String.t() + def source(%Schema{} = schema, opts) do + [source_prelude(opts), generated_fragments(schema, opts)] + |> List.flatten() + |> Enum.map_join("\n\n", &Rust.to_fragment/1) + |> Kernel.<>("\n") + end - Options: - - * `:definitions` - schema definition names to generate, including their - dependencies. Defaults to the requested entrypoint definitions, or all - definitions when no entrypoints are requested. - * `:entrypoints` - `:all`, `{:nif_stubs, module}`, a NIF stub module, - definition names, or `{nif_name, definition_name}` entries for generated - NIFs. Definition names infer `decode_` NIFs. `:all` - generates one inferred NIF for every schema definition. A NIF stub module - infers entries from one-arity `decode_*` stubs whose suffix matches a schema - definition. For `{:nif_stubs, module}`, the module must expose `stubs/0`. - * `:module_prefix` - Elixir module prefix for decoded structs. - * `:decoder` - Rust path imported as `Decoder`. Defaults to - `"crate::runtime::Decoder"`. - * `:decoder_sources` - Rust source file or files that define the imported - `Decoder`, read for which of its methods return a `NifResult`. Defaults to - the reference `Decoder`; pass yours when its signatures differ. - * `:features` - decoder families to generate. Defaults to `[:full]`. - Use `:sparse` and `:skip` for generic sparse map decoders and skip - helpers used by projection-oriented native backends, and `:records` for - native records Rust code can work with before encoding them; see - `KiwiCodec.RustlerGenerator.Records`. - * `:unknown_fields` - how message decoders handle fields the schema does - not define: `:error` (default) rejects the data; `:runtime_schema` skips - them using a schema read with the data. See - `KiwiCodec.RustlerGenerator.UnknownFields` for the `Decoder` contract. - * `:sparse_messages` - sparse message generation strategy. Defaults to - `:match`. Use `:descriptor` to generate compact descriptor-backed sparse - message decoders and benchmark before adopting on hot schemas. - - """ - @spec source(Schema.t(), keyword()) :: String.t() - def source(%Schema{} = schema, opts) do - [source_prelude(opts), generated_fragments(schema, opts)] - |> List.flatten() - |> Enum.map_join("\n\n", &Rust.to_fragment/1) - |> Kernel.<>("\n") - end + defp generated_fragments(%Schema{} = schema, opts) do + entrypoints = opts |> Keyword.get(:entrypoints, []) |> normalize_entrypoints(schema) + definitions = Keyword.get(opts, :definitions, inferred_definitions(entrypoints)) + module_prefix = Keyword.fetch!(opts, :module_prefix) - defp generated_fragments(%Schema{} = schema, opts) do - entrypoints = opts |> Keyword.get(:entrypoints, []) |> normalize_entrypoints(schema) - definitions = Keyword.get(opts, :definitions, inferred_definitions(entrypoints)) - module_prefix = Keyword.fetch!(opts, :module_prefix) + definition_map = Selection.definition_map(schema) + selected = Selection.definitions(schema, definitions, definition_map) - definition_map = Selection.definition_map(schema) - selected = Selection.definitions(schema, definitions, definition_map) + features = Keyword.get(opts, :features, [:full]) - features = Keyword.get(opts, :features, [:full]) + [ + Splice.rustler_helpers(opts), + feature_fragments(features, selected, module_prefix, definition_map, opts), + Entrypoint.fragments(entrypoints) + ] + end - [ - Splice.rustler_helpers(opts), - feature_fragments(features, selected, module_prefix, definition_map, opts), - Entrypoint.fragments(entrypoints) - ] - end + defp feature_fragments(features, selected, module_prefix, definition_map, opts) do + shared_sparse_skip? = + :sparse in features and :skip in features and + Keyword.get(opts, :sparse_messages, :match) == :descriptor + + Enum.flat_map(features, fn + :full -> + Definition.fragments(selected, module_prefix, definition_map) + + :sparse -> + Sparse.fragments(selected, module_prefix, definition_map, + full?: :full in features, + message_mode: + if(shared_sparse_skip?, + do: :descriptor_with_skip, + else: Keyword.get(opts, :sparse_messages, :match) + ) + ) + + :records -> + Records.fragments(selected, module_prefix) + + :skip -> + Skip.fragments(selected, definition_map, messages?: not shared_sparse_skip?) + + feature -> + raise ArgumentError, "unknown Rustler generator feature #{inspect(feature)}" + end) + end - defp feature_fragments(features, selected, module_prefix, definition_map, opts) do - shared_sparse_skip? = - :sparse in features and :skip in features and - Keyword.get(opts, :sparse_messages, :match) == :descriptor - - Enum.flat_map(features, fn - :full -> - Definition.fragments(selected, module_prefix, definition_map) - - :sparse -> - Sparse.fragments(selected, module_prefix, definition_map, - full?: :full in features, - message_mode: - if(shared_sparse_skip?, - do: :descriptor_with_skip, - else: Keyword.get(opts, :sparse_messages, :match) - ) - ) - - :records -> - Records.fragments(selected, module_prefix) - - :skip -> - Skip.fragments(selected, definition_map, messages?: not shared_sparse_skip?) - - feature -> - raise ArgumentError, "unknown Rustler generator feature #{inspect(feature)}" - end) - end + defp normalize_entrypoints(:all, %Schema{} = schema) do + schema.definitions + |> Enum.map(& &1.name) + |> normalize_entrypoints(schema) + end - defp normalize_entrypoints(:all, %Schema{} = schema) do - schema.definitions - |> Enum.map(& &1.name) - |> normalize_entrypoints(schema) - end + defp normalize_entrypoints({:nif_stubs, module}, %Schema{} = schema) when is_atom(module) do + Code.ensure_loaded!(module) - defp normalize_entrypoints({:nif_stubs, module}, %Schema{} = schema) when is_atom(module) do - Code.ensure_loaded!(module) + unless function_exported?(module, :stubs, 0) do + raise ArgumentError, "expected #{inspect(module)} to expose stubs/0" + end - unless function_exported?(module, :stubs, 0) do - raise ArgumentError, "expected #{inspect(module)} to expose stubs/0" + module.stubs() |> entrypoints_from_exports(schema) end - module.stubs() |> entrypoints_from_exports(schema) - end - - defp normalize_entrypoints(module, %Schema{} = schema) when is_atom(module) do - Code.ensure_loaded!(module) - module.module_info(:exports) |> entrypoints_from_exports(schema) - end + defp normalize_entrypoints(module, %Schema{} = schema) when is_atom(module) do + Code.ensure_loaded!(module) + module.module_info(:exports) |> entrypoints_from_exports(schema) + end - defp normalize_entrypoints(entrypoints, %Schema{}) do - Enum.map(entrypoints, fn - {nif_name, definition_name} -> {nif_name, to_string(definition_name)} - definition_name -> {inferred_entrypoint_name(definition_name), to_string(definition_name)} - end) - end + defp normalize_entrypoints(entrypoints, %Schema{}) do + Enum.map(entrypoints, fn + {nif_name, definition_name} -> {nif_name, to_string(definition_name)} + definition_name -> {inferred_entrypoint_name(definition_name), to_string(definition_name)} + end) + end - defp entrypoints_from_exports(exports, %Schema{} = schema) do - definitions_by_underscore = Map.new(schema.definitions, &{Macro.underscore(&1.name), &1.name}) + defp entrypoints_from_exports(exports, %Schema{} = schema) do + definitions_by_underscore = + Map.new(schema.definitions, &{Macro.underscore(&1.name), &1.name}) - exports - |> Enum.flat_map(fn - {name, 1} -> entrypoint_from_export(name, definitions_by_underscore) - _export -> [] - end) - |> Enum.sort_by(fn {nif_name, _definition_name} -> to_string(nif_name) end) - end + exports + |> Enum.flat_map(fn + {name, 1} -> entrypoint_from_export(name, definitions_by_underscore) + _export -> [] + end) + |> Enum.sort_by(fn {nif_name, _definition_name} -> to_string(nif_name) end) + end - defp entrypoint_from_export(name, definitions_by_underscore) do - name = to_string(name) + defp entrypoint_from_export(name, definitions_by_underscore) do + name = to_string(name) - with "decode_" <> suffix <- name, - {:ok, definition_name} <- Map.fetch(definitions_by_underscore, suffix) do - [{name, definition_name}] - else - _other -> [] + with "decode_" <> suffix <- name, + {:ok, definition_name} <- Map.fetch(definitions_by_underscore, suffix) do + [{name, definition_name}] + else + _other -> [] + end end - end - defp inferred_entrypoint_name(definition_name) do - definition_name - |> to_string() - |> Macro.underscore() - |> then(&"decode_#{&1}") - end + defp inferred_entrypoint_name(definition_name) do + definition_name + |> to_string() + |> Macro.underscore() + |> then(&"decode_#{&1}") + end - defp inferred_definitions([]), do: [] + defp inferred_definitions([]), do: [] - defp inferred_definitions(entrypoints) do - Enum.map(entrypoints, fn {_nif_name, definition_name} -> definition_name end) - end + defp inferred_definitions(entrypoints) do + Enum.map(entrypoints, fn {_nif_name, definition_name} -> definition_name end) + end - defp source_prelude(opts) do - decoder = Keyword.get(opts, :decoder, "crate::runtime::Decoder") + defp source_prelude(opts) do + decoder = Keyword.get(opts, :decoder, "crate::runtime::Decoder") - [ - A.use([:rustler, :types, :atom, :Atom]), - A.use({[:rustler], [:Binary, :Encoder, :Env, :Error, :NifResult, :Term]}), - A.use({[:std, :sync], [:OnceLock]}), - A.use(decoder) - ] + [ + A.use([:rustler, :types, :atom, :Atom]), + A.use({[:rustler], [:Binary, :Encoder, :Env, :Error, :NifResult, :Term]}), + A.use({[:std, :sync], [:OnceLock]}), + A.use(decoder) + ] + end end end diff --git a/codegen/kiwi_codec/rustler_generator/atom_cache.ex b/codegen/kiwi_codec/rustler_generator/atom_cache.ex index 7e91f8d..2ff35aa 100644 --- a/codegen/kiwi_codec/rustler_generator/atom_cache.ex +++ b/codegen/kiwi_codec/rustler_generator/atom_cache.ex @@ -1,29 +1,31 @@ -defmodule KiwiCodec.RustlerGenerator.AtomCache do - use RustQ.Meta +if Code.ensure_loaded?(RustQ) do + defmodule KiwiCodec.RustlerGenerator.AtomCache do + use RustQ.Meta - @moduledoc """ - Interns the atoms generated sparse decoders put in every map: the module name, the - `__kiwi_module__` key, and each field name. + @moduledoc """ + Interns the atoms generated sparse decoders put in every map: the module name, the + `__kiwi_module__` key, and each field name. - Creating an atom from its name takes the VM's atom table lock and hashes the text, which - dominated decoding large files. `kiwi_atom(env, name)` does it once per name and scheduler - thread. - """ + Creating an atom from its name takes the VM's atom table lock and hashes the text, which + dominated decoding large files. `kiwi_atom(env, name)` does it once per name and scheduler + thread. + """ - alias RustQ.Type, as: R + alias RustQ.Type, as: R - @template Path.expand("templates/atom_cache.rs", __DIR__) - @external_resource @template + @template Path.expand("templates/atom_cache.rs", __DIR__) + @external_resource @template - @doc "Returns the Rust items defining `kiwi_atom`." - @spec fragments() :: [RustQ.Rust.Fragment.t()] - def fragments do - [RustQ.Rust.fragment(:item, @template |> File.read!() |> String.trim_trailing())] - end + @doc "Returns the Rust items defining `kiwi_atom`." + @spec fragments() :: [RustQ.Rust.Fragment.t()] + def fragments do + [RustQ.Rust.fragment(:item, @template |> File.read!() |> String.trim_trailing())] + end - # Typed declaration for callers; the emitted definition is the template. - @spec kiwi_atom(R.path(:Env, R.lifetime(:_)), R.raw(:"&'static str")) :: R.path(:Atom) - defrust kiwi_atom(env, name) do - Atom.from_str(env, name).unwrap() + # Typed declaration for callers; the emitted definition is the template. + @spec kiwi_atom(R.path(:Env, R.lifetime(:_)), R.raw(:"&'static str")) :: R.path(:Atom) + defrust kiwi_atom(env, name) do + Atom.from_str(env, name).unwrap() + end end end diff --git a/codegen/kiwi_codec/rustler_generator/decoder.ex b/codegen/kiwi_codec/rustler_generator/decoder.ex index 501295b..c6536af 100644 --- a/codegen/kiwi_codec/rustler_generator/decoder.ex +++ b/codegen/kiwi_codec/rustler_generator/decoder.ex @@ -1,42 +1,44 @@ -defmodule KiwiCodec.RustlerGenerator.Decoder do - @moduledoc """ - The `Decoder<'a>` generated decoders read Kiwi data with. +if Code.ensure_loaded?(RustQ) do + defmodule KiwiCodec.RustlerGenerator.Decoder do + @moduledoc """ + The `Decoder<'a>` generated decoders read Kiwi data with. - The native crate provides it, imported from `crate::runtime::Decoder` unless the - generator's `:decoder` option names another path. `decoder.rs`, beside this module, is a - complete implementation to copy or start from; the tests build every generator feature - against it. It names the runtime schema type `crate::RuntimeSchema`, which a crate using - `unknown_fields: :runtime_schema` defines as `pub use KiwiRuntimeSchema as RuntimeSchema;`, - and any other crate as an empty `pub enum RuntimeSchema {}`. + The native crate provides it, imported from `crate::runtime::Decoder` unless the + generator's `:decoder` option names another path. `decoder.rs`, beside this module, is a + complete implementation to copy or start from; the tests build every generator feature + against it. It names the runtime schema type `crate::RuntimeSchema`, which a crate using + `unknown_fields: :runtime_schema` defines as `pub use KiwiRuntimeSchema as RuntimeSchema;`, + and any other crate as an empty `pub enum RuntimeSchema {}`. - Every feature uses `new(&'a [u8])`, `finish(&self) -> NifResult<()>`, and the primitive - readers, each returning `NifResult`: `read_bool`, `read_byte`, `read_var_uint`, - `read_var_uint64`, `read_var_int`, `read_var_int64`, `read_var_float_value`, - `read_var_float(env)`, `read_string(env)`, `read_byte_array(env)`, `skip_string`, - `skip_byte_array`, and `read_repeated(read)`. + Every feature uses `new(&'a [u8])`, `finish(&self) -> NifResult<()>`, and the primitive + readers, each returning `NifResult`: `read_bool`, `read_byte`, `read_var_uint`, + `read_var_uint64`, `read_var_int`, `read_var_int64`, `read_var_float_value`, + `read_var_float(env)`, `read_string(env)`, `read_byte_array(env)`, `skip_string`, + `skip_byte_array`, and `read_repeated(read)`. - * Records also need `read_byte_array_value`, `read_string_str`, and `remaining`. - * `unknown_fields: :runtime_schema` also needs `runtime_schema(&self) -> - Option<&'a KiwiRuntimeSchema>`, returning `None` to keep the `:error` behavior, and - `read_string_value`. + * Records also need `read_byte_array_value`, `read_string_str`, and `remaining`. + * `unknown_fields: :runtime_schema` also needs `runtime_schema(&self) -> + Option<&'a KiwiRuntimeSchema>`, returning `None` to keep the `:error` behavior, and + `read_string_value`. - The generator reads the method signatures from `decoder.rs` to know which calls return a - `NifResult`. A decoder whose signatures differ is passed as `:decoder_sources`. - """ + The generator reads the method signatures from `decoder.rs` to know which calls return a + `NifResult`. A decoder whose signatures differ is passed as `:decoder_sources`. + """ - @source Path.expand("decoder.rs", __DIR__) - @external_resource @source + @source Path.expand("decoder.rs", __DIR__) + @external_resource @source - @doc "Returns the path of the reference `Decoder` source." - @spec source() :: Path.t() - def source, do: @source + @doc "Returns the path of the reference `Decoder` source." + @spec source() :: Path.t() + def source, do: @source - @doc "Returns `decoder_sources`, or the reference `Decoder` source when there are none." - @spec sources([Path.t()] | Path.t()) :: [Path.t()] - def sources(decoder_sources) do - case List.wrap(decoder_sources) do - [] -> [@source] - sources -> sources + @doc "Returns `decoder_sources`, or the reference `Decoder` source when there are none." + @spec sources([Path.t()] | Path.t()) :: [Path.t()] + def sources(decoder_sources) do + case List.wrap(decoder_sources) do + [] -> [@source] + sources -> sources + end end end end diff --git a/codegen/kiwi_codec/rustler_generator/decoder_macro.ex b/codegen/kiwi_codec/rustler_generator/decoder_macro.ex index 6c9f633..e0b4ac6 100644 --- a/codegen/kiwi_codec/rustler_generator/decoder_macro.ex +++ b/codegen/kiwi_codec/rustler_generator/decoder_macro.ex @@ -1,127 +1,129 @@ -defmodule KiwiCodec.RustlerGenerator.DecoderMacro do - @moduledoc """ - Rusty-Elixir macro support for generated Rustler decoders. - - Entrypoint wrappers are authored with `defnif`. Full schema decoders use the - same semantic boundary, but intentionally lower to compact Rust macro - invocations instead of expanded Rust function bodies. This keeps generator - authoring in Elixir/RustQ while preserving small generated source. - """ - - alias KiwiCodec.RustlerGenerator.Full - alias KiwiCodec.RustlerGenerator.Name - alias KiwiCodec.RustlerGenerator.RustExpr - alias KiwiCodec.Schema.Enum, as: SchemaEnum - alias KiwiCodec.Schema.{Message, Struct} - alias RustQ.Rust.Identifier - - @spec enum_decoder(SchemaEnum.t()) :: RustQ.Rust.Fragment.t() - def enum_decoder(%SchemaEnum{} = definition) do - Full.Helpers.macro_call(:kiwi_enum_decoder, - fn: Name.decoder_function(definition.name), - env: :env, - decoder: :decoder, - variants: - Enum.map(definition.variants, fn field -> - [variant_value: field.value, variant_name: Name.field_name(field.name)] - end) - ) - end +if Code.ensure_loaded?(RustQ) do + defmodule KiwiCodec.RustlerGenerator.DecoderMacro do + @moduledoc """ + Rusty-Elixir macro support for generated Rustler decoders. + + Entrypoint wrappers are authored with `defnif`. Full schema decoders use the + same semantic boundary, but intentionally lower to compact Rust macro + invocations instead of expanded Rust function bodies. This keeps generator + authoring in Elixir/RustQ while preserving small generated source. + """ + + alias KiwiCodec.RustlerGenerator.Full + alias KiwiCodec.RustlerGenerator.Name + alias KiwiCodec.RustlerGenerator.RustExpr + alias KiwiCodec.Schema.Enum, as: SchemaEnum + alias KiwiCodec.Schema.{Message, Struct} + alias RustQ.Rust.Identifier + + @spec enum_decoder(SchemaEnum.t()) :: RustQ.Rust.Fragment.t() + def enum_decoder(%SchemaEnum{} = definition) do + Full.Helpers.macro_call(:kiwi_enum_decoder, + fn: Name.decoder_function(definition.name), + env: :env, + decoder: :decoder, + variants: + Enum.map(definition.variants, fn field -> + [variant_value: field.value, variant_name: Name.field_name(field.name)] + end) + ) + end - @spec struct_decoder(Struct.t(), String.t(), map(), (term(), map() -> RustQ.Rust.AST.expr())) :: - RustQ.Rust.Fragment.t() - def struct_decoder(%Struct{} = definition, module_prefix, definition_map, field_expr_fun) do - Full.Helpers.macro_call(:kiwi_struct_decoder, - fn: Name.decoder_function(definition.name), - env: :env, - decoder: :decoder, - module_static: Name.module_atom_static(definition.name), - keys_static: Name.struct_keys_static(definition.name), - module: Name.module_name(module_prefix, definition.name), - keys: key_rows(definition.fields), - fields: field_expr_rows(definition.fields, definition_map, field_expr_fun) - ) - end + @spec struct_decoder(Struct.t(), String.t(), map(), (term(), map() -> RustQ.Rust.AST.expr())) :: + RustQ.Rust.Fragment.t() + def struct_decoder(%Struct{} = definition, module_prefix, definition_map, field_expr_fun) do + Full.Helpers.macro_call(:kiwi_struct_decoder, + fn: Name.decoder_function(definition.name), + env: :env, + decoder: :decoder, + module_static: Name.module_atom_static(definition.name), + keys_static: Name.struct_keys_static(definition.name), + module: Name.module_name(module_prefix, definition.name), + keys: key_rows(definition.fields), + fields: field_expr_rows(definition.fields, definition_map, field_expr_fun) + ) + end - @spec message_decoder(Message.t(), String.t(), map()) :: RustQ.Rust.Fragment.t() - def message_decoder(%Message{} = definition, module_prefix, definition_map) do - Full.Helpers.macro_call(:kiwi_message_decoder, - fn: Name.decoder_function(definition.name), - fields_fn: Name.message_fields_function(definition.name), - env: :env, - decoder: :decoder, - module_static: Name.module_atom_static(definition.name), - keys_static: Name.struct_keys_static(definition.name), - module: Name.module_name(module_prefix, definition.name), - definition: definition.name, - keys: key_rows(definition.fields), - fields: message_field_rows(definition.fields, definition_map) - ) - end + @spec message_decoder(Message.t(), String.t(), map()) :: RustQ.Rust.Fragment.t() + def message_decoder(%Message{} = definition, module_prefix, definition_map) do + Full.Helpers.macro_call(:kiwi_message_decoder, + fn: Name.decoder_function(definition.name), + fields_fn: Name.message_fields_function(definition.name), + env: :env, + decoder: :decoder, + module_static: Name.module_atom_static(definition.name), + keys_static: Name.struct_keys_static(definition.name), + module: Name.module_name(module_prefix, definition.name), + definition: definition.name, + keys: key_rows(definition.fields), + fields: message_field_rows(definition.fields, definition_map) + ) + end - defmacro entrypoint(nif_name, decoder_name) do - nif_name = expand_arg!(nif_name, __CALLER__) - decoder_name = expand_arg!(decoder_name, __CALLER__) - - quote do - @nif schedule: "DirtyCpu" - @spec unquote(nif_name)(binary()) :: R.nif_result(term()) - defnif unquote(nif_name)(bytes) do - decoder = Decoder.new(bytes.as_slice()) - - case unquote(decoder_name)(nif_env(), mut_ref(decoder)) do - {:ok, term} -> - case decoder.finish() do - {:ok, _done} -> {:ok, term} - {:error, reason} -> {:error, reason} - end - - {:error, reason} -> - {:error, reason} + defmacro entrypoint(nif_name, decoder_name) do + nif_name = expand_arg!(nif_name, __CALLER__) + decoder_name = expand_arg!(decoder_name, __CALLER__) + + quote do + @nif schedule: "DirtyCpu" + @spec unquote(nif_name)(binary()) :: R.nif_result(term()) + defnif unquote(nif_name)(bytes) do + decoder = Decoder.new(bytes.as_slice()) + + case unquote(decoder_name)(nif_env(), mut_ref(decoder)) do + {:ok, term} -> + case decoder.finish() do + {:ok, _done} -> {:ok, term} + {:error, reason} -> {:error, reason} + end + + {:error, reason} -> + {:error, reason} + end end end end - end - defp expand_arg!(quoted, caller) do - {value, _binding} = Code.eval_quoted(quoted, [], caller) - value - end + defp expand_arg!(quoted, caller) do + {value, _binding} = Code.eval_quoted(quoted, [], caller) + value + end - defp key_rows(fields) do - Enum.map(fields, &[key: Name.field_name(&1.name)]) - end + defp key_rows(fields) do + Enum.map(fields, &[key: Name.field_name(&1.name)]) + end - defp field_expr_rows(fields, definition_map, field_expr_fun) do - Enum.map(fields, &[field_expr: field_expr_fun.(&1, definition_map)]) - end + defp field_expr_rows(fields, definition_map, field_expr_fun) do + Enum.map(fields, &[field_expr: field_expr_fun.(&1, definition_map)]) + end - defp message_field_rows(fields, definition_map) do - fields - |> Enum.with_index() - |> Enum.map(fn {field, index} -> - [ - field_id: field.id, - field_index: index + 1, - field_repeated: repeated_field?(field), - field_decode: full_decode_function(field, definition_map) - ] - end) - end + defp message_field_rows(fields, definition_map) do + fields + |> Enum.with_index() + |> Enum.map(fn {field, index} -> + [ + field_id: field.id, + field_index: index + 1, + field_repeated: repeated_field?(field), + field_decode: full_decode_function(field, definition_map) + ] + end) + end - defp repeated_field?(%{array?: true, type: "byte"}), do: false - defp repeated_field?(%{array?: array?}), do: array? + defp repeated_field?(%{array?: true, type: "byte"}), do: false + defp repeated_field?(%{array?: array?}), do: array? - defp full_decode_function(%{array?: true, type: "byte"}, _definition_map), - do: :kiwi_full_bytes_value + defp full_decode_function(%{array?: true, type: "byte"}, _definition_map), + do: :kiwi_full_bytes_value - defp full_decode_function(%{array?: true} = field, definition_map), - do: full_decode_function(%{field | array?: false}, definition_map) + defp full_decode_function(%{array?: true} = field, definition_map), + do: full_decode_function(%{field | array?: false}, definition_map) - defp full_decode_function(%{type: type}, definition_map) do - case RustExpr.primitive(type) do - nil -> Name.decoder_function(Map.fetch!(definition_map, type).name) - _expr -> Identifier.atom!("kiwi_full_#{type}_value") + defp full_decode_function(%{type: type}, definition_map) do + case RustExpr.primitive(type) do + nil -> Name.decoder_function(Map.fetch!(definition_map, type).name) + _expr -> Identifier.atom!("kiwi_full_#{type}_value") + end end end end diff --git a/codegen/kiwi_codec/rustler_generator/definition.ex b/codegen/kiwi_codec/rustler_generator/definition.ex index ec4e0da..30affda 100644 --- a/codegen/kiwi_codec/rustler_generator/definition.ex +++ b/codegen/kiwi_codec/rustler_generator/definition.ex @@ -1,67 +1,71 @@ -defmodule KiwiCodec.RustlerGenerator.Definition do - @moduledoc """ - Generates Rust items for Kiwi schema definitions. - """ +if Code.ensure_loaded?(RustQ) do + defmodule KiwiCodec.RustlerGenerator.Definition do + @moduledoc """ + Generates Rust items for Kiwi schema definitions. + """ - alias KiwiCodec.RustlerGenerator.DecoderMacro - alias KiwiCodec.RustlerGenerator.Name - alias KiwiCodec.RustlerGenerator.RustExpr - alias KiwiCodec.Schema.Enum, as: SchemaEnum - alias KiwiCodec.Schema.{Message, Struct} - alias RustQ.Rust - alias RustQ.Rust.AST.Builder, as: A + alias KiwiCodec.RustlerGenerator.DecoderMacro + alias KiwiCodec.RustlerGenerator.Name + alias KiwiCodec.RustlerGenerator.RustExpr + alias KiwiCodec.Schema.Enum, as: SchemaEnum + alias KiwiCodec.Schema.{Message, Struct} + alias RustQ.Rust + alias RustQ.Rust.AST.Builder, as: A - @spec fragments([KiwiCodec.Schema.definition()], String.t(), map()) :: [RustQ.Rust.Fragment.t()] - def fragments(definitions, module_prefix, definition_map) do - definitions - |> Enum.flat_map(&items(&1, module_prefix, definition_map)) - |> Enum.map(&Rust.to_fragment/1) - end + @spec fragments([KiwiCodec.Schema.definition()], String.t(), map()) :: [ + RustQ.Rust.Fragment.t() + ] + def fragments(definitions, module_prefix, definition_map) do + definitions + |> Enum.flat_map(&items(&1, module_prefix, definition_map)) + |> Enum.map(&Rust.to_fragment/1) + end - defp items(%SchemaEnum{} = definition, _module_prefix, _definition_map) do - [DecoderMacro.enum_decoder(definition)] - end + defp items(%SchemaEnum{} = definition, _module_prefix, _definition_map) do + [DecoderMacro.enum_decoder(definition)] + end - defp items(%Struct{} = definition, module_prefix, definition_map) do - [ - atom_static(Name.module_atom_static(definition.name)), - keys_static(Name.struct_keys_static(definition.name)), - DecoderMacro.struct_decoder(definition, module_prefix, definition_map, &field_expr/2) - ] - end + defp items(%Struct{} = definition, module_prefix, definition_map) do + [ + atom_static(Name.module_atom_static(definition.name)), + keys_static(Name.struct_keys_static(definition.name)), + DecoderMacro.struct_decoder(definition, module_prefix, definition_map, &field_expr/2) + ] + end - defp items(%Message{} = definition, module_prefix, definition_map) do - [ - atom_static(Name.module_atom_static(definition.name)), - keys_static(Name.struct_keys_static(definition.name)), - DecoderMacro.message_decoder(definition, module_prefix, definition_map) - ] - end + defp items(%Message{} = definition, module_prefix, definition_map) do + [ + atom_static(Name.module_atom_static(definition.name)), + keys_static(Name.struct_keys_static(definition.name)), + DecoderMacro.message_decoder(definition, module_prefix, definition_map) + ] + end - defp atom_static(name) do - A.static(name, "OnceLock", A.path_call([:OnceLock, :new])) - end + defp atom_static(name) do + A.static(name, "OnceLock", A.path_call([:OnceLock, :new])) + end - defp keys_static(name) do - A.static(name, "OnceLock>", A.path_call([:OnceLock, :new])) - end + defp keys_static(name) do + A.static(name, "OnceLock>", A.path_call([:OnceLock, :new])) + end - defp field_expr(%{array?: true, type: "byte"}, _definition_map) do - A.try(A.method(:decoder, :read_byte_array, [:env])) - end + defp field_expr(%{array?: true, type: "byte"}, _definition_map) do + A.try(A.method(:decoder, :read_byte_array, [:env])) + end - defp field_expr(%{array?: true, type: type}, definition_map) do - read_item = A.closure([:decoder], read(type, definition_map)) - A.try(A.method(:decoder, :read_repeated, [read_item])) - end + defp field_expr(%{array?: true, type: type}, definition_map) do + read_item = A.closure([:decoder], read(type, definition_map)) + A.try(A.method(:decoder, :read_repeated, [read_item])) + end - defp field_expr(%{type: type}, definition_map), do: A.try(read(type, definition_map)) + defp field_expr(%{type: type}, definition_map), do: A.try(read(type, definition_map)) - defp read(type, definition_map) do - if KiwiCodec.PrimitiveType.name?(type) do - RustExpr.read(type) - else - A.call(Name.decoder_function(Map.fetch!(definition_map, type).name), [:env, :decoder]) + defp read(type, definition_map) do + if KiwiCodec.PrimitiveType.name?(type) do + RustExpr.read(type) + else + A.call(Name.decoder_function(Map.fetch!(definition_map, type).name), [:env, :decoder]) + end end end end diff --git a/codegen/kiwi_codec/rustler_generator/entrypoint.ex b/codegen/kiwi_codec/rustler_generator/entrypoint.ex index 519ca4f..0bcfc60 100644 --- a/codegen/kiwi_codec/rustler_generator/entrypoint.ex +++ b/codegen/kiwi_codec/rustler_generator/entrypoint.ex @@ -1,48 +1,50 @@ -defmodule KiwiCodec.RustlerGenerator.Entrypoint do - @moduledoc """ - Generates Rustler NIF entrypoint items for Kiwi decoders. - """ +if Code.ensure_loaded?(RustQ) do + defmodule KiwiCodec.RustlerGenerator.Entrypoint do + @moduledoc """ + Generates Rustler NIF entrypoint items for Kiwi decoders. + """ - alias KiwiCodec.RustlerGenerator.Name - alias RustQ.Meta.AST, as: MetaAST - alias RustQ.Rust.Identifier + alias KiwiCodec.RustlerGenerator.Name + alias RustQ.Meta.AST, as: MetaAST + alias RustQ.Rust.Identifier - @type entrypoint :: {atom() | String.t(), String.t()} + @type entrypoint :: {atom() | String.t(), String.t()} - @spec fragments([entrypoint()]) :: [RustQ.Rust.Fragment.t()] - def fragments(entrypoints), do: Enum.map(entrypoints, &item/1) + @spec fragments([entrypoint()]) :: [RustQ.Rust.Fragment.t()] + def fragments(entrypoints), do: Enum.map(entrypoints, &item/1) - defp item({nif_name, definition_name}) do - module = generated_module!(nif_name, definition_name) - MetaAST.function!(module, Identifier.atom!(to_string(nif_name))) - end + defp item({nif_name, definition_name}) do + module = generated_module!(nif_name, definition_name) + MetaAST.function!(module, Identifier.atom!(to_string(nif_name))) + end - defp generated_module!(nif_name, definition_name) do - nif_name = Identifier.atom!(to_string(nif_name)) - decoder_name = Name.decoder_function(definition_name) - - module = - Module.concat([ - KiwiCodec.RustlerGenerator.Generated, - "Entrypoint#{nif_name}#{:erlang.phash2(definition_name)}" - ]) - - if Code.ensure_loaded?(module) do - module - else - Module.create( - module, - quote do - use RustQ.Meta - alias RustQ.Type, as: R - import KiwiCodec.RustlerGenerator.DecoderMacro, only: [entrypoint: 2] - - entrypoint(unquote(nif_name), unquote(decoder_name)) - end, - Macro.Env.location(__ENV__) - ) - - module + defp generated_module!(nif_name, definition_name) do + nif_name = Identifier.atom!(to_string(nif_name)) + decoder_name = Name.decoder_function(definition_name) + + module = + Module.concat([ + KiwiCodec.RustlerGenerator.Generated, + "Entrypoint#{nif_name}#{:erlang.phash2(definition_name)}" + ]) + + if Code.ensure_loaded?(module) do + module + else + Module.create( + module, + quote do + use RustQ.Meta + alias RustQ.Type, as: R + import KiwiCodec.RustlerGenerator.DecoderMacro, only: [entrypoint: 2] + + entrypoint(unquote(nif_name), unquote(decoder_name)) + end, + Macro.Env.location(__ENV__) + ) + + module + end end end end diff --git a/codegen/kiwi_codec/rustler_generator/full/helpers.ex b/codegen/kiwi_codec/rustler_generator/full/helpers.ex index 5730dc0..e78e5ec 100644 --- a/codegen/kiwi_codec/rustler_generator/full/helpers.ex +++ b/codegen/kiwi_codec/rustler_generator/full/helpers.ex @@ -1,355 +1,357 @@ -defmodule KiwiCodec.RustlerGenerator.Full.Helpers do - use RustQ.Meta, - rust_sources: [KiwiCodec.RustlerGenerator.Decoder.source()], - callable_modules: [ - RustQ.Rustler.Atom, - RustQ.Rustler.Term, - KiwiCodec.RustlerGenerator.UnknownFields +if Code.ensure_loaded?(RustQ) do + defmodule KiwiCodec.RustlerGenerator.Full.Helpers do + use RustQ.Meta, + rust_sources: [KiwiCodec.RustlerGenerator.Decoder.source()], + callable_modules: [ + RustQ.Rustler.Atom, + RustQ.Rustler.Term, + KiwiCodec.RustlerGenerator.UnknownFields + ] + + @moduledoc """ + RustQ-authored full Kiwi decoder macro helpers. + """ + + alias RustQ.Meta.AST, as: MetaAST + alias RustQ.Type, as: R + + @macros [ + :kiwi_enum_decoder, + :kiwi_struct_decoder, + :kiwi_message_decoder ] - @moduledoc """ - RustQ-authored full Kiwi decoder macro helpers. - """ - - alias RustQ.Meta.AST, as: MetaAST - alias RustQ.Type, as: R - - @macros [ - :kiwi_enum_decoder, - :kiwi_struct_decoder, - :kiwi_message_decoder - ] - - @functions [ - :kiwi_enum_value, - :kiwi_full_bool_value, - :kiwi_full_byte_value, - :kiwi_full_float_value, - :kiwi_full_int_value, - :kiwi_full_int64_value, - :kiwi_full_string_value, - :kiwi_full_uint_value, - :kiwi_full_uint64_value, - :kiwi_full_bytes_value, - :kiwi_message_fields, - :kiwi_full_field_value - ] - - @typedoc "The Rust `KiwiFullDecodeFn`: decodes one value of a type into a term." - @type kiwi_full_decode_fn :: - R.raw(:"for<'a> fn(Env<'a>, &mut Decoder<'_>) -> NifResult>") - - @typedoc """ - The Rust `KiwiFullField`: a message field id, its index among the struct's values, and how to - decode it. - """ - @type kiwi_full_field :: %{ - required(:id) => R.u32(), - required(:index) => R.usize(), - required(:repeated) => R.bool(), - required(:decode) => kiwi_full_decode_fn() - } - - @typedoc "The Rust `KiwiEnumVariant`: an enum value and the atom name it decodes to." - @type kiwi_enum_variant :: %{ - required(:value) => R.u32(), - required(:name) => R.raw(:"&'static str") - } - - @spec fragments([atom()]) :: [RustQ.Rust.Fragment.t()] - def fragments(macros \\ @macros) do - __MODULE__.__rustq_type_items__() ++ - MetaAST.macro_items!(__MODULE__, macros) ++ MetaAST.functions!(__MODULE__, @functions) - end + @functions [ + :kiwi_enum_value, + :kiwi_full_bool_value, + :kiwi_full_byte_value, + :kiwi_full_float_value, + :kiwi_full_int_value, + :kiwi_full_int64_value, + :kiwi_full_string_value, + :kiwi_full_uint_value, + :kiwi_full_uint64_value, + :kiwi_full_bytes_value, + :kiwi_message_fields, + :kiwi_full_field_value + ] + + @typedoc "The Rust `KiwiFullDecodeFn`: decodes one value of a type into a term." + @type kiwi_full_decode_fn :: + R.raw(:"for<'a> fn(Env<'a>, &mut Decoder<'_>) -> NifResult>") + + @typedoc """ + The Rust `KiwiFullField`: a message field id, its index among the struct's values, and how to + decode it. + """ + @type kiwi_full_field :: %{ + required(:id) => R.u32(), + required(:index) => R.usize(), + required(:repeated) => boolean(), + required(:decode) => kiwi_full_decode_fn() + } + + @typedoc "The Rust `KiwiEnumVariant`: an enum value and the atom name it decodes to." + @type kiwi_enum_variant :: %{ + required(:value) => R.u32(), + required(:name) => R.raw(:"&'static str") + } + + @spec fragments([atom()]) :: [RustQ.Rust.Fragment.t()] + def fragments(macros \\ @macros) do + __MODULE__.__rustq_type_items__() ++ + MetaAST.macro_items!(__MODULE__, macros) ++ MetaAST.functions!(__MODULE__, @functions) + end + + @spec macro_call(atom(), keyword()) :: RustQ.Rust.Fragment.t() + def macro_call(name, args), do: MetaAST.macro_call!(__MODULE__, name, args) + + defrustmacro kiwi_struct_decoder( + fn: name(:ident), + env: env(:ident), + decoder: decoder(:ident), + module_static: module_static(:ident), + keys_static: keys_static(:ident), + module: module_name(:literal), + keys: + repeat do + key(:literal) + end, + fields: + repeat do + field_expr(:expr) + end + ) do + @spec name(R.path(:Env, R.lifetime(:a)), R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: + R.nif_result(term()) + defrust name(env, decoder) do + module_atom = cached_atom(env, ref(module_static), module_name) + + keys = + cached_struct_keys( + env, + ref(keys_static), + ref( + array([ + repeat keys do + key + end + ]) + ) + ) - @spec macro_call(atom(), keyword()) :: RustQ.Rust.Fragment.t() - def macro_call(name, args), do: MetaAST.macro_call!(__MODULE__, name, args) - - defrustmacro kiwi_struct_decoder( - fn: name(:ident), - env: env(:ident), - decoder: decoder(:ident), - module_static: module_static(:ident), - keys_static: keys_static(:ident), - module: module_name(:literal), - keys: - repeat do - key(:literal) - end, - fields: - repeat do - field_expr(:expr) - end - ) do - @spec name(R.path(:Env, R.lifetime(:a)), R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: - R.nif_result(term()) - defrust name(env, decoder) do - module_atom = cached_atom(env, ref(module_static), module_name) - - keys = - cached_struct_keys( + make_struct_from_arrays( env, - ref(keys_static), + keys, ref( array([ - repeat keys do - key + module_atom.to_term(env), + repeat fields do + field_expr.encode(env) end ]) ) ) + end + end + + defrustmacro kiwi_message_decoder( + fn: decoder_name(:ident), + fields_fn: fields_name(:ident), + env: env(:ident), + decoder: decoder(:ident), + module_static: module_static(:ident), + keys_static: keys_static(:ident), + module: module_name(:literal), + definition: definition_name(:literal), + keys: + repeat do + key(:literal) + end, + fields: + repeat do + field_id(:literal) + field_index(:literal) + field_repeated(:literal) + field_decode(:ident) + end + ) do + @spec decoder_name( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))) + ) :: R.nif_result(term()) + defrust decoder_name(env, decoder) do + module_atom = cached_atom(env, ref(module_static), module_name) + + keys = + cached_struct_keys( + env, + ref(keys_static), + ref( + array([ + repeat keys do + key + end + ]) + ) + ) - make_struct_from_arrays( - env, - keys, - ref( + values = default_struct_values(env, module_atom, keys.len() - 1) + fields_name(env, decoder, keys, values) + end + + @spec fields_name( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))), + R.slice(R.path(:Atom)), + R.vec(term()) + ) :: R.nif_result(term()) + defrust fields_name(env, decoder, struct_keys, struct_values) do + kiwi_message_fields( + env, + decoder, + definition_name, + struct_keys, + struct_values, array([ - module_atom.to_term(env), repeat fields do - field_expr.encode(env) + struct_literal(KiwiFullField, + id: field_id, + index: field_index, + repeated: field_repeated, + decode: field_decode + ) end ]) ) - ) + end end - end - - defrustmacro kiwi_message_decoder( - fn: decoder_name(:ident), - fields_fn: fields_name(:ident), - env: env(:ident), - decoder: decoder(:ident), - module_static: module_static(:ident), - keys_static: keys_static(:ident), - module: module_name(:literal), - definition: definition_name(:literal), - keys: - repeat do - key(:literal) - end, - fields: - repeat do - field_id(:literal) - field_index(:literal) - field_repeated(:literal) - field_decode(:ident) - end - ) do - @spec decoder_name( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))) - ) :: R.nif_result(term()) - defrust decoder_name(env, decoder) do - module_atom = cached_atom(env, ref(module_static), module_name) - keys = - cached_struct_keys( + defrustmacro kiwi_enum_decoder( + fn: name(:ident), + env: env(:ident), + decoder: decoder(:ident), + variants: + repeat do + variant_value(:literal) + variant_name(:literal) + end + ) do + @spec name(R.path(:Env, R.lifetime(:a)), R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: + R.nif_result(term()) + defrust name(env, decoder) do + kiwi_enum_value( env, - ref(keys_static), - ref( - array([ - repeat keys do - key - end - ]) - ) + decoder, + array([ + repeat variants do + struct_literal(KiwiEnumVariant, + value: variant_value, + name: variant_name + ) + end + ]) ) + end + end - values = default_struct_values(env, module_atom, keys.len() - 1) - fields_name(env, decoder, keys, values) + @spec kiwi_full_bool_value( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))) + ) :: R.nif_result(term()) + defrust kiwi_full_bool_value(env, decoder) do + value = decoder.read_bool() + {:ok, value.encode(env)} end - @spec fields_name( + @spec kiwi_full_byte_value( R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))), - R.slice(R.path(:Atom)), - R.vec(term()) + R.mut_ref(R.path(:Decoder, R.lifetime(:_))) ) :: R.nif_result(term()) - defrust fields_name(env, decoder, struct_keys, struct_values) do - kiwi_message_fields( - env, - decoder, - definition_name, - struct_keys, - struct_values, - array([ - repeat fields do - struct_literal(KiwiFullField, - id: field_id, - index: field_index, - repeated: field_repeated, - decode: field_decode - ) - end - ]) - ) + defrust kiwi_full_byte_value(env, decoder) do + value = decoder.read_byte() + {:ok, value.encode(env)} end - end - defrustmacro kiwi_enum_decoder( - fn: name(:ident), - env: env(:ident), - decoder: decoder(:ident), - variants: - repeat do - variant_value(:literal) - variant_name(:literal) - end - ) do - @spec name(R.path(:Env, R.lifetime(:a)), R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: - R.nif_result(term()) - defrust name(env, decoder) do - kiwi_enum_value( - env, - decoder, - array([ - repeat variants do - struct_literal(KiwiEnumVariant, - value: variant_value, - name: variant_name - ) - end - ]) - ) + @spec kiwi_full_float_value( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))) + ) :: R.nif_result(term()) + defrust kiwi_full_float_value(env, decoder) do + decoder.read_var_float(env) end - end - @spec kiwi_full_bool_value( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))) - ) :: R.nif_result(term()) - defrust kiwi_full_bool_value(env, decoder) do - value = decoder.read_bool() - {:ok, value.encode(env)} - end + @spec kiwi_full_int_value( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))) + ) :: R.nif_result(term()) + defrust kiwi_full_int_value(env, decoder) do + value = decoder.read_var_int() + {:ok, value.encode(env)} + end - @spec kiwi_full_byte_value( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))) - ) :: R.nif_result(term()) - defrust kiwi_full_byte_value(env, decoder) do - value = decoder.read_byte() - {:ok, value.encode(env)} - end + @spec kiwi_full_int64_value( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))) + ) :: R.nif_result(term()) + defrust kiwi_full_int64_value(env, decoder) do + value = decoder.read_var_int64() + {:ok, value.encode(env)} + end - @spec kiwi_full_float_value( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))) - ) :: R.nif_result(term()) - defrust kiwi_full_float_value(env, decoder) do - decoder.read_var_float(env) - end + @spec kiwi_full_string_value( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))) + ) :: R.nif_result(term()) + defrust kiwi_full_string_value(env, decoder) do + decoder.read_string(env) + end - @spec kiwi_full_int_value( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))) - ) :: R.nif_result(term()) - defrust kiwi_full_int_value(env, decoder) do - value = decoder.read_var_int() - {:ok, value.encode(env)} - end + @spec kiwi_full_uint_value( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))) + ) :: R.nif_result(term()) + defrust kiwi_full_uint_value(env, decoder) do + value = decoder.read_var_uint() + {:ok, value.encode(env)} + end - @spec kiwi_full_int64_value( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))) - ) :: R.nif_result(term()) - defrust kiwi_full_int64_value(env, decoder) do - value = decoder.read_var_int64() - {:ok, value.encode(env)} - end + @spec kiwi_full_uint64_value( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))) + ) :: R.nif_result(term()) + defrust kiwi_full_uint64_value(env, decoder) do + value = decoder.read_var_uint64() + {:ok, value.encode(env)} + end - @spec kiwi_full_string_value( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))) - ) :: R.nif_result(term()) - defrust kiwi_full_string_value(env, decoder) do - decoder.read_string(env) - end + @spec kiwi_full_bytes_value( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))) + ) :: R.nif_result(term()) + defrust kiwi_full_bytes_value(env, decoder) do + decoder.read_byte_array(env) + end - @spec kiwi_full_uint_value( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))) - ) :: R.nif_result(term()) - defrust kiwi_full_uint_value(env, decoder) do - value = decoder.read_var_uint() - {:ok, value.encode(env)} - end + @spec kiwi_message_fields( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))), + R.str(), + R.slice(R.path(:Atom)), + R.vec(term()), + R.slice(R.path(:KiwiFullField)) + ) :: R.nif_result(term()) + defrust kiwi_message_fields(env, decoder, definition_name, keys, values, fields) do + field_id = decoder.read_var_uint() + + if field_id == 0 do + make_struct_from_arrays(env, keys, ref(values)) + else + case fields.binary_search_by_key(field_id, fn field -> field.id end) do + {:ok, index} -> + field = fields.get(index).unwrap() + assign!(index(values, field.index), kiwi_full_field_value(env, decoder, field)) + kiwi_message_fields(env, decoder, definition_name, keys, values, fields) + + {:error, _index} -> + kiwi_unknown_field(decoder, definition_name, field_id) + kiwi_message_fields(env, decoder, definition_name, keys, values, fields) + end + end + end - @spec kiwi_full_uint64_value( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))) - ) :: R.nif_result(term()) - defrust kiwi_full_uint64_value(env, decoder) do - value = decoder.read_var_uint64() - {:ok, value.encode(env)} - end + @spec kiwi_full_field_value( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))), + R.ref(R.path(:KiwiFullField)) + ) :: R.nif_result(term()) + defrust kiwi_full_field_value(env, decoder, field) do + decode = field.decode + + if field.repeated do + values = decoder.read_repeated(fn decoder -> decode(env, decoder) end) + {:ok, values.encode(env)} + else + decode(env, decoder) + end + end - @spec kiwi_full_bytes_value( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))) - ) :: R.nif_result(term()) - defrust kiwi_full_bytes_value(env, decoder) do - decoder.read_byte_array(env) - end + @spec kiwi_enum_value( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))), + R.slice(R.path(:KiwiEnumVariant)) + ) :: R.nif_result(term()) + defrust kiwi_enum_value(env, decoder, variants) do + value = decoder.read_var_uint() - @spec kiwi_message_fields( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))), - R.str(), - R.slice(R.path(:Atom)), - R.vec(term()), - R.slice(R.path(:KiwiFullField)) - ) :: R.nif_result(term()) - defrust kiwi_message_fields(env, decoder, definition_name, keys, values, fields) do - field_id = decoder.read_var_uint() - - if field_id == 0 do - make_struct_from_arrays(env, keys, ref(values)) - else - case fields.binary_search_by_key(field_id, fn field -> field.id end) do + case variants.binary_search_by_key(value, fn variant -> variant.value end) do {:ok, index} -> - field = fields.get(index).unwrap() - assign!(index(values, field.index), kiwi_full_field_value(env, decoder, field)) - kiwi_message_fields(env, decoder, definition_name, keys, values, fields) + variant = variants.get(index).unwrap() + {:ok, Atom.from_str(env, variant.name).unwrap().encode(env)} {:error, _index} -> - kiwi_unknown_field(decoder, definition_name, field_id) - kiwi_message_fields(env, decoder, definition_name, keys, values, fields) + {:ok, value.encode(env)} end end end - - @spec kiwi_full_field_value( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))), - R.ref(R.path(:KiwiFullField)) - ) :: R.nif_result(term()) - defrust kiwi_full_field_value(env, decoder, field) do - decode = field.decode - - if field.repeated do - values = decoder.read_repeated(fn decoder -> decode(env, decoder) end) - {:ok, values.encode(env)} - else - decode(env, decoder) - end - end - - @spec kiwi_enum_value( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))), - R.slice(R.path(:KiwiEnumVariant)) - ) :: R.nif_result(term()) - defrust kiwi_enum_value(env, decoder, variants) do - value = decoder.read_var_uint() - - case variants.binary_search_by_key(value, fn variant -> variant.value end) do - {:ok, index} -> - variant = variants.get(index).unwrap() - {:ok, Atom.from_str(env, variant.name).unwrap().encode(env)} - - {:error, _index} -> - {:ok, value.encode(env)} - end - end end diff --git a/codegen/kiwi_codec/rustler_generator/name.ex b/codegen/kiwi_codec/rustler_generator/name.ex index ea99cea..c5bb458 100644 --- a/codegen/kiwi_codec/rustler_generator/name.ex +++ b/codegen/kiwi_codec/rustler_generator/name.ex @@ -1,55 +1,57 @@ -defmodule KiwiCodec.RustlerGenerator.Name do - @moduledoc """ - Naming helpers for Rustler decoder code generated from Kiwi schemas. - """ - - alias RustQ.Rust.Identifier - - @spec decoder_function(String.t()) :: atom() - def decoder_function(name) do - name - |> decoder_base() - |> Kernel.<>("_from_decoder") - |> Identifier.atom!() - end - - @spec message_fields_function(String.t()) :: atom() - def message_fields_function(name) do - name - |> decoder_base() - |> Kernel.<>("_fields_from_decoder") - |> Identifier.atom!() - end - - @spec module_atom_static(String.t()) :: atom() - def module_atom_static(name), do: static_name(name, "MODULE_ATOM") - - @spec struct_keys_static(String.t()) :: atom() - def struct_keys_static(name), do: static_name(name, "STRUCT_KEYS") - - @spec static_alias(atom()) :: Macro.t() - def static_alias(name), do: {:__aliases__, [], [name]} - - @spec module_name(String.t(), String.t()) :: String.t() - def module_name(module_prefix, name), do: "Elixir.#{module_prefix}.#{name}" - - @spec field_name(String.t()) :: String.t() - def field_name(name), do: Macro.underscore(name) - - @doc """ - Returns the snake_case Rust identifier for a schema name. Schema names are identifiers, and - a name that is not, as a schema decoded from binary may hold, fails where it is rendered. - """ - @spec rust_identifier(String.t()) :: String.t() - def rust_identifier(name), do: Macro.underscore(name) - - defp decoder_base(name), do: "decode_#{rust_identifier(name)}" - - defp static_name(name, suffix) do - name - |> rust_identifier() - |> String.upcase() - |> Kernel.<>("_#{suffix}") - |> Identifier.atom!() +if Code.ensure_loaded?(RustQ) do + defmodule KiwiCodec.RustlerGenerator.Name do + @moduledoc """ + Naming helpers for Rustler decoder code generated from Kiwi schemas. + """ + + alias RustQ.Rust.Identifier + + @spec decoder_function(String.t()) :: atom() + def decoder_function(name) do + name + |> decoder_base() + |> Kernel.<>("_from_decoder") + |> Identifier.atom!() + end + + @spec message_fields_function(String.t()) :: atom() + def message_fields_function(name) do + name + |> decoder_base() + |> Kernel.<>("_fields_from_decoder") + |> Identifier.atom!() + end + + @spec module_atom_static(String.t()) :: atom() + def module_atom_static(name), do: static_name(name, "MODULE_ATOM") + + @spec struct_keys_static(String.t()) :: atom() + def struct_keys_static(name), do: static_name(name, "STRUCT_KEYS") + + @spec static_alias(atom()) :: Macro.t() + def static_alias(name), do: {:__aliases__, [], [name]} + + @spec module_name(String.t(), String.t()) :: String.t() + def module_name(module_prefix, name), do: "Elixir.#{module_prefix}.#{name}" + + @spec field_name(String.t()) :: String.t() + def field_name(name), do: Macro.underscore(name) + + @doc """ + Returns the snake_case Rust identifier for a schema name. Schema names are identifiers, and + a name that is not, as a schema decoded from binary may hold, fails where it is rendered. + """ + @spec rust_identifier(String.t()) :: String.t() + def rust_identifier(name), do: Macro.underscore(name) + + defp decoder_base(name), do: "decode_#{rust_identifier(name)}" + + defp static_name(name, suffix) do + name + |> rust_identifier() + |> String.upcase() + |> Kernel.<>("_#{suffix}") + |> Identifier.atom!() + end end end diff --git a/codegen/kiwi_codec/rustler_generator/records.ex b/codegen/kiwi_codec/rustler_generator/records.ex index dbf52cf..ac5c68e 100644 --- a/codegen/kiwi_codec/rustler_generator/records.ex +++ b/codegen/kiwi_codec/rustler_generator/records.ex @@ -1,133 +1,135 @@ -defmodule KiwiCodec.RustlerGenerator.Records do - @moduledoc """ - Generates native Kiwi records: decoded values Rust code can read, change, and share without - building Elixir terms, and encode later as the sparse maps generated decoders return. - - The record runtime is schema-independent. The schema only supplies a definition table, read - by the decoder and encoder, and constants naming each definition, field, and enum value for - code that works with records: - - kiwi_record_definitions::NODE_CHANGE - kiwi_records::node_change::FILL_PAINTS - kiwi_records::node_phase::REMOVED - - Decoding uses the `Decoder` methods `KiwiCodec.RustlerGenerator.Decoder` lists for records. - Unknown message fields go through `kiwi_unknown_field`; see - `KiwiCodec.RustlerGenerator.UnknownFields`. - """ - - alias KiwiCodec.RustlerGenerator.Name - alias KiwiCodec.Schema.Enum, as: SchemaEnum - alias KiwiCodec.Schema.Struct - alias RustQ.Rust.AST.Builder, as: A - alias RustQ.Rust.Identifier - - @template Path.expand("templates/records_runtime.rs", __DIR__) - @external_resource @template - - @primitives ~w(bool byte int uint float string int64 uint64) - - @kinds %{SchemaEnum => 0, Struct => 1, KiwiCodec.Schema.Message => 2} - - @doc "Returns the record runtime, definition table, and constants for `definitions`." - @spec fragments([KiwiCodec.Schema.definition()], String.t()) :: [term()] - def fragments(definitions, module_prefix) do - indexes = - definitions |> Enum.with_index() |> Map.new(fn {def, index} -> {def.name, index} end) - - [ - RustQ.Rust.fragment(:item, @template |> File.read!() |> String.trim_trailing()), - definition_table(definitions, indexes, module_prefix), - constants(definitions, indexes) - ] - end +if Code.ensure_loaded?(RustQ) do + defmodule KiwiCodec.RustlerGenerator.Records do + @moduledoc """ + Generates native Kiwi records: decoded values Rust code can read, change, and share without + building Elixir terms, and encode later as the sparse maps generated decoders return. + + The record runtime is schema-independent. The schema only supplies a definition table, read + by the decoder and encoder, and constants naming each definition, field, and enum value for + code that works with records: + + kiwi_record_definitions::NODE_CHANGE + kiwi_records::node_change::FILL_PAINTS + kiwi_records::node_phase::REMOVED + + Decoding uses the `Decoder` methods `KiwiCodec.RustlerGenerator.Decoder` lists for records. + Unknown message fields go through `kiwi_unknown_field`; see + `KiwiCodec.RustlerGenerator.UnknownFields`. + """ + + alias KiwiCodec.RustlerGenerator.Name + alias KiwiCodec.Schema.Enum, as: SchemaEnum + alias KiwiCodec.Schema.Struct + alias RustQ.Rust.AST.Builder, as: A + alias RustQ.Rust.Identifier + + @template Path.expand("templates/records_runtime.rs", __DIR__) + @external_resource @template + + @primitives ~w(bool byte int uint float string int64 uint64) + + @kinds %{SchemaEnum => 0, Struct => 1, KiwiCodec.Schema.Message => 2} + + @doc "Returns the record runtime, definition table, and constants for `definitions`." + @spec fragments([KiwiCodec.Schema.definition()], String.t()) :: [term()] + def fragments(definitions, module_prefix) do + indexes = + definitions |> Enum.with_index() |> Map.new(fn {def, index} -> {def.name, index} end) + + [ + RustQ.Rust.fragment(:item, @template |> File.read!() |> String.trim_trailing()), + definition_table(definitions, indexes, module_prefix), + constants(definitions, indexes) + ] + end - defp definition_table(definitions, indexes, module_prefix) do - A.static( - :KIWI_RECORD_DEFINITIONS, - {:ref, {:slice, :KiwiRecordDefinition}}, - A.slice(Enum.map(definitions, &definition(&1, indexes, module_prefix))) - ) - end + defp definition_table(definitions, indexes, module_prefix) do + A.static( + :KIWI_RECORD_DEFINITIONS, + {:ref, {:slice, :KiwiRecordDefinition}}, + A.slice(Enum.map(definitions, &definition(&1, indexes, module_prefix))) + ) + end - defp definition(definition, indexes, module_prefix) do - A.struct_expr(:KiwiRecordDefinition, - kind: A.lit(Map.fetch!(@kinds, definition.__struct__)), - name: A.lit(definition.name), - module: A.lit(Name.module_name(module_prefix, definition.name)), - fields: A.slice(definition |> fields() |> Enum.map(&field(&1, indexes))), - variants: A.slice(definition |> variants() |> Enum.map(&variant/1)) - ) - end + defp definition(definition, indexes, module_prefix) do + A.struct_expr(:KiwiRecordDefinition, + kind: A.lit(Map.fetch!(@kinds, definition.__struct__)), + name: A.lit(definition.name), + module: A.lit(Name.module_name(module_prefix, definition.name)), + fields: A.slice(definition |> fields() |> Enum.map(&field(&1, indexes))), + variants: A.slice(definition |> variants() |> Enum.map(&variant/1)) + ) + end - defp field(field, indexes) do - A.struct_expr(:KiwiRecordField, - id: A.lit(field.id), - name: A.lit(Name.field_name(field.name)), - type_index: A.lit(type_index(field.type, indexes)), - array: A.lit(field.array?) - ) - end + defp field(field, indexes) do + A.struct_expr(:KiwiRecordField, + id: A.lit(field.id), + name: A.lit(Name.field_name(field.name)), + type_index: A.lit(type_index(field.type, indexes)), + array: A.lit(field.array?) + ) + end - defp variant(variant) do - A.struct_expr(:KiwiRecordVariant, - value: A.lit(variant.value), - name: A.lit(Macro.underscore(variant.name)) - ) - end + defp variant(variant) do + A.struct_expr(:KiwiRecordVariant, + value: A.lit(variant.value), + name: A.lit(Macro.underscore(variant.name)) + ) + end - # Primitive types are bitwise-negated indexes, definitions their table index. - defp type_index(type, indexes) do - case Enum.find_index(@primitives, &(&1 == type)) do - nil -> Map.fetch!(indexes, type) - primitive -> Bitwise.bnot(primitive) + # Primitive types are bitwise-negated indexes, definitions their table index. + defp type_index(type, indexes) do + case Enum.find_index(@primitives, &(&1 == type)) do + nil -> Map.fetch!(indexes, type) + primitive -> Bitwise.bnot(primitive) + end end - end - # Struct fields keep wire order, which their ids follow; message fields sort by id. - defp fields(%Struct{fields: fields}), do: fields - defp fields(%{fields: fields}), do: Enum.sort_by(fields, & &1.id) - defp fields(_enum), do: [] - - defp variants(%SchemaEnum{variants: variants}), do: Enum.sort_by(variants, & &1.value) - defp variants(_definition), do: [] - - # Definition indexes live apart from field and enum constants, which take any schema name. - defp constants(definitions, indexes) do - [ - A.module( - :kiwi_record_definitions, - Enum.map(definitions, fn definition -> - name = definition.name |> Name.rust_identifier() |> String.upcase() - A.const(name, :u32, A.lit(Map.fetch!(indexes, definition.name)), vis: :pub) - end), - vis: :pub - ), - A.module( - :kiwi_records, - Enum.map(definitions, fn definition -> - A.module( - module_identifier(definition.name), - Enum.map(fields(definition), &field_constant/1) ++ - Enum.map(variants(definition), &variant_constant/1), - vis: :pub - ) - end), - vis: :pub - ) - ] - end + # Struct fields keep wire order, which their ids follow; message fields sort by id. + defp fields(%Struct{fields: fields}), do: fields + defp fields(%{fields: fields}), do: Enum.sort_by(fields, & &1.id) + defp fields(_enum), do: [] + + defp variants(%SchemaEnum{variants: variants}), do: Enum.sort_by(variants, & &1.value) + defp variants(_definition), do: [] + + # Definition indexes live apart from field and enum constants, which take any schema name. + defp constants(definitions, indexes) do + [ + A.module( + :kiwi_record_definitions, + Enum.map(definitions, fn definition -> + name = definition.name |> Name.rust_identifier() |> String.upcase() + A.const(name, :u32, A.lit(Map.fetch!(indexes, definition.name)), vis: :pub) + end), + vis: :pub + ), + A.module( + :kiwi_records, + Enum.map(definitions, fn definition -> + A.module( + module_identifier(definition.name), + Enum.map(fields(definition), &field_constant/1) ++ + Enum.map(variants(definition), &variant_constant/1), + vis: :pub + ) + end), + vis: :pub + ) + ] + end - defp field_constant(field) do - name = field.name |> Name.rust_identifier() |> String.upcase() - A.const(name, :u32, A.lit(field.id), vis: :pub) - end + defp field_constant(field) do + name = field.name |> Name.rust_identifier() |> String.upcase() + A.const(name, :u32, A.lit(field.id), vis: :pub) + end - defp variant_constant(variant) do - name = variant.name |> Name.rust_identifier() |> String.upcase() - A.const(name, :u32, A.lit(variant.value), vis: :pub) - end + defp variant_constant(variant) do + name = variant.name |> Name.rust_identifier() |> String.upcase() + A.const(name, :u32, A.lit(variant.value), vis: :pub) + end - # A keyword, such as the module for a `Type` definition, renders as a raw identifier. - defp module_identifier(name), do: name |> Name.rust_identifier() |> Identifier.atom!() + # A keyword, such as the module for a `Type` definition, renders as a raw identifier. + defp module_identifier(name), do: name |> Name.rust_identifier() |> Identifier.atom!() + end end diff --git a/codegen/kiwi_codec/rustler_generator/rust_expr.ex b/codegen/kiwi_codec/rustler_generator/rust_expr.ex index 236a5ab..d8fa6ba 100644 --- a/codegen/kiwi_codec/rustler_generator/rust_expr.ex +++ b/codegen/kiwi_codec/rustler_generator/rust_expr.ex @@ -1,34 +1,36 @@ -defmodule KiwiCodec.RustlerGenerator.RustExpr do - @moduledoc """ - Rust expressions generated decoders read Kiwi values with, built as RustQ AST. +if Code.ensure_loaded?(RustQ) do + defmodule KiwiCodec.RustlerGenerator.RustExpr do + @moduledoc """ + Rust expressions generated decoders read Kiwi values with, built as RustQ AST. - Each reads one value from the `decoder` in scope, with `env` for values that become terms. - """ + Each reads one value from the `decoder` in scope, with `env` for values that become terms. + """ - alias RustQ.Rust.AST - alias RustQ.Rust.AST.Builder, as: A + alias RustQ.Rust.AST + alias RustQ.Rust.AST.Builder, as: A - @doc """ - Returns `decoder.read_*(...)?` for a primitive type, or `nil` for a schema definition. - """ - @spec primitive(String.t()) :: AST.expr() | nil - def primitive(type) do - if KiwiCodec.PrimitiveType.name?(type), do: A.try(read(type)) - end + @doc """ + Returns `decoder.read_*(...)?` for a primitive type, or `nil` for a schema definition. + """ + @spec primitive(String.t()) :: AST.expr() | nil + def primitive(type) do + if KiwiCodec.PrimitiveType.name?(type), do: A.try(read(type)) + end - @doc "Returns the `decoder.read_*(...)` call reading a primitive type, without `?`." - @spec read(KiwiCodec.PrimitiveType.name()) :: AST.expr() - def read(type) do - {method, args} = primitive_decoder(type) - A.method(:decoder, method, args) - end + @doc "Returns the `decoder.read_*(...)` call reading a primitive type, without `?`." + @spec read(KiwiCodec.PrimitiveType.name()) :: AST.expr() + def read(type) do + {method, args} = primitive_decoder(type) + A.method(:decoder, method, args) + end - defp primitive_decoder("bool"), do: {:read_bool, []} - defp primitive_decoder("byte"), do: {:read_byte, []} - defp primitive_decoder("string"), do: {:read_string, [:env]} - defp primitive_decoder("float"), do: {:read_var_float, [:env]} - defp primitive_decoder("int"), do: {:read_var_int, []} - defp primitive_decoder("int64"), do: {:read_var_int64, []} - defp primitive_decoder("uint"), do: {:read_var_uint, []} - defp primitive_decoder("uint64"), do: {:read_var_uint64, []} + defp primitive_decoder("bool"), do: {:read_bool, []} + defp primitive_decoder("byte"), do: {:read_byte, []} + defp primitive_decoder("string"), do: {:read_string, [:env]} + defp primitive_decoder("float"), do: {:read_var_float, [:env]} + defp primitive_decoder("int"), do: {:read_var_int, []} + defp primitive_decoder("int64"), do: {:read_var_int64, []} + defp primitive_decoder("uint"), do: {:read_var_uint, []} + defp primitive_decoder("uint64"), do: {:read_var_uint64, []} + end end diff --git a/codegen/kiwi_codec/rustler_generator/selection.ex b/codegen/kiwi_codec/rustler_generator/selection.ex index d62a016..dff6ee0 100644 --- a/codegen/kiwi_codec/rustler_generator/selection.ex +++ b/codegen/kiwi_codec/rustler_generator/selection.ex @@ -1,43 +1,45 @@ -defmodule KiwiCodec.RustlerGenerator.Selection do - @moduledoc """ - Selects schema definitions and their dependencies for Rustler generation. - """ - - alias KiwiCodec.Schema - - @spec definition_map(Schema.t()) :: %{String.t() => Schema.definition()} - def definition_map(%Schema{} = schema), do: Map.new(schema.definitions, &{&1.name, &1}) - - @spec definitions(Schema.t(), [atom() | String.t()], map()) :: [Schema.definition()] - def definitions(%Schema{} = schema, [], _definition_map), do: schema.definitions - - def definitions(%Schema{} = schema, names, definition_map) do - names - |> Enum.map(&to_string/1) - |> include_dependencies(definition_map, MapSet.new()) - |> then(fn selected_names -> - Enum.filter(schema.definitions, &MapSet.member?(selected_names, &1.name)) - end) - end +if Code.ensure_loaded?(RustQ) do + defmodule KiwiCodec.RustlerGenerator.Selection do + @moduledoc """ + Selects schema definitions and their dependencies for Rustler generation. + """ + + alias KiwiCodec.Schema + + @spec definition_map(Schema.t()) :: %{String.t() => Schema.definition()} + def definition_map(%Schema{} = schema), do: Map.new(schema.definitions, &{&1.name, &1}) + + @spec definitions(Schema.t(), [atom() | String.t()], map()) :: [Schema.definition()] + def definitions(%Schema{} = schema, [], _definition_map), do: schema.definitions + + def definitions(%Schema{} = schema, names, definition_map) do + names + |> Enum.map(&to_string/1) + |> include_dependencies(definition_map, MapSet.new()) + |> then(fn selected_names -> + Enum.filter(schema.definitions, &MapSet.member?(selected_names, &1.name)) + end) + end - defp include_dependencies([], _definition_map, acc), do: acc + defp include_dependencies([], _definition_map, acc), do: acc - defp include_dependencies([name | names], definition_map, acc) do - if MapSet.member?(acc, name) do - include_dependencies(names, definition_map, acc) - else - definition = Map.fetch!(definition_map, name) + defp include_dependencies([name | names], definition_map, acc) do + if MapSet.member?(acc, name) do + include_dependencies(names, definition_map, acc) + else + definition = Map.fetch!(definition_map, name) - dependencies = - definition - |> fields() - |> Enum.map(& &1.type) - |> Enum.filter(&Map.has_key?(definition_map, &1)) + dependencies = + definition + |> fields() + |> Enum.map(& &1.type) + |> Enum.filter(&Map.has_key?(definition_map, &1)) - include_dependencies(names ++ dependencies, definition_map, MapSet.put(acc, name)) + include_dependencies(names ++ dependencies, definition_map, MapSet.put(acc, name)) + end end - end - defp fields(%{fields: fields}), do: fields - defp fields(_enum), do: [] + defp fields(%{fields: fields}), do: fields + defp fields(_enum), do: [] + end end diff --git a/codegen/kiwi_codec/rustler_generator/skip.ex b/codegen/kiwi_codec/rustler_generator/skip.ex index c8b291c..8fa69c4 100644 --- a/codegen/kiwi_codec/rustler_generator/skip.ex +++ b/codegen/kiwi_codec/rustler_generator/skip.ex @@ -1,121 +1,126 @@ -defmodule KiwiCodec.RustlerGenerator.Skip do - @moduledoc """ - Generates generic Kiwi schema skip decoders for Rustler native backends. - - Skip decoders consume encoded values without allocating decoded Elixir terms. - They are schema-generic and intended for callers that project only selected - fields from a Kiwi payload. - """ - - alias KiwiCodec.RustlerGenerator.Name - alias __MODULE__.Descriptor - alias __MODULE__.ValueHelpers - alias KiwiCodec.Schema.Enum, as: SchemaEnum - alias KiwiCodec.Schema.{Message, Struct} - alias RustQ.Meta.AST, as: MetaAST - alias RustQ.Rust.AST - alias RustQ.Rust.AST.Builder, as: A - alias RustQ.Rust.AST.PatternBuilder, as: P - - defmodule Kind do +if Code.ensure_loaded?(RustQ) do + defmodule KiwiCodec.RustlerGenerator.Skip do @moduledoc """ - Semantic skip operation for one Kiwi field value. - """ - - @enforce_keys [:mode, :function] - defstruct [:mode, :function] - - @type mode :: :one | :repeated | :bytes - @type t :: %__MODULE__{mode: mode(), function: atom()} - end + Generates generic Kiwi schema skip decoders for Rustler native backends. - @spec fragments([KiwiCodec.Schema.definition()], map(), keyword()) :: [RustQ.Rust.Fragment.t()] - def fragments(definitions, definition_map, opts \\ []) do - messages? = Keyword.get(opts, :messages?, true) - Enum.map(definitions, &definition(&1, definition_map, messages?)) - end - - @spec message_arm(map(), map()) :: AST.Arm.t() - def message_arm(field, definition_map) do - %AST.Arm{ - pattern: P.lit(field.id), - body: [A.return_stmt(skip_call(field, definition_map))] - } - end - - defp definition(definition, definition_map, messages?) - defp definition(%SchemaEnum{}, _definition_map, _messages?), do: [] - - defp definition(%Struct{name: name, fields: fields}, definition_map, _messages?) do - MetaAST.macro_call!(ValueHelpers, :kiwi_skip_struct_decoder, - fn: skip_function_name(name), - decoder: :decoder, - fields: skip_descriptor_rows(fields, definition_map) - ) - end - - defp definition(%Message{}, _definition_map, false), do: [] - - defp definition(%Message{name: name, fields: fields}, definition_map, true) do - MetaAST.macro_call!(ValueHelpers, :kiwi_skip_message_decoder, - fn: skip_function_name(name), - decoder: :decoder, - definition: name, - fields: skip_message_descriptor_rows(fields, definition_map) - ) - end + Skip decoders consume encoded values without allocating decoded Elixir terms. + They are schema-generic and intended for callers that project only selected + fields from a Kiwi payload. + """ - defp skip_message_descriptor_rows(fields, definition_map) do - Enum.map(fields, fn field -> + alias KiwiCodec.RustlerGenerator.Name + alias __MODULE__.Descriptor + alias __MODULE__.ValueHelpers + alias KiwiCodec.Schema.Enum, as: SchemaEnum + alias KiwiCodec.Schema.{Message, Struct} + alias RustQ.Meta.AST, as: MetaAST + alias RustQ.Rust.AST + alias RustQ.Rust.AST.Builder, as: A + alias RustQ.Rust.AST.PatternBuilder, as: P + + defmodule Kind do + @moduledoc """ + Semantic skip operation for one Kiwi field value. + """ + + @enforce_keys [:mode, :function] + defstruct [:mode, :function] + + @type mode :: :one | :repeated | :bytes + @type t :: %__MODULE__{mode: mode(), function: atom()} + end + + @spec fragments([KiwiCodec.Schema.definition()], map(), keyword()) :: [ + RustQ.Rust.Fragment.t() + ] + def fragments(definitions, definition_map, opts \\ []) do + messages? = Keyword.get(opts, :messages?, true) + Enum.map(definitions, &definition(&1, definition_map, messages?)) + end + + @spec message_arm(map(), map()) :: AST.Arm.t() + def message_arm(field, definition_map) do + %AST.Arm{ + pattern: P.lit(field.id), + body: [A.return_stmt(skip_call(field, definition_map))] + } + end + + defp definition(definition, definition_map, messages?) + defp definition(%SchemaEnum{}, _definition_map, _messages?), do: [] + + defp definition(%Struct{name: name, fields: fields}, definition_map, _messages?) do + MetaAST.macro_call!(ValueHelpers, :kiwi_skip_struct_decoder, + fn: skip_function_name(name), + decoder: :decoder, + fields: skip_descriptor_rows(fields, definition_map) + ) + end + + defp definition(%Message{}, _definition_map, false), do: [] + + defp definition(%Message{name: name, fields: fields}, definition_map, true) do + MetaAST.macro_call!(ValueHelpers, :kiwi_skip_message_decoder, + fn: skip_function_name(name), + decoder: :decoder, + definition: name, + fields: skip_message_descriptor_rows(fields, definition_map) + ) + end + + defp skip_message_descriptor_rows(fields, definition_map) do + Enum.map(fields, fn field -> + field + |> skip_descriptor_row(definition_map) + |> Keyword.put(:field_id, field.id) + end) + end + + defp skip_descriptor_rows(fields, definition_map) do + Enum.map(fields, &skip_descriptor_row(&1, definition_map)) + end + + defp skip_descriptor_row(field, definition_map) do + kind = skip_kind(field, definition_map) + + [ + field_repeated: match?(%Kind{mode: :repeated}, kind), + field_bytes: match?(%Kind{mode: :bytes}, kind), + field_skip: kind.function + ] + end + + defp skip_call(field, definition_map) do field - |> skip_descriptor_row(definition_map) - |> Keyword.put(:field_id, field.id) - end) - end - - defp skip_descriptor_rows(fields, definition_map) do - Enum.map(fields, &skip_descriptor_row(&1, definition_map)) - end - - defp skip_descriptor_row(field, definition_map) do - kind = skip_kind(field, definition_map) - - [ - field_repeated: match?(%Kind{mode: :repeated}, kind), - field_bytes: match?(%Kind{mode: :bytes}, kind), - field_skip: kind.function - ] - end + |> skip_kind(definition_map) + |> skip_kind_call() + end - defp skip_call(field, definition_map) do - field - |> skip_kind(definition_map) - |> skip_kind_call() - end + defp skip_function_name(name), do: ["skip_", Name.rust_identifier(name), "_from_decoder"] - defp skip_function_name(name), do: ["skip_", Name.rust_identifier(name), "_from_decoder"] + defp skip_kind(%{array?: true, type: "byte"}, _definition_map), + do: %Kind{mode: :bytes, function: :kiwi_skip_bytes_value} - defp skip_kind(%{array?: true, type: "byte"}, _definition_map), - do: %Kind{mode: :bytes, function: :kiwi_skip_bytes_value} + defp skip_kind(%{array?: true, type: type}, definition_map) do + %Kind{mode: :repeated, function: scalar_skip_function(type, definition_map)} + end - defp skip_kind(%{array?: true, type: type}, definition_map) do - %Kind{mode: :repeated, function: scalar_skip_function(type, definition_map)} - end + defp skip_kind(%{type: type}, definition_map) do + %Kind{mode: :one, function: scalar_skip_function(type, definition_map)} + end - defp skip_kind(%{type: type}, definition_map) do - %Kind{mode: :one, function: scalar_skip_function(type, definition_map)} - end + defp skip_kind_call(%Kind{mode: :bytes}), + do: A.try(A.call(:kiwi_skip_bytes_value, [:decoder])) - defp skip_kind_call(%Kind{mode: :bytes}), do: A.try(A.call(:kiwi_skip_bytes_value, [:decoder])) + defp skip_kind_call(%Kind{mode: :repeated, function: function}) do + A.try(A.call(:kiwi_skip_repeated, [A.var(:decoder), A.path(function)])) + end - defp skip_kind_call(%Kind{mode: :repeated, function: function}) do - A.try(A.call(:kiwi_skip_repeated, [A.var(:decoder), A.path(function)])) - end + defp skip_kind_call(%Kind{mode: :one, function: function}) do + A.try(A.call(function, [:decoder])) + end - defp skip_kind_call(%Kind{mode: :one, function: function}) do - A.try(A.call(function, [:decoder])) + defp scalar_skip_function(type, definition_map), + do: Descriptor.scalar_function(type, definition_map) end - - defp scalar_skip_function(type, definition_map), - do: Descriptor.scalar_function(type, definition_map) end diff --git a/codegen/kiwi_codec/rustler_generator/skip/descriptor.ex b/codegen/kiwi_codec/rustler_generator/skip/descriptor.ex index 0c9c457..7f4e271 100644 --- a/codegen/kiwi_codec/rustler_generator/skip/descriptor.ex +++ b/codegen/kiwi_codec/rustler_generator/skip/descriptor.ex @@ -1,26 +1,28 @@ -defmodule KiwiCodec.RustlerGenerator.Skip.Descriptor do - @moduledoc """ - Names the Rust function that skips one value of a Kiwi type: a primitive's - shared skip helper, `kiwi_skip_uint_value` for an enum, or the generated - `skip__from_decoder` for a struct or message. - """ +if Code.ensure_loaded?(RustQ) do + defmodule KiwiCodec.RustlerGenerator.Skip.Descriptor do + @moduledoc """ + Names the Rust function that skips one value of a Kiwi type: a primitive's + shared skip helper, `kiwi_skip_uint_value` for an enum, or the generated + `skip__from_decoder` for a struct or message. + """ - alias KiwiCodec.RustlerGenerator.Name - alias KiwiCodec.Schema.Enum, as: SchemaEnum - alias RustQ.Rust.Identifier + alias KiwiCodec.RustlerGenerator.Name + alias KiwiCodec.Schema.Enum, as: SchemaEnum + alias RustQ.Rust.Identifier - @doc "Returns the skip function for `type`, given the schema's definitions by name." - @spec scalar_function(String.t(), map()) :: atom() - def scalar_function(type, definition_map) do - cond do - KiwiCodec.PrimitiveType.name?(type) -> - Identifier.atom!("kiwi_skip_#{type}_value") + @doc "Returns the skip function for `type`, given the schema's definitions by name." + @spec scalar_function(String.t(), map()) :: atom() + def scalar_function(type, definition_map) do + cond do + KiwiCodec.PrimitiveType.name?(type) -> + Identifier.atom!("kiwi_skip_#{type}_value") - match?(%SchemaEnum{}, Map.get(definition_map, type)) -> - :kiwi_skip_uint_value + match?(%SchemaEnum{}, Map.get(definition_map, type)) -> + :kiwi_skip_uint_value - Map.has_key?(definition_map, type) -> - Identifier.atom!("skip_#{Name.rust_identifier(type)}_from_decoder") + Map.has_key?(definition_map, type) -> + Identifier.atom!("skip_#{Name.rust_identifier(type)}_from_decoder") + end end end end diff --git a/codegen/kiwi_codec/rustler_generator/skip/helpers.ex b/codegen/kiwi_codec/rustler_generator/skip/helpers.ex index 349e5c1..cbfa3b9 100644 --- a/codegen/kiwi_codec/rustler_generator/skip/helpers.ex +++ b/codegen/kiwi_codec/rustler_generator/skip/helpers.ex @@ -1,237 +1,239 @@ -defmodule KiwiCodec.RustlerGenerator.Skip.Helpers do - @moduledoc """ - Builds RustQ-authored shared skip helper functions for generated Rustler decoders. - - These helpers are only used when the caller provides `:decoder_sources`, so - RustQ can read the downstream `Decoder` implementation and infer fallible - method propagation from real Rust metadata. - """ - - alias RustQ.Meta.AST, as: MetaAST - - @macros [ - :kiwi_skip_struct_decoder - ] - - @functions [ - :kiwi_skip_bool_value, - :kiwi_skip_byte_value, - :kiwi_skip_float_value, - :kiwi_skip_int_value, - :kiwi_skip_int64_value, - :kiwi_skip_string_value, - :kiwi_skip_uint_value, - :kiwi_skip_uint64_value, - :kiwi_skip_bytes_value, - :kiwi_skip_repeated, - :kiwi_skip_struct_field, - :kiwi_skip_struct_fields, - :kiwi_skip_message_fields - ] - - @spec fragments([Path.t()] | Path.t()) :: [RustQ.Rust.Fragment.t()] - def fragments(decoder_sources) do - module = generated_module!(decoder_sources) - - module.__rustq_type_items__() ++ - MetaAST.macro_items!(module, @macros) ++ - MetaAST.functions!(module, @functions) - end +if Code.ensure_loaded?(RustQ) do + defmodule KiwiCodec.RustlerGenerator.Skip.Helpers do + @moduledoc """ + Builds RustQ-authored shared skip helper functions for generated Rustler decoders. + + These helpers are only used when the caller provides `:decoder_sources`, so + RustQ can read the downstream `Decoder` implementation and infer fallible + method propagation from real Rust metadata. + """ + + alias RustQ.Meta.AST, as: MetaAST + + @macros [ + :kiwi_skip_struct_decoder + ] - defp generated_module!(decoder_sources) do - decoder_sources = KiwiCodec.RustlerGenerator.Decoder.sources(decoder_sources) + @functions [ + :kiwi_skip_bool_value, + :kiwi_skip_byte_value, + :kiwi_skip_float_value, + :kiwi_skip_int_value, + :kiwi_skip_int64_value, + :kiwi_skip_string_value, + :kiwi_skip_uint_value, + :kiwi_skip_uint64_value, + :kiwi_skip_bytes_value, + :kiwi_skip_repeated, + :kiwi_skip_struct_field, + :kiwi_skip_struct_fields, + :kiwi_skip_message_fields + ] - module = - Module.concat([ - KiwiCodec.RustlerGenerator.Generated, - "SkipHelpers#{:erlang.phash2(decoder_sources)}" - ]) + @spec fragments([Path.t()] | Path.t()) :: [RustQ.Rust.Fragment.t()] + def fragments(decoder_sources) do + module = generated_module!(decoder_sources) - if Code.ensure_loaded?(module) do - module - else - Module.create(module, quoted_module(decoder_sources), Macro.Env.location(__ENV__)) - module + module.__rustq_type_items__() ++ + MetaAST.macro_items!(module, @macros) ++ + MetaAST.functions!(module, @functions) end - end - defp quoted_module(decoder_sources) do - quote do - use RustQ.Meta, - rust_sources: unquote(decoder_sources), - callable_modules: [ - KiwiCodec.RustlerGenerator.Skip.ValueHelpers, - KiwiCodec.RustlerGenerator.UnknownFields - ] - - alias RustQ.Type, as: R - - unquote_splicing(type_definitions()) - unquote_splicing(macro_definitions()) - unquote_splicing(value_helper_definitions()) - unquote_splicing(struct_helper_definitions()) - unquote_splicing(message_helper_definitions()) + defp generated_module!(decoder_sources) do + decoder_sources = KiwiCodec.RustlerGenerator.Decoder.sources(decoder_sources) + + module = + Module.concat([ + KiwiCodec.RustlerGenerator.Generated, + "SkipHelpers#{:erlang.phash2(decoder_sources)}" + ]) + + if Code.ensure_loaded?(module) do + module + else + Module.create(module, quoted_module(decoder_sources), Macro.Env.location(__ENV__)) + module + end end - end - defp type_definitions do - [ + defp quoted_module(decoder_sources) do quote do - @type kiwi_skip_fn :: R.raw(:"fn(&mut Decoder<'_>) -> NifResult<()>") + use RustQ.Meta, + rust_sources: unquote(decoder_sources), + callable_modules: [ + KiwiCodec.RustlerGenerator.Skip.ValueHelpers, + KiwiCodec.RustlerGenerator.UnknownFields + ] + + alias RustQ.Type, as: R + + unquote_splicing(type_definitions()) + unquote_splicing(macro_definitions()) + unquote_splicing(value_helper_definitions()) + unquote_splicing(struct_helper_definitions()) + unquote_splicing(message_helper_definitions()) + end + end - @type kiwi_skip_kind :: - R.enum(one: [kiwi_skip_fn()], repeated: [kiwi_skip_fn()], bytes: []) + defp type_definitions do + [ + quote do + @type kiwi_skip_fn :: R.raw(:"fn(&mut Decoder<'_>) -> NifResult<()>") - @type kiwi_skip_field :: %{ - required(:id) => R.u32(), - required(:kind) => kiwi_skip_kind() - } + @type kiwi_skip_kind :: + R.enum(one: [kiwi_skip_fn()], repeated: [kiwi_skip_fn()], bytes: []) - @type kiwi_skip_struct_field :: %{ - required(:repeated) => R.bool(), - required(:bytes) => R.bool(), - required(:skip) => kiwi_skip_fn() - } - end - ] - end + @type kiwi_skip_field :: %{ + required(:id) => R.u32(), + required(:kind) => kiwi_skip_kind() + } - defp macro_definitions do - [ - quote do - defrustmacro kiwi_skip_struct_decoder( - fn: name(:ident), - decoder: decoder(:ident), - fields: - repeat do - field_repeated(:literal) - field_bytes(:literal) - field_skip(:ident) - end - ) do - @spec name(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: R.nif_result(R.unit()) - defrust name(decoder) do - kiwi_skip_struct_fields( - decoder, - array([ - repeat fields do - struct_literal(KiwiSkipStructField, - repeated: field_repeated, - bytes: field_bytes, - skip: field_skip - ) - end - ]), - 0 - ) + @type kiwi_skip_struct_field :: %{ + required(:repeated) => boolean(), + required(:bytes) => boolean(), + required(:skip) => kiwi_skip_fn() + } + end + ] + end + + defp macro_definitions do + [ + quote do + defrustmacro kiwi_skip_struct_decoder( + fn: name(:ident), + decoder: decoder(:ident), + fields: + repeat do + field_repeated(:literal) + field_bytes(:literal) + field_skip(:ident) + end + ) do + @spec name(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: R.nif_result(R.unit()) + defrust name(decoder) do + kiwi_skip_struct_fields( + decoder, + array([ + repeat fields do + struct_literal(KiwiSkipStructField, + repeated: field_repeated, + bytes: field_bytes, + skip: field_skip + ) + end + ]), + 0 + ) + end end end - end - ] - end + ] + end + + defp value_helper_definitions do + [ + quoted_skip_value(:kiwi_skip_bool_value, :read_bool), + quoted_skip_value(:kiwi_skip_byte_value, :read_byte), + quoted_skip_value(:kiwi_skip_float_value, :read_var_float_value), + quoted_skip_value(:kiwi_skip_int_value, :read_var_int), + quoted_skip_value(:kiwi_skip_int64_value, :read_var_int64), + quoted_skip_value(:kiwi_skip_string_value, :skip_string), + quoted_skip_value(:kiwi_skip_uint_value, :read_var_uint), + quoted_skip_value(:kiwi_skip_uint64_value, :read_var_uint64), + quoted_skip_value(:kiwi_skip_bytes_value, :skip_byte_array), + quote do + @spec kiwi_skip_repeated( + R.mut_ref(R.path(:Decoder, R.lifetime(:_))), + R.path(:KiwiSkipFn) + ) :: R.nif_result(R.unit()) + defrust kiwi_skip_repeated(decoder, item) do + decoder.read_repeated(fn decoder -> item(decoder) end) + :ok + end + end + ] + end - defp value_helper_definitions do - [ - quoted_skip_value(:kiwi_skip_bool_value, :read_bool), - quoted_skip_value(:kiwi_skip_byte_value, :read_byte), - quoted_skip_value(:kiwi_skip_float_value, :read_var_float_value), - quoted_skip_value(:kiwi_skip_int_value, :read_var_int), - quoted_skip_value(:kiwi_skip_int64_value, :read_var_int64), - quoted_skip_value(:kiwi_skip_string_value, :skip_string), - quoted_skip_value(:kiwi_skip_uint_value, :read_var_uint), - quoted_skip_value(:kiwi_skip_uint64_value, :read_var_uint64), - quoted_skip_value(:kiwi_skip_bytes_value, :skip_byte_array), + defp quoted_skip_value(function, read_method) do quote do - @spec kiwi_skip_repeated( - R.mut_ref(R.path(:Decoder, R.lifetime(:_))), - R.path(:KiwiSkipFn) - ) :: R.nif_result(R.unit()) - defrust kiwi_skip_repeated(decoder, item) do - decoder.read_repeated(fn decoder -> item(decoder) end) + @spec unquote(function)(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: + R.nif_result(R.unit()) + defrust unquote(function)(decoder) do + decoder.unquote(read_method)() :ok end end - ] - end - - defp quoted_skip_value(function, read_method) do - quote do - @spec unquote(function)(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: - R.nif_result(R.unit()) - defrust unquote(function)(decoder) do - decoder.unquote(read_method)() - :ok - end end - end - defp struct_helper_definitions do - [ - quote do - @spec kiwi_skip_struct_field( - R.mut_ref(R.path(:Decoder, R.lifetime(:_))), - R.ref(R.path(:KiwiSkipStructField)) - ) :: R.nif_result(R.unit()) - defrust kiwi_skip_struct_field(decoder, field) do - skip = field.skip - - if field.bytes do - kiwi_skip_bytes_value(decoder) - else - if field.repeated do - kiwi_skip_repeated(decoder, skip) + defp struct_helper_definitions do + [ + quote do + @spec kiwi_skip_struct_field( + R.mut_ref(R.path(:Decoder, R.lifetime(:_))), + R.ref(R.path(:KiwiSkipStructField)) + ) :: R.nif_result(R.unit()) + defrust kiwi_skip_struct_field(decoder, field) do + skip = field.skip + + if field.bytes do + kiwi_skip_bytes_value(decoder) else - skip(decoder) + if field.repeated do + kiwi_skip_repeated(decoder, skip) + else + skip(decoder) + end end end - end - end, - quote do - @spec kiwi_skip_struct_fields( - R.mut_ref(R.path(:Decoder, R.lifetime(:_))), - R.slice(R.path(:KiwiSkipStructField)), - R.usize() - ) :: R.nif_result(R.unit()) - defrust kiwi_skip_struct_fields(decoder, fields, index) do - if index == fields.len() do - :ok - else - field = fields.get(index).unwrap() - kiwi_skip_struct_field(decoder, field) - kiwi_skip_struct_fields(decoder, fields, index + 1) + end, + quote do + @spec kiwi_skip_struct_fields( + R.mut_ref(R.path(:Decoder, R.lifetime(:_))), + R.slice(R.path(:KiwiSkipStructField)), + R.usize() + ) :: R.nif_result(R.unit()) + defrust kiwi_skip_struct_fields(decoder, fields, index) do + if index == fields.len() do + :ok + else + field = fields.get(index).unwrap() + kiwi_skip_struct_field(decoder, field) + kiwi_skip_struct_fields(decoder, fields, index + 1) + end end end - end - ] - end - - defp message_helper_definitions do - [ - quote do - @spec kiwi_skip_message_fields( - R.mut_ref(R.path(:Decoder, R.lifetime(:_))), - R.str(), - R.slice(R.path(:KiwiSkipField)) - ) :: R.nif_result(R.unit()) - defrust kiwi_skip_message_fields(decoder, definition_name, fields) do - case decoder.read_var_uint() do - 0 -> - :ok - - field_id -> - case fields.binary_search_by_key(field_id, fn field -> field.id end) do - {:ok, index} -> - matched_field = fields.get(index).unwrap() - kiwi_skip_kind(decoder, matched_field.kind) + ] + end - {:error, _index} -> - kiwi_unknown_field(decoder, definition_name, field_id) - end + defp message_helper_definitions do + [ + quote do + @spec kiwi_skip_message_fields( + R.mut_ref(R.path(:Decoder, R.lifetime(:_))), + R.str(), + R.slice(R.path(:KiwiSkipField)) + ) :: R.nif_result(R.unit()) + defrust kiwi_skip_message_fields(decoder, definition_name, fields) do + case decoder.read_var_uint() do + 0 -> + :ok + + field_id -> + case fields.binary_search_by_key(field_id, fn field -> field.id end) do + {:ok, index} -> + matched_field = fields.get(index).unwrap() + kiwi_skip_kind(decoder, matched_field.kind) + + {:error, _index} -> + kiwi_unknown_field(decoder, definition_name, field_id) + end - kiwi_skip_message_fields(decoder, definition_name, fields) + kiwi_skip_message_fields(decoder, definition_name, fields) + end end end - end - ] + ] + end end end diff --git a/codegen/kiwi_codec/rustler_generator/skip/value_helpers.ex b/codegen/kiwi_codec/rustler_generator/skip/value_helpers.ex index cf163f9..d179153 100644 --- a/codegen/kiwi_codec/rustler_generator/skip/value_helpers.ex +++ b/codegen/kiwi_codec/rustler_generator/skip/value_helpers.ex @@ -1,250 +1,256 @@ -defmodule KiwiCodec.RustlerGenerator.Skip.ValueHelpers do - use RustQ.Meta, - rust_sources: [KiwiCodec.RustlerGenerator.Decoder.source()], - callable_modules: [KiwiCodec.RustlerGenerator.UnknownFields] - - @moduledoc """ - RustQ-authored primitive Kiwi skip helpers used by generated Rustler decoders. - """ - - alias RustQ.Meta.AST, as: MetaAST - alias RustQ.Type, as: R - - @macros [ - :kiwi_skip_struct_decoder, - :kiwi_skip_message_decoder - ] - - @dispatch_functions [ - :kiwi_skip_kind - ] - - @typedoc "The Rust `KiwiSkipFn`: skips one value of a type." - @type kiwi_skip_fn :: R.raw(:"fn(&mut Decoder<'_>) -> NifResult<()>") - - @typedoc "The Rust `KiwiSkipKind`: how a field's value is skipped, once, repeated, or as bytes." - @type kiwi_skip_kind :: - R.enum(one: [kiwi_skip_fn()], repeated: [kiwi_skip_fn()], bytes: []) - - @typedoc "The Rust `KiwiSkipField`: a message field id and how to skip its value." - @type kiwi_skip_field :: %{ - required(:id) => R.u32(), - required(:kind) => kiwi_skip_kind() - } - - @spec macro_fragments([atom()]) :: [RustQ.Rust.Fragment.t()] - def macro_fragments(names \\ @macros) do - MetaAST.macro_items!(__MODULE__, names) - end +if Code.ensure_loaded?(RustQ) do + defmodule KiwiCodec.RustlerGenerator.Skip.ValueHelpers do + use RustQ.Meta, + rust_sources: [KiwiCodec.RustlerGenerator.Decoder.source()], + callable_modules: [KiwiCodec.RustlerGenerator.UnknownFields] + + @moduledoc """ + RustQ-authored primitive Kiwi skip helpers used by generated Rustler decoders. + """ + + alias RustQ.Meta.AST, as: MetaAST + alias RustQ.Type, as: R + + @macros [ + :kiwi_skip_struct_decoder, + :kiwi_skip_message_decoder + ] + + @dispatch_functions [ + :kiwi_skip_kind + ] + + @typedoc "The Rust `KiwiSkipFn`: skips one value of a type." + @type kiwi_skip_fn :: R.raw(:"fn(&mut Decoder<'_>) -> NifResult<()>") + + # The Rust `KiwiSkipKind`: how a field's value is skipped, once, repeated, or as bytes. + @typep kiwi_skip_kind :: + R.enum(one: [kiwi_skip_fn()], repeated: [kiwi_skip_fn()], bytes: []) + + @typedoc "The Rust `KiwiSkipField`: a message field id and how to skip its value." + @type kiwi_skip_field :: %{ + required(:id) => R.u32(), + required(:kind) => kiwi_skip_kind() + } + + @spec macro_fragments([atom()]) :: [RustQ.Rust.Fragment.t()] + def macro_fragments(names \\ @macros) do + MetaAST.macro_items!(__MODULE__, names) + end - @doc """ - Returns `kiwi_skip_kind`, which skips a message field's value, and with - `descriptor_kind?: true` also `kiwi_skip_descriptor_kind`, which skip message macros call. - """ - @spec dispatch_fragments(keyword()) :: [RustQ.Rust.Fragment.t()] - def dispatch_fragments(opts \\ []) do - functions = - if Keyword.get(opts, :descriptor_kind?, false), - do: [:kiwi_skip_descriptor_kind | @dispatch_functions], - else: @dispatch_functions - - MetaAST.functions!(__MODULE__, functions) - end + @doc """ + Returns `kiwi_skip_kind`, which skips a message field's value, and with + `descriptor_kind?: true` also `kiwi_skip_descriptor_kind`, which skip message macros call. + """ + @spec dispatch_fragments(keyword()) :: [RustQ.Rust.Fragment.t()] + def dispatch_fragments(opts \\ []) do + functions = + if Keyword.get(opts, :descriptor_kind?, false), + do: [:kiwi_skip_descriptor_kind | @dispatch_functions], + else: @dispatch_functions + + MetaAST.functions!(__MODULE__, functions) + end - defrustmacro kiwi_skip_struct_decoder( - fn: name(:ident), - decoder: decoder(:ident), - fields: - repeat do - field_repeated(:literal) - field_bytes(:literal) - field_skip(:ident) - end - ) do - @spec name(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: R.nif_result(R.unit()) - defrust name(decoder) do - kiwi_skip_struct_fields( - decoder, - array([ - repeat fields do - kiwi_skip_descriptor_kind(field_repeated, field_bytes, field_skip) - end - ]) - ) + defrustmacro kiwi_skip_struct_decoder( + fn: name(:ident), + decoder: decoder(:ident), + fields: + repeat do + field_repeated(:literal) + field_bytes(:literal) + field_skip(:ident) + end + ) do + @spec name(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: R.nif_result(R.unit()) + defrust name(decoder) do + kiwi_skip_struct_fields( + decoder, + array([ + repeat fields do + kiwi_skip_descriptor_kind(field_repeated, field_bytes, field_skip) + end + ]) + ) + end end - end - defrustmacro kiwi_skip_message_decoder( - fn: name(:ident), - decoder: decoder(:ident), - definition: definition_name(:literal), - fields: - repeat do - field_id(:literal) - field_repeated(:literal) - field_bytes(:literal) - field_skip(:ident) - end - ) do - @spec name(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: R.nif_result(R.unit()) - defrust name(decoder) do - kiwi_skip_message_fields( - decoder, - definition_name, - array([ - repeat fields do - struct_literal(KiwiSkipField, - id: field_id, - kind: kiwi_skip_descriptor_kind(field_repeated, field_bytes, field_skip) - ) - end - ]) - ) + defrustmacro kiwi_skip_message_decoder( + fn: name(:ident), + decoder: decoder(:ident), + definition: definition_name(:literal), + fields: + repeat do + field_id(:literal) + field_repeated(:literal) + field_bytes(:literal) + field_skip(:ident) + end + ) do + @spec name(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: R.nif_result(R.unit()) + defrust name(decoder) do + kiwi_skip_message_fields( + decoder, + definition_name, + array([ + repeat fields do + struct_literal(KiwiSkipField, + id: field_id, + kind: kiwi_skip_descriptor_kind(field_repeated, field_bytes, field_skip) + ) + end + ]) + ) + end end - end - @spec kiwi_skip_descriptor_kind(R.bool(), R.bool(), R.path(:KiwiSkipFn)) :: - R.path(:KiwiSkipKind) - defrust kiwi_skip_descriptor_kind(repeated, bytes, skip) do - if bytes do - enum_variant(KiwiSkipKind, :bytes) - else - if repeated do - enum_variant(KiwiSkipKind, :repeated, skip) + @spec kiwi_skip_descriptor_kind(boolean(), boolean(), R.path(:KiwiSkipFn)) :: + R.path(:KiwiSkipKind) + defrust kiwi_skip_descriptor_kind(repeated, bytes, skip) do + if bytes do + enum_variant(KiwiSkipKind, :bytes) else - enum_variant(KiwiSkipKind, :one, skip) + if repeated do + enum_variant(KiwiSkipKind, :repeated, skip) + else + enum_variant(KiwiSkipKind, :one, skip) + end end end - end - @spec kiwi_skip_bool_value(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: - R.nif_result(R.unit()) - defrust kiwi_skip_bool_value(decoder) do - decoder.read_bool() - :ok - end + @spec kiwi_skip_bool_value(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: + R.nif_result(R.unit()) + defrust kiwi_skip_bool_value(decoder) do + decoder.read_bool() + :ok + end - @spec kiwi_skip_byte_value(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: - R.nif_result(R.unit()) - defrust kiwi_skip_byte_value(decoder) do - decoder.read_byte() - :ok - end + @spec kiwi_skip_byte_value(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: + R.nif_result(R.unit()) + defrust kiwi_skip_byte_value(decoder) do + decoder.read_byte() + :ok + end - @spec kiwi_skip_float_value(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: - R.nif_result(R.unit()) - defrust kiwi_skip_float_value(decoder) do - decoder.read_var_float_value() - :ok - end + @spec kiwi_skip_float_value(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: + R.nif_result(R.unit()) + defrust kiwi_skip_float_value(decoder) do + decoder.read_var_float_value() + :ok + end - @spec kiwi_skip_int_value(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: R.nif_result(R.unit()) - defrust kiwi_skip_int_value(decoder) do - decoder.read_var_int() - :ok - end + @spec kiwi_skip_int_value(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: + R.nif_result(R.unit()) + defrust kiwi_skip_int_value(decoder) do + decoder.read_var_int() + :ok + end - @spec kiwi_skip_int64_value(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: - R.nif_result(R.unit()) - defrust kiwi_skip_int64_value(decoder) do - decoder.read_var_int64() - :ok - end + @spec kiwi_skip_int64_value(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: + R.nif_result(R.unit()) + defrust kiwi_skip_int64_value(decoder) do + decoder.read_var_int64() + :ok + end - @spec kiwi_skip_string_value(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: - R.nif_result(R.unit()) - defrust kiwi_skip_string_value(decoder) do - decoder.skip_string() - :ok - end + @spec kiwi_skip_string_value(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: + R.nif_result(R.unit()) + defrust kiwi_skip_string_value(decoder) do + decoder.skip_string() + :ok + end - @spec kiwi_skip_uint_value(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: - R.nif_result(R.unit()) - defrust kiwi_skip_uint_value(decoder) do - decoder.read_var_uint() - :ok - end + @spec kiwi_skip_uint_value(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: + R.nif_result(R.unit()) + defrust kiwi_skip_uint_value(decoder) do + decoder.read_var_uint() + :ok + end - @spec kiwi_skip_uint64_value(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: - R.nif_result(R.unit()) - defrust kiwi_skip_uint64_value(decoder) do - decoder.read_var_uint64() - :ok - end + @spec kiwi_skip_uint64_value(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: + R.nif_result(R.unit()) + defrust kiwi_skip_uint64_value(decoder) do + decoder.read_var_uint64() + :ok + end - @spec kiwi_skip_bytes_value(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: - R.nif_result(R.unit()) - defrust kiwi_skip_bytes_value(decoder) do - decoder.skip_byte_array() - :ok - end + @spec kiwi_skip_bytes_value(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: + R.nif_result(R.unit()) + defrust kiwi_skip_bytes_value(decoder) do + decoder.skip_byte_array() + :ok + end - @spec kiwi_skip_repeated(R.mut_ref(R.path(:Decoder, R.lifetime(:_))), R.path(:KiwiSkipFn)) :: - R.nif_result(R.unit()) - defrust kiwi_skip_repeated(decoder, item) do - decoder.read_repeated(fn decoder -> item(decoder) end) - :ok - end + @spec kiwi_skip_repeated(R.mut_ref(R.path(:Decoder, R.lifetime(:_))), R.path(:KiwiSkipFn)) :: + R.nif_result(R.unit()) + defrust kiwi_skip_repeated(decoder, item) do + decoder.read_repeated(fn decoder -> item(decoder) end) + :ok + end - @spec kiwi_skip_kind(R.mut_ref(R.path(:Decoder, R.lifetime(:_))), R.ref(R.path(:KiwiSkipKind))) :: - R.nif_result(R.unit()) - defrust kiwi_skip_kind(decoder, kind) do - case kind do - enum_variant(KiwiSkipKind, :one, skip) -> - skip(decoder) + @spec kiwi_skip_kind( + R.mut_ref(R.path(:Decoder, R.lifetime(:_))), + R.ref(R.path(:KiwiSkipKind)) + ) :: + R.nif_result(R.unit()) + defrust kiwi_skip_kind(decoder, kind) do + case kind do + enum_variant(KiwiSkipKind, :one, skip) -> + skip(decoder) + + enum_variant(KiwiSkipKind, :repeated, skip) -> + kiwi_skip_repeated(decoder, deref(skip)) + + enum_variant(KiwiSkipKind, :bytes) -> + kiwi_skip_bytes_value(decoder) + end + end - enum_variant(KiwiSkipKind, :repeated, skip) -> - kiwi_skip_repeated(decoder, deref(skip)) + @spec kiwi_skip_struct_fields( + R.mut_ref(R.path(:Decoder, R.lifetime(:_))), + R.slice(R.path(:KiwiSkipKind)) + ) :: R.nif_result(R.unit()) + defrust kiwi_skip_struct_fields(decoder, fields) do + kiwi_skip_struct_fields_remaining(decoder, fields, 0) + end - enum_variant(KiwiSkipKind, :bytes) -> - kiwi_skip_bytes_value(decoder) + @spec kiwi_skip_struct_fields_remaining( + R.mut_ref(R.path(:Decoder, R.lifetime(:_))), + R.slice(R.path(:KiwiSkipKind)), + R.usize() + ) :: R.nif_result(R.unit()) + defrust kiwi_skip_struct_fields_remaining(decoder, fields, index) do + if index == fields.len() do + :ok + else + kind = fields.get(index).unwrap() + kiwi_skip_kind(decoder, kind) + kiwi_skip_struct_fields_remaining(decoder, fields, index + 1) + end end - end - @spec kiwi_skip_struct_fields( - R.mut_ref(R.path(:Decoder, R.lifetime(:_))), - R.slice(R.path(:KiwiSkipKind)) - ) :: R.nif_result(R.unit()) - defrust kiwi_skip_struct_fields(decoder, fields) do - kiwi_skip_struct_fields_remaining(decoder, fields, 0) - end + @spec kiwi_skip_message_fields( + R.mut_ref(R.path(:Decoder, R.lifetime(:_))), + R.str(), + R.slice(R.path(:KiwiSkipField)) + ) :: R.nif_result(R.unit()) + defrust kiwi_skip_message_fields(decoder, definition_name, fields) do + field_id = decoder.read_var_uint() - @spec kiwi_skip_struct_fields_remaining( - R.mut_ref(R.path(:Decoder, R.lifetime(:_))), - R.slice(R.path(:KiwiSkipKind)), - R.usize() - ) :: R.nif_result(R.unit()) - defrust kiwi_skip_struct_fields_remaining(decoder, fields, index) do - if index == fields.len() do - :ok - else - kind = fields.get(index).unwrap() - kiwi_skip_kind(decoder, kind) - kiwi_skip_struct_fields_remaining(decoder, fields, index + 1) - end - end + if field_id == 0 do + :ok + else + case fields.binary_search_by_key(field_id, fn field -> field.id end) do + {:ok, index} -> + field = fields.get(index).unwrap() + kiwi_skip_kind(decoder, field.kind) - @spec kiwi_skip_message_fields( - R.mut_ref(R.path(:Decoder, R.lifetime(:_))), - R.str(), - R.slice(R.path(:KiwiSkipField)) - ) :: R.nif_result(R.unit()) - defrust kiwi_skip_message_fields(decoder, definition_name, fields) do - field_id = decoder.read_var_uint() + {:error, _index} -> + kiwi_unknown_field(decoder, definition_name, field_id) + end - if field_id == 0 do - :ok - else - case fields.binary_search_by_key(field_id, fn field -> field.id end) do - {:ok, index} -> - field = fields.get(index).unwrap() - kiwi_skip_kind(decoder, field.kind) - - {:error, _index} -> - kiwi_unknown_field(decoder, definition_name, field_id) + kiwi_skip_message_fields(decoder, definition_name, fields) end - - kiwi_skip_message_fields(decoder, definition_name, fields) end end end diff --git a/codegen/kiwi_codec/rustler_generator/sparse.ex b/codegen/kiwi_codec/rustler_generator/sparse.ex index 464953a..496ed78 100644 --- a/codegen/kiwi_codec/rustler_generator/sparse.ex +++ b/codegen/kiwi_codec/rustler_generator/sparse.ex @@ -1,321 +1,323 @@ -defmodule KiwiCodec.RustlerGenerator.Sparse do - @moduledoc """ - Generates generic sparse Kiwi schema decoders for Rustler native backends. - - Sparse decoders return maps containing only fields present in the payload plus - a `__kiwi_module__` key identifying the generated Elixir schema module. - """ - - alias KiwiCodec.RustlerGenerator.Name - alias __MODULE__.Helpers - alias KiwiCodec.RustlerGenerator.Skip - alias KiwiCodec.Schema.Enum, as: SchemaEnum - alias KiwiCodec.Schema.{Message, Struct} - alias RustQ.Meta.AST, as: MetaAST - alias RustQ.Rust.Identifier - - @spec fragments([KiwiCodec.Schema.definition()], String.t(), map(), keyword()) :: [ - RustQ.Rust.Fragment.t() - ] - def fragments(definitions, module_prefix, definition_map, opts \\ []) do - full? = Keyword.get(opts, :full?, false) - message_mode = Keyword.get(opts, :message_mode, :match) - - definitions - |> Enum.flat_map(fn definition -> - definition(definition, module_prefix, definition_map, full?, message_mode) - |> List.wrap() - end) - end - - defp definition( - %Struct{name: name, fields: fields}, - module_prefix, - definition_map, - _full?, - _message_mode - ) do - sparse_struct_decoder(name, module_prefix, fields, definition_map) - end - - defp definition( - %SchemaEnum{name: name}, - _module_prefix, - _definition_map, - true, - _message_mode - ) do - sparse_enum_passthrough_function(name) - end +if Code.ensure_loaded?(RustQ) do + defmodule KiwiCodec.RustlerGenerator.Sparse do + @moduledoc """ + Generates generic sparse Kiwi schema decoders for Rustler native backends. + + Sparse decoders return maps containing only fields present in the payload plus + a `__kiwi_module__` key identifying the generated Elixir schema module. + """ + + alias KiwiCodec.RustlerGenerator.Name + alias __MODULE__.Helpers + alias KiwiCodec.RustlerGenerator.Skip + alias KiwiCodec.Schema.Enum, as: SchemaEnum + alias KiwiCodec.Schema.{Message, Struct} + alias RustQ.Meta.AST, as: MetaAST + alias RustQ.Rust.Identifier + + @spec fragments([KiwiCodec.Schema.definition()], String.t(), map(), keyword()) :: [ + RustQ.Rust.Fragment.t() + ] + def fragments(definitions, module_prefix, definition_map, opts \\ []) do + full? = Keyword.get(opts, :full?, false) + message_mode = Keyword.get(opts, :message_mode, :match) + + definitions + |> Enum.flat_map(fn definition -> + definition(definition, module_prefix, definition_map, full?, message_mode) + |> List.wrap() + end) + end - defp definition( - %SchemaEnum{name: name, variants: variants}, - _module_prefix, - _definition_map, - false, - _message_mode - ) do - Helpers.macro_call(:kiwi_sparse_enum_decoder, - fn: "decode_sparse_#{Name.rust_identifier(name)}_from_decoder", - env: :env, - decoder: :decoder, - variants: - variants - |> Enum.sort_by(& &1.value) - |> Enum.map(&[variant_value: &1.value, variant_name: Macro.underscore(&1.name)]) - ) - end + defp definition( + %Struct{name: name, fields: fields}, + module_prefix, + definition_map, + _full?, + _message_mode + ) do + sparse_struct_decoder(name, module_prefix, fields, definition_map) + end - defp definition( - %Message{name: name, fields: fields}, - module_prefix, - definition_map, - _full?, - :match - ) do - sparse_message_function( - name, - module_name(module_prefix, name), - fields, - definition_map - ) - end + defp definition( + %SchemaEnum{name: name}, + _module_prefix, + _definition_map, + true, + _message_mode + ) do + sparse_enum_passthrough_function(name) + end - defp definition( - %Message{name: name, fields: fields}, - module_prefix, - definition_map, - _full?, - :descriptor_with_skip - ) do - ident = Name.rust_identifier(name) - - Helpers.macro_call(:kiwi_sparse_skip_message_descriptor_decoder, - sparse_fn: "decode_sparse_#{ident}_from_decoder", - skip_fn: "skip_#{ident}_from_decoder", - env: :env, - decoder: :decoder, - module: module_name(module_prefix, name), - definition: name, - capacity: length(fields) + 1, - fields: - fields - |> Enum.sort_by(& &1.id) - |> Enum.map(&sparse_skip_descriptor_row(&1, definition_map)) - ) - end + defp definition( + %SchemaEnum{name: name, variants: variants}, + _module_prefix, + _definition_map, + false, + _message_mode + ) do + Helpers.macro_call(:kiwi_sparse_enum_decoder, + fn: "decode_sparse_#{Name.rust_identifier(name)}_from_decoder", + env: :env, + decoder: :decoder, + variants: + variants + |> Enum.sort_by(& &1.value) + |> Enum.map(&[variant_value: &1.value, variant_name: Macro.underscore(&1.name)]) + ) + end - defp definition( - %Message{name: name, fields: fields}, - module_prefix, - definition_map, - _full?, - :descriptor - ) do - Helpers.macro_call(:kiwi_sparse_message_descriptor_decoder, - fn: "decode_sparse_#{Name.rust_identifier(name)}_from_decoder", - env: :env, - decoder: :decoder, - module: module_name(module_prefix, name), - definition: name, - capacity: length(fields) + 1, - fields: - fields - |> Enum.sort_by(& &1.id) - |> Enum.map(&sparse_message_descriptor_row(&1, definition_map)) - ) - end + defp definition( + %Message{name: name, fields: fields}, + module_prefix, + definition_map, + _full?, + :match + ) do + sparse_message_function( + name, + module_name(module_prefix, name), + fields, + definition_map + ) + end - defp sparse_struct_decoder(name, module_prefix, fields, definition_map) do - Helpers.macro_call(:kiwi_sparse_struct_decoder, - fn: "decode_sparse_#{Name.rust_identifier(name)}_from_decoder", - env: :env, - decoder: :decoder, - module: module_name(module_prefix, name), - capacity: length(fields) + 1, - fields: Enum.map(fields, &sparse_struct_row(&1, definition_map)) - ) - end + defp definition( + %Message{name: name, fields: fields}, + module_prefix, + definition_map, + _full?, + :descriptor_with_skip + ) do + ident = Name.rust_identifier(name) + + Helpers.macro_call(:kiwi_sparse_skip_message_descriptor_decoder, + sparse_fn: "decode_sparse_#{ident}_from_decoder", + skip_fn: "skip_#{ident}_from_decoder", + env: :env, + decoder: :decoder, + module: module_name(module_prefix, name), + definition: name, + capacity: length(fields) + 1, + fields: + fields + |> Enum.sort_by(& &1.id) + |> Enum.map(&sparse_skip_descriptor_row(&1, definition_map)) + ) + end - defp sparse_enum_passthrough_function(name) do - function = sparse_function_name(name) - module = sparse_enum_passthrough_module!(name) - module |> MetaAST.functions!([function]) |> List.first() - end + defp definition( + %Message{name: name, fields: fields}, + module_prefix, + definition_map, + _full?, + :descriptor + ) do + Helpers.macro_call(:kiwi_sparse_message_descriptor_decoder, + fn: "decode_sparse_#{Name.rust_identifier(name)}_from_decoder", + env: :env, + decoder: :decoder, + module: module_name(module_prefix, name), + definition: name, + capacity: length(fields) + 1, + fields: + fields + |> Enum.sort_by(& &1.id) + |> Enum.map(&sparse_message_descriptor_row(&1, definition_map)) + ) + end - defp sparse_message_function(name, module_name, fields, definition_map) do - function = sparse_function_name(name) - module = sparse_message_module!(name, module_name, fields, definition_map) - module |> MetaAST.functions!([function]) |> List.first() - end + defp sparse_struct_decoder(name, module_prefix, fields, definition_map) do + Helpers.macro_call(:kiwi_sparse_struct_decoder, + fn: "decode_sparse_#{Name.rust_identifier(name)}_from_decoder", + env: :env, + decoder: :decoder, + module: module_name(module_prefix, name), + capacity: length(fields) + 1, + fields: Enum.map(fields, &sparse_struct_row(&1, definition_map)) + ) + end - defp sparse_enum_passthrough_module!(name) do - module = - Module.concat([ - KiwiCodec.RustlerGenerator.Generated, - "SparseEnumPassthrough#{:erlang.phash2(name)}" - ]) - - if Code.ensure_loaded?(module) do - module - else - Module.create(module, sparse_enum_passthrough_body(name), Macro.Env.location(__ENV__)) - module + defp sparse_enum_passthrough_function(name) do + function = sparse_function_name(name) + module = sparse_enum_passthrough_module!(name) + module |> MetaAST.functions!([function]) |> List.first() end - end - defp sparse_enum_passthrough_body(name) do - function = sparse_function_name(name) - full_function = Name.decoder_function(name) + defp sparse_message_function(name, module_name, fields, definition_map) do + function = sparse_function_name(name) + module = sparse_message_module!(name, module_name, fields, definition_map) + module |> MetaAST.functions!([function]) |> List.first() + end - quote do - use RustQ.Meta - alias RustQ.Type, as: R + defp sparse_enum_passthrough_module!(name) do + module = + Module.concat([ + KiwiCodec.RustlerGenerator.Generated, + "SparseEnumPassthrough#{:erlang.phash2(name)}" + ]) + + if Code.ensure_loaded?(module) do + module + else + Module.create(module, sparse_enum_passthrough_body(name), Macro.Env.location(__ENV__)) + module + end + end - @spec unquote(function)( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))) - ) :: R.nif_result(term()) - defrust unquote(function)(env, decoder) do - unquote(full_function)(env, decoder) + defp sparse_enum_passthrough_body(name) do + function = sparse_function_name(name) + full_function = Name.decoder_function(name) + + quote do + use RustQ.Meta + alias RustQ.Type, as: R + + @spec unquote(function)( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))) + ) :: R.nif_result(term()) + defrust unquote(function)(env, decoder) do + unquote(full_function)(env, decoder) + end end end - end - defp sparse_message_module!(name, module_name, fields, definition_map) do - module = - Module.concat([ - KiwiCodec.RustlerGenerator.Generated, - "SparseMessage#{:erlang.phash2({name, module_name, Enum.map(fields, & &1.id)})}" - ]) - - if Code.ensure_loaded?(module) do - module - else - Module.create( - module, - sparse_message_module_body(name, module_name, fields, definition_map), - Macro.Env.location(__ENV__) - ) + defp sparse_message_module!(name, module_name, fields, definition_map) do + module = + Module.concat([ + KiwiCodec.RustlerGenerator.Generated, + "SparseMessage#{:erlang.phash2({name, module_name, Enum.map(fields, & &1.id)})}" + ]) + + if Code.ensure_loaded?(module) do + module + else + Module.create( + module, + sparse_message_module_body(name, module_name, fields, definition_map), + Macro.Env.location(__ENV__) + ) - module + module + end end - end - defp sparse_message_module_body(name, module_name, fields, definition_map) do - sparse_message_body(name, module_name, fields, definition_map) - end + defp sparse_message_module_body(name, module_name, fields, definition_map) do + sparse_message_body(name, module_name, fields, definition_map) + end - defp sparse_message_body(name, module_name, fields, definition_map) do - function = sparse_function_name(name) - - entries = - Enum.map(Enum.sort_by(fields, & &1.id), &sparse_message_field_entry(&1, definition_map)) - - capacity = length(fields) + 1 - - quote do - use RustQ.Meta - alias RustQ.Type, as: R - - @spec unquote(function)( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))) - ) :: R.nif_result(term()) - defrust unquote(function)(env, decoder) do - kiwi_sparse_message_fields( - env, - decoder, - unquote(module_name), - unquote(name), - unquote(capacity), - ref(array([unquote_splicing(entries)])) - ) + defp sparse_message_body(name, module_name, fields, definition_map) do + function = sparse_function_name(name) + + entries = + Enum.map(Enum.sort_by(fields, & &1.id), &sparse_message_field_entry(&1, definition_map)) + + capacity = length(fields) + 1 + + quote do + use RustQ.Meta + alias RustQ.Type, as: R + + @spec unquote(function)( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))) + ) :: R.nif_result(term()) + defrust unquote(function)(env, decoder) do + kiwi_sparse_message_fields( + env, + decoder, + unquote(module_name), + unquote(name), + unquote(capacity), + ref(array([unquote_splicing(entries)])) + ) + end end end - end - defp sparse_message_field_entry(field, definition_map) do - quote do - struct_literal(KiwiSparseField, - id: unquote(field.id), - name: unquote(Macro.underscore(field.name)), - repeated: unquote(sparse_repeated?(field)), - decode: unquote(Macro.var(sparse_decode_function(field, definition_map), nil)) - ) + defp sparse_message_field_entry(field, definition_map) do + quote do + struct_literal(KiwiSparseField, + id: unquote(field.id), + name: unquote(Macro.underscore(field.name)), + repeated: unquote(sparse_repeated?(field)), + decode: unquote(Macro.var(sparse_decode_function(field, definition_map), nil)) + ) + end end - end - defp sparse_repeated?(%{array?: true, type: "byte"}), do: false - defp sparse_repeated?(%{array?: array?}), do: array? + defp sparse_repeated?(%{array?: true, type: "byte"}), do: false + defp sparse_repeated?(%{array?: array?}), do: array? - defp sparse_decode_function(%{array?: true, type: "byte"}, _definition_map), - do: :kiwi_sparse_bytes_value + defp sparse_decode_function(%{array?: true, type: "byte"}, _definition_map), + do: :kiwi_sparse_bytes_value - defp sparse_decode_function(%{array?: true} = field, definition_map), - do: sparse_decode_function(%{field | array?: false}, definition_map) + defp sparse_decode_function(%{array?: true} = field, definition_map), + do: sparse_decode_function(%{field | array?: false}, definition_map) - defp sparse_decode_function(field, definition_map), - do: - field - |> descriptor_scalar_function(definition_map) - |> IO.iodata_to_binary() - |> Identifier.atom!() + defp sparse_decode_function(field, definition_map), + do: + field + |> descriptor_scalar_function(definition_map) + |> IO.iodata_to_binary() + |> Identifier.atom!() - defp sparse_struct_row(field, definition_map) do - [ - field_name: Macro.underscore(field.name), - field_repeated: sparse_repeated?(field), - field_decode: sparse_decode_function(field, definition_map) - ] - end + defp sparse_struct_row(field, definition_map) do + [ + field_name: Macro.underscore(field.name), + field_repeated: sparse_repeated?(field), + field_decode: sparse_decode_function(field, definition_map) + ] + end - defp sparse_message_descriptor_row(field, definition_map) do - [ - field_id: field.id, - field_name: Macro.underscore(field.name), - field_repeated: sparse_repeated?(field), - field_decode: sparse_decode_function(field, definition_map) - ] - end + defp sparse_message_descriptor_row(field, definition_map) do + [ + field_id: field.id, + field_name: Macro.underscore(field.name), + field_repeated: sparse_repeated?(field), + field_decode: sparse_decode_function(field, definition_map) + ] + end - defp sparse_skip_descriptor_row(field, definition_map) do - {skip_repeated?, skip_bytes?, skip} = skip_descriptor(field, definition_map) - - [ - field_id: field.id, - field_name: Name.field_name(field.name), - field_repeated: sparse_repeated?(field), - field_decode: sparse_decode_function(field, definition_map), - field_skip_repeated: skip_repeated?, - field_skip_bytes: skip_bytes?, - field_skip: skip - ] - end + defp sparse_skip_descriptor_row(field, definition_map) do + {skip_repeated?, skip_bytes?, skip} = skip_descriptor(field, definition_map) + + [ + field_id: field.id, + field_name: Name.field_name(field.name), + field_repeated: sparse_repeated?(field), + field_decode: sparse_decode_function(field, definition_map), + field_skip_repeated: skip_repeated?, + field_skip_bytes: skip_bytes?, + field_skip: skip + ] + end - defp skip_descriptor(%{array?: true, type: "byte"}, _definition_map), - do: {false, true, :kiwi_skip_bytes_value} + defp skip_descriptor(%{array?: true, type: "byte"}, _definition_map), + do: {false, true, :kiwi_skip_bytes_value} - defp skip_descriptor(%{array?: true} = field, definition_map) do - {_repeated?, _bytes?, skip} = skip_descriptor(%{field | array?: false}, definition_map) - {true, false, skip} - end + defp skip_descriptor(%{array?: true} = field, definition_map) do + {_repeated?, _bytes?, skip} = skip_descriptor(%{field | array?: false}, definition_map) + {true, false, skip} + end - defp skip_descriptor(%{type: type}, definition_map), - do: {false, false, Skip.Descriptor.scalar_function(type, definition_map)} + defp skip_descriptor(%{type: type}, definition_map), + do: {false, false, Skip.Descriptor.scalar_function(type, definition_map)} - defp sparse_function_name(name), - do: Identifier.atom!("decode_sparse_#{Name.rust_identifier(name)}_from_decoder") + defp sparse_function_name(name), + do: Identifier.atom!("decode_sparse_#{Name.rust_identifier(name)}_from_decoder") - defp descriptor_scalar_function(%{type: type}, definition_map) do - cond do - KiwiCodec.PrimitiveType.name?(type) -> - ["kiwi_sparse_", type, "_value"] + defp descriptor_scalar_function(%{type: type}, definition_map) do + cond do + KiwiCodec.PrimitiveType.name?(type) -> + ["kiwi_sparse_", type, "_value"] - Map.has_key?(definition_map, type) -> - ["decode_sparse_", Name.rust_identifier(type), "_from_decoder"] + Map.has_key?(definition_map, type) -> + ["decode_sparse_", Name.rust_identifier(type), "_from_decoder"] + end end - end - defp module_name(module_prefix, name), do: "Elixir.#{module_prefix}.#{name}" + defp module_name(module_prefix, name), do: "Elixir.#{module_prefix}.#{name}" + end end diff --git a/codegen/kiwi_codec/rustler_generator/sparse/helpers.ex b/codegen/kiwi_codec/rustler_generator/sparse/helpers.ex index 75a282b..0154e3b 100644 --- a/codegen/kiwi_codec/rustler_generator/sparse/helpers.ex +++ b/codegen/kiwi_codec/rustler_generator/sparse/helpers.ex @@ -1,555 +1,509 @@ -defmodule KiwiCodec.RustlerGenerator.Sparse.Helpers do - @moduledoc """ - Builds RustQ-authored shared sparse helper functions for generated Rustler decoders. - - These helpers are used when callers provide `:decoder_sources`, allowing - RustQ to read the downstream `Decoder` implementation and infer fallible - method propagation from real Rust metadata. - """ - - alias RustQ.Meta.AST, as: MetaAST - - @macros [ - :kiwi_sparse_enum_decoder, - :kiwi_sparse_struct_decoder, - :kiwi_sparse_message_descriptor_decoder, - :kiwi_sparse_skip_message_descriptor_decoder - ] - - @functions [ - :kiwi_sparse_bool_value, - :kiwi_sparse_byte_value, - :kiwi_sparse_float_value, - :kiwi_sparse_int_value, - :kiwi_sparse_int64_value, - :kiwi_sparse_string_value, - :kiwi_sparse_uint_value, - :kiwi_sparse_uint64_value, - :kiwi_sparse_bytes_value, - :kiwi_sparse_enum_value, - :kiwi_sparse_field_value, - :kiwi_sparse_struct_field_value, - :kiwi_sparse_struct_fields, - :kiwi_skip_descriptor_kind, - :kiwi_sparse_struct_fields_remaining, - :kiwi_sparse_message_fields, - :kiwi_sparse_message_fields_remaining - ] - - @spec macro_call(atom(), keyword()) :: RustQ.Rust.Fragment.t() - def macro_call(name, args) do - [] - |> generated_module!() - |> MetaAST.macro_call!(name, args) - end - - @spec fragments([Path.t()] | Path.t(), keyword()) :: [RustQ.Rust.Fragment.t()] - def fragments(decoder_sources, opts \\ []) do - module = generated_module!(decoder_sources) - macros = Keyword.get(opts, :macros, @macros) +if Code.ensure_loaded?(RustQ) do + defmodule KiwiCodec.RustlerGenerator.Sparse.Helpers do + @moduledoc """ + Builds RustQ-authored shared sparse helper functions for generated Rustler decoders. + + These helpers are used when callers provide `:decoder_sources`, allowing + RustQ to read the downstream `Decoder` implementation and infer fallible + method propagation from real Rust metadata. + """ + + alias RustQ.Meta.AST, as: MetaAST + + @macros [ + :kiwi_sparse_enum_decoder, + :kiwi_sparse_struct_decoder, + :kiwi_sparse_message_descriptor_decoder, + :kiwi_sparse_skip_message_descriptor_decoder + ] - module.__rustq_type_items__() ++ - MetaAST.macro_items!(module, macros) ++ MetaAST.functions!(module, @functions) - end + @functions [ + :kiwi_sparse_bool_value, + :kiwi_sparse_byte_value, + :kiwi_sparse_float_value, + :kiwi_sparse_int_value, + :kiwi_sparse_int64_value, + :kiwi_sparse_string_value, + :kiwi_sparse_uint_value, + :kiwi_sparse_uint64_value, + :kiwi_sparse_bytes_value, + :kiwi_sparse_enum_value, + :kiwi_sparse_field_value, + :kiwi_sparse_struct_field_value, + :kiwi_sparse_struct_fields, + :kiwi_skip_descriptor_kind, + :kiwi_sparse_struct_fields_remaining, + :kiwi_sparse_message_fields, + :kiwi_sparse_message_fields_remaining + ] - defp generated_module!(decoder_sources) do - decoder_sources = KiwiCodec.RustlerGenerator.Decoder.sources(decoder_sources) + @spec macro_call(atom(), keyword()) :: RustQ.Rust.Fragment.t() + def macro_call(name, args) do + [] + |> generated_module!() + |> MetaAST.macro_call!(name, args) + end - module = - Module.concat([ - KiwiCodec.RustlerGenerator.Generated, - "SparseHelpers#{:erlang.phash2(decoder_sources)}" - ]) + @spec fragments([Path.t()] | Path.t(), keyword()) :: [RustQ.Rust.Fragment.t()] + def fragments(decoder_sources, opts \\ []) do + module = generated_module!(decoder_sources) + macros = Keyword.get(opts, :macros, @macros) - if Code.ensure_loaded?(module) do - module - else - Module.create(module, quoted_module(decoder_sources), Macro.Env.location(__ENV__)) - module + module.__rustq_type_items__() ++ + MetaAST.macro_items!(module, macros) ++ MetaAST.functions!(module, @functions) end - end - defp quoted_module(decoder_sources) do - quote do - use RustQ.Meta, - rust_sources: unquote(decoder_sources), - callable_modules: [ - KiwiCodec.RustlerGenerator.AtomCache, - KiwiCodec.RustlerGenerator.Skip.ValueHelpers, - KiwiCodec.RustlerGenerator.UnknownFields - ] - - alias RustQ.Type, as: R - - @type kiwi_sparse_decode_fn :: - R.raw(:"for<'a> fn(Env<'a>, &mut Decoder<'_>) -> NifResult>") - - @type kiwi_sparse_enum_variant :: %{ - required(:value) => R.u32(), - required(:name) => R.raw(:"&'static str") - } - - @type kiwi_sparse_field :: %{ - required(:id) => R.u32(), - required(:name) => R.raw(:"&'static str"), - required(:repeated) => R.bool(), - required(:decode) => R.path(:KiwiSparseDecodeFn) - } - - @type kiwi_sparse_struct_field :: %{ - required(:name) => R.raw(:"&'static str"), - required(:repeated) => R.bool(), - required(:decode) => R.path(:KiwiSparseDecodeFn) - } - - unquote_splicing(macro_definitions()) - unquote_splicing(value_helper_definitions()) - unquote_splicing(message_helper_definitions()) + defp generated_module!(decoder_sources) do + decoder_sources = KiwiCodec.RustlerGenerator.Decoder.sources(decoder_sources) + + module = + Module.concat([ + KiwiCodec.RustlerGenerator.Generated, + "SparseHelpers#{:erlang.phash2(decoder_sources)}" + ]) + + if Code.ensure_loaded?(module) do + module + else + Module.create(module, quoted_module(decoder_sources), Macro.Env.location(__ENV__)) + module + end end - end - defp macro_definitions do - [ + defp quoted_module(decoder_sources) do quote do - defrustmacro kiwi_sparse_enum_decoder( - fn: name(:ident), - env: env(:ident), - decoder: decoder(:ident), - variants: - repeat do - variant_value(:literal) - variant_name(:literal) - end - ) do - @spec name( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))) - ) :: R.nif_result(term()) - defrust name(env, decoder) do - kiwi_sparse_enum_value( - env, - decoder, - array([ - repeat variants do - struct_literal(KiwiSparseEnumVariant, - value: variant_value, - name: variant_name - ) - end - ]) - ) - end - end - end, - quote do - defrustmacro kiwi_sparse_struct_decoder( - fn: name(:ident), - env: env(:ident), - decoder: decoder(:ident), - module: module_name(:literal), - capacity: capacity(:literal), - fields: - repeat do - field_name(:literal) - field_repeated(:literal) - field_decode(:ident) - end - ) do - @spec name( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))) - ) :: R.nif_result(term()) - defrust name(env, decoder) do - kiwi_sparse_struct_fields( - env, - decoder, - module_name, - capacity, - array([ - repeat fields do - struct_literal(KiwiSparseStructField, - name: field_name, - repeated: field_repeated, - decode: field_decode - ) - end - ]) - ) + use RustQ.Meta, + rust_sources: unquote(decoder_sources), + callable_modules: [ + KiwiCodec.RustlerGenerator.AtomCache, + KiwiCodec.RustlerGenerator.Skip.ValueHelpers, + KiwiCodec.RustlerGenerator.UnknownFields + ] + + alias RustQ.Type, as: R + + @type kiwi_sparse_decode_fn :: + R.raw(:"for<'a> fn(Env<'a>, &mut Decoder<'_>) -> NifResult>") + + @type kiwi_sparse_enum_variant :: %{ + required(:value) => R.u32(), + required(:name) => R.raw(:"&'static str") + } + + @type kiwi_sparse_field :: %{ + required(:id) => R.u32(), + required(:name) => R.raw(:"&'static str"), + required(:repeated) => boolean(), + required(:decode) => R.path(:KiwiSparseDecodeFn) + } + + @type kiwi_sparse_struct_field :: %{ + required(:name) => R.raw(:"&'static str"), + required(:repeated) => boolean(), + required(:decode) => R.path(:KiwiSparseDecodeFn) + } + + unquote_splicing(macro_definitions()) + unquote_splicing(value_helper_definitions()) + unquote_splicing(message_helper_definitions()) + end + end + + defp macro_definitions do + [ + quote do + defrustmacro kiwi_sparse_enum_decoder( + fn: name(:ident), + env: env(:ident), + decoder: decoder(:ident), + variants: + repeat do + variant_value(:literal) + variant_name(:literal) + end + ) do + @spec name( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))) + ) :: R.nif_result(term()) + defrust name(env, decoder) do + kiwi_sparse_enum_value( + env, + decoder, + array([ + repeat variants do + struct_literal(KiwiSparseEnumVariant, + value: variant_value, + name: variant_name + ) + end + ]) + ) + end end - end - end, - quote do - defrustmacro kiwi_sparse_message_descriptor_decoder( - fn: name(:ident), - env: env(:ident), - decoder: decoder(:ident), - module: module_name(:literal), - definition: definition_name(:literal), - capacity: capacity(:literal), - fields: - repeat do - field_id(:literal) - field_name(:literal) - field_repeated(:literal) - field_decode(:ident) - end - ) do - @spec name( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))) - ) :: R.nif_result(term()) - defrust name(env, decoder) do - kiwi_sparse_message_fields( - env, - decoder, - module_name, - definition_name, - capacity, - array([ - repeat fields do - struct_literal(KiwiSparseField, - id: field_id, - name: field_name, - repeated: field_repeated, - decode: field_decode - ) - end - ]) - ) + end, + quote do + defrustmacro kiwi_sparse_struct_decoder( + fn: name(:ident), + env: env(:ident), + decoder: decoder(:ident), + module: module_name(:literal), + capacity: capacity(:literal), + fields: + repeat do + field_name(:literal) + field_repeated(:literal) + field_decode(:ident) + end + ) do + @spec name( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))) + ) :: R.nif_result(term()) + defrust name(env, decoder) do + kiwi_sparse_struct_fields( + env, + decoder, + module_name, + capacity, + array([ + repeat fields do + struct_literal(KiwiSparseStructField, + name: field_name, + repeated: field_repeated, + decode: field_decode + ) + end + ]) + ) + end end - end - end, - quote do - defrustmacro kiwi_sparse_skip_message_descriptor_decoder( - sparse_fn: sparse_name(:ident), - skip_fn: skip_name(:ident), - env: env(:ident), - decoder: decoder(:ident), - module: module_name(:literal), - definition: definition_name(:literal), - capacity: capacity(:literal), - fields: - repeat do - field_id(:literal) - field_name(:literal) - field_repeated(:literal) - field_decode(:ident) - field_skip_repeated(:literal) - field_skip_bytes(:literal) - field_skip(:ident) - end - ) do - @spec sparse_name( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))) - ) :: R.nif_result(term()) - defrust sparse_name(env, decoder) do - kiwi_sparse_message_fields( - env, - decoder, - module_name, - definition_name, - capacity, - array([ - repeat fields do - struct_literal(KiwiSparseField, - id: field_id, - name: field_name, - repeated: field_repeated, - decode: field_decode - ) - end - ]) - ) + end, + quote do + defrustmacro kiwi_sparse_message_descriptor_decoder( + fn: name(:ident), + env: env(:ident), + decoder: decoder(:ident), + module: module_name(:literal), + definition: definition_name(:literal), + capacity: capacity(:literal), + fields: + repeat do + field_id(:literal) + field_name(:literal) + field_repeated(:literal) + field_decode(:ident) + end + ) do + @spec name( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))) + ) :: R.nif_result(term()) + defrust name(env, decoder) do + kiwi_sparse_message_fields( + env, + decoder, + module_name, + definition_name, + capacity, + array([ + repeat fields do + struct_literal(KiwiSparseField, + id: field_id, + name: field_name, + repeated: field_repeated, + decode: field_decode + ) + end + ]) + ) + end end - - @spec skip_name(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: R.nif_result(R.unit()) - defrust skip_name(decoder) do - kiwi_skip_message_fields( - decoder, - definition_name, - array([ - repeat fields do - struct_literal(KiwiSkipField, - id: field_id, - kind: - kiwi_skip_descriptor_kind( - field_skip_repeated, - field_skip_bytes, - field_skip - ) - ) - end - ]) - ) + end, + quote do + defrustmacro kiwi_sparse_skip_message_descriptor_decoder( + sparse_fn: sparse_name(:ident), + skip_fn: skip_name(:ident), + env: env(:ident), + decoder: decoder(:ident), + module: module_name(:literal), + definition: definition_name(:literal), + capacity: capacity(:literal), + fields: + repeat do + field_id(:literal) + field_name(:literal) + field_repeated(:literal) + field_decode(:ident) + field_skip_repeated(:literal) + field_skip_bytes(:literal) + field_skip(:ident) + end + ) do + @spec sparse_name( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))) + ) :: R.nif_result(term()) + defrust sparse_name(env, decoder) do + kiwi_sparse_message_fields( + env, + decoder, + module_name, + definition_name, + capacity, + array([ + repeat fields do + struct_literal(KiwiSparseField, + id: field_id, + name: field_name, + repeated: field_repeated, + decode: field_decode + ) + end + ]) + ) + end + + @spec skip_name(R.mut_ref(R.path(:Decoder, R.lifetime(:_)))) :: R.nif_result(R.unit()) + defrust skip_name(decoder) do + kiwi_skip_message_fields( + decoder, + definition_name, + array([ + repeat fields do + struct_literal(KiwiSkipField, + id: field_id, + kind: + kiwi_skip_descriptor_kind( + field_skip_repeated, + field_skip_bytes, + field_skip + ) + ) + end + ]) + ) + end end end - end - ] - end + ] + end - defp value_helper_definitions do - [ - quoted_encoded_value(:kiwi_sparse_bool_value, :read_bool), - quoted_encoded_value(:kiwi_sparse_byte_value, :read_byte), - quoted_passthrough_value(:kiwi_sparse_float_value, :read_var_float), - quoted_encoded_value(:kiwi_sparse_int_value, :read_var_int), - quoted_encoded_value(:kiwi_sparse_int64_value, :read_var_int64), - quoted_passthrough_value(:kiwi_sparse_string_value, :read_string), - quoted_encoded_value(:kiwi_sparse_uint_value, :read_var_uint), - quoted_encoded_value(:kiwi_sparse_uint64_value, :read_var_uint64), - quoted_passthrough_value(:kiwi_sparse_bytes_value, :read_byte_array) - ] - end + defp value_helper_definitions do + [ + quoted_encoded_value(:kiwi_sparse_bool_value, :read_bool), + quoted_encoded_value(:kiwi_sparse_byte_value, :read_byte), + quoted_passthrough_value(:kiwi_sparse_float_value, :read_var_float), + quoted_encoded_value(:kiwi_sparse_int_value, :read_var_int), + quoted_encoded_value(:kiwi_sparse_int64_value, :read_var_int64), + quoted_passthrough_value(:kiwi_sparse_string_value, :read_string), + quoted_encoded_value(:kiwi_sparse_uint_value, :read_var_uint), + quoted_encoded_value(:kiwi_sparse_uint64_value, :read_var_uint64), + quoted_passthrough_value(:kiwi_sparse_bytes_value, :read_byte_array) + ] + end - defp quoted_encoded_value(function, read_method) do - quote do - @spec unquote(function)( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))) - ) :: R.nif_result(term()) - defrust unquote(function)(env, decoder) do - value = decoder.unquote(read_method)() - {:ok, value.encode(env)} + defp quoted_encoded_value(function, read_method) do + quote do + @spec unquote(function)( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))) + ) :: R.nif_result(term()) + defrust unquote(function)(env, decoder) do + value = decoder.unquote(read_method)() + {:ok, value.encode(env)} + end end end - end - defp quoted_passthrough_value(function, read_method) do - quote do - @spec unquote(function)( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))) - ) :: R.nif_result(term()) - defrust unquote(function)(env, decoder) do - decoder.unquote(read_method)(env) + defp quoted_passthrough_value(function, read_method) do + quote do + @spec unquote(function)( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))) + ) :: R.nif_result(term()) + defrust unquote(function)(env, decoder) do + decoder.unquote(read_method)(env) + end end end - end - defp message_helper_definitions do - [ - quote_enum_value(), - quote_field_value(), - quote_struct_field_value(), - quote_struct_fields(), - quote_struct_fields_remaining(), - quote_skip_descriptor_kind(), - quote_message_fields(), - quote_message_fields_remaining() - ] - end + defp message_helper_definitions do + [ + quote_enum_value(), + quote_field_value(), + quote_struct_field_value(), + quote_struct_fields(), + quote_struct_fields_remaining(), + quote_skip_descriptor_kind(), + quote_message_fields(), + quote_message_fields_remaining() + ] + end - defp quote_enum_value do - quote do - @spec kiwi_sparse_enum_value( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))), - R.slice(R.path(:KiwiSparseEnumVariant)) - ) :: R.nif_result(term()) - defrust kiwi_sparse_enum_value(env, decoder, variants) do - value = decoder.read_var_uint() - - case variants.binary_search_by_key(value, fn variant -> variant.value end) do - {:ok, index} -> - variant = variants.get(index).unwrap() - {:ok, kiwi_atom(env, variant.name).encode(env)} - - {:error, _index} -> - {:ok, value.encode(env)} + defp quote_enum_value do + quote do + @spec kiwi_sparse_enum_value( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))), + R.slice(R.path(:KiwiSparseEnumVariant)) + ) :: R.nif_result(term()) + defrust kiwi_sparse_enum_value(env, decoder, variants) do + value = decoder.read_var_uint() + + case variants.binary_search_by_key(value, fn variant -> variant.value end) do + {:ok, index} -> + variant = variants.get(index).unwrap() + {:ok, kiwi_atom(env, variant.name).encode(env)} + + {:error, _index} -> + {:ok, value.encode(env)} + end end end end - end - defp quote_field_value do - quote do - @spec kiwi_sparse_field_value( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))), - R.ref(R.path(:KiwiSparseField)) - ) :: R.nif_result(term()) - defrust kiwi_sparse_field_value(env, decoder, field) do - decode = field.decode - - if field.repeated do - values = decoder.read_repeated(fn decoder -> decode(env, decoder) end) - {:ok, values.encode(env)} - else - decode(env, decoder) + defp quote_field_value do + quote do + @spec kiwi_sparse_field_value( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))), + R.ref(R.path(:KiwiSparseField)) + ) :: R.nif_result(term()) + defrust kiwi_sparse_field_value(env, decoder, field) do + decode = field.decode + + if field.repeated do + values = decoder.read_repeated(fn decoder -> decode(env, decoder) end) + {:ok, values.encode(env)} + else + decode(env, decoder) + end end end end - end - defp quote_struct_field_value do - quote do - @spec kiwi_sparse_struct_field_value( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))), - R.ref(R.path(:KiwiSparseStructField)) - ) :: R.nif_result(term()) - defrust kiwi_sparse_struct_field_value(env, decoder, field) do - decode = field.decode - - if field.repeated do - value = decoder.read_repeated(fn decoder -> decode(env, decoder) end) - {:ok, value.encode(env)} - else - decode(env, decoder) + defp quote_struct_field_value do + quote do + @spec kiwi_sparse_struct_field_value( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))), + R.ref(R.path(:KiwiSparseStructField)) + ) :: R.nif_result(term()) + defrust kiwi_sparse_struct_field_value(env, decoder, field) do + decode = field.decode + + if field.repeated do + value = decoder.read_repeated(fn decoder -> decode(env, decoder) end) + {:ok, value.encode(env)} + else + decode(env, decoder) + end end end end - end - defp quote_struct_fields do - quote do - @spec kiwi_sparse_struct_fields( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))), - R.raw(:"&'static str"), - R.usize(), - R.slice(R.path(:KiwiSparseStructField)) - ) :: R.nif_result(term()) - defrust kiwi_sparse_struct_fields(env, decoder, module_name, capacity, fields) do - module_atom = kiwi_atom(env, module_name) - module_key_atom = kiwi_atom(env, "__kiwi_module__") - keys = Vec.with_capacity(capacity) - values = Vec.with_capacity(capacity) - keys.push(module_key_atom.encode(env)) - values.push(module_atom.encode(env)) - - kiwi_sparse_struct_fields_remaining( - env, - decoder, - fields, - 0, - keys, - values - ) - - Term.map_from_term_arrays(env, ref(keys), ref(values)) - end - end - end + defp quote_struct_fields do + quote do + @spec kiwi_sparse_struct_fields( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))), + R.raw(:"&'static str"), + R.usize(), + R.slice(R.path(:KiwiSparseStructField)) + ) :: R.nif_result(term()) + defrust kiwi_sparse_struct_fields(env, decoder, module_name, capacity, fields) do + module_atom = kiwi_atom(env, module_name) + module_key_atom = kiwi_atom(env, "__kiwi_module__") + keys = Vec.with_capacity(capacity) + values = Vec.with_capacity(capacity) + keys.push(module_key_atom.encode(env)) + values.push(module_atom.encode(env)) + + kiwi_sparse_struct_fields_remaining( + env, + decoder, + fields, + 0, + keys, + values + ) - defp quote_struct_fields_remaining do - quote do - @spec kiwi_sparse_struct_fields_remaining( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))), - R.slice(R.path(:KiwiSparseStructField)), - R.usize(), - R.mut_ref(R.vec(term())), - R.mut_ref(R.vec(term())) - ) :: R.nif_result(R.unit()) - defrust kiwi_sparse_struct_fields_remaining(env, decoder, fields, index, keys, values) do - if index == fields.len() do - :ok - else - field = fields.get(index).unwrap() - keys.push(kiwi_atom(env, field.name).encode(env)) - - values.push(kiwi_sparse_struct_field_value(env, decoder, field)) - - kiwi_sparse_struct_fields_remaining(env, decoder, fields, index + 1, keys, values) + Term.map_from_term_arrays(env, ref(keys), ref(values)) end end end - end - defp quote_skip_descriptor_kind do - quote do - @spec kiwi_skip_descriptor_kind(R.bool(), R.bool(), R.path(:KiwiSkipFn)) :: - R.path(:KiwiSkipKind) - defrust kiwi_skip_descriptor_kind(repeated, bytes, skip) do - non_bytes_kind = - if repeated do - enum_variant(KiwiSkipKind, :repeated, skip) + defp quote_struct_fields_remaining do + quote do + @spec kiwi_sparse_struct_fields_remaining( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))), + R.slice(R.path(:KiwiSparseStructField)), + R.usize(), + R.mut_ref(R.vec(term())), + R.mut_ref(R.vec(term())) + ) :: R.nif_result(R.unit()) + defrust kiwi_sparse_struct_fields_remaining(env, decoder, fields, index, keys, values) do + if index == fields.len() do + :ok else - enum_variant(KiwiSkipKind, :one, skip) - end + field = fields.get(index).unwrap() + keys.push(kiwi_atom(env, field.name).encode(env)) + + values.push(kiwi_sparse_struct_field_value(env, decoder, field)) - if bytes do - enum_variant(KiwiSkipKind, :bytes) - else - non_bytes_kind + kiwi_sparse_struct_fields_remaining(env, decoder, fields, index + 1, keys, values) + end end end end - end - defp quote_message_fields do - quote do - @spec kiwi_sparse_message_fields( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))), - R.raw(:"&'static str"), - R.str(), - R.usize(), - R.slice(R.path(:KiwiSparseField)) - ) :: R.nif_result(term()) - defrust kiwi_sparse_message_fields( - env, - decoder, - module_name, - definition_name, - capacity, - fields - ) do - module_atom = kiwi_atom(env, module_name) - module_key_atom = kiwi_atom(env, "__kiwi_module__") - keys = Vec.with_capacity(capacity) - values = Vec.with_capacity(capacity) - keys.push(module_key_atom.encode(env)) - values.push(module_atom.encode(env)) - - kiwi_sparse_message_fields_remaining( - env, - decoder, - definition_name, - fields, - keys, - values - ) - - Term.map_from_term_arrays(env, ref(keys), ref(values)) + defp quote_skip_descriptor_kind do + quote do + @spec kiwi_skip_descriptor_kind(boolean(), boolean(), R.path(:KiwiSkipFn)) :: + R.path(:KiwiSkipKind) + defrust kiwi_skip_descriptor_kind(repeated, bytes, skip) do + non_bytes_kind = + if repeated do + enum_variant(KiwiSkipKind, :repeated, skip) + else + enum_variant(KiwiSkipKind, :one, skip) + end + + if bytes do + enum_variant(KiwiSkipKind, :bytes) + else + non_bytes_kind + end + end end end - end - - defp quote_message_fields_remaining do - quote do - @spec kiwi_sparse_message_fields_remaining( - R.path(:Env, R.lifetime(:a)), - R.mut_ref(R.path(:Decoder, R.lifetime(:_))), - R.str(), - R.slice(R.path(:KiwiSparseField)), - R.mut_ref(R.vec(term())), - R.mut_ref(R.vec(term())) - ) :: R.nif_result(R.unit()) - defrust kiwi_sparse_message_fields_remaining( - env, - decoder, - definition_name, - fields, - keys, - values - ) do - field_id = decoder.read_var_uint() - - if field_id == 0 do - :ok - else - case fields.binary_search_by_key(field_id, fn field -> field.id end) do - {:ok, index} -> - field = fields.get(index).unwrap() - keys.push(kiwi_atom(env, field.name).encode(env)) - values.push(kiwi_sparse_field_value(env, decoder, field)) - {:error, _index} -> - kiwi_unknown_field(decoder, definition_name, field_id) - end + defp quote_message_fields do + quote do + @spec kiwi_sparse_message_fields( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))), + R.raw(:"&'static str"), + R.str(), + R.usize(), + R.slice(R.path(:KiwiSparseField)) + ) :: R.nif_result(term()) + defrust kiwi_sparse_message_fields( + env, + decoder, + module_name, + definition_name, + capacity, + fields + ) do + module_atom = kiwi_atom(env, module_name) + module_key_atom = kiwi_atom(env, "__kiwi_module__") + keys = Vec.with_capacity(capacity) + values = Vec.with_capacity(capacity) + keys.push(module_key_atom.encode(env)) + values.push(module_atom.encode(env)) kiwi_sparse_message_fields_remaining( env, @@ -559,6 +513,54 @@ defmodule KiwiCodec.RustlerGenerator.Sparse.Helpers do keys, values ) + + Term.map_from_term_arrays(env, ref(keys), ref(values)) + end + end + end + + defp quote_message_fields_remaining do + quote do + @spec kiwi_sparse_message_fields_remaining( + R.path(:Env, R.lifetime(:a)), + R.mut_ref(R.path(:Decoder, R.lifetime(:_))), + R.str(), + R.slice(R.path(:KiwiSparseField)), + R.mut_ref(R.vec(term())), + R.mut_ref(R.vec(term())) + ) :: R.nif_result(R.unit()) + defrust kiwi_sparse_message_fields_remaining( + env, + decoder, + definition_name, + fields, + keys, + values + ) do + field_id = decoder.read_var_uint() + + if field_id == 0 do + :ok + else + case fields.binary_search_by_key(field_id, fn field -> field.id end) do + {:ok, index} -> + field = fields.get(index).unwrap() + keys.push(kiwi_atom(env, field.name).encode(env)) + values.push(kiwi_sparse_field_value(env, decoder, field)) + + {:error, _index} -> + kiwi_unknown_field(decoder, definition_name, field_id) + end + + kiwi_sparse_message_fields_remaining( + env, + decoder, + definition_name, + fields, + keys, + values + ) + end end end end diff --git a/codegen/kiwi_codec/rustler_generator/splice.ex b/codegen/kiwi_codec/rustler_generator/splice.ex index 4200cf3..9cbd01f 100644 --- a/codegen/kiwi_codec/rustler_generator/splice.ex +++ b/codegen/kiwi_codec/rustler_generator/splice.ex @@ -1,107 +1,110 @@ -defmodule KiwiCodec.RustlerGenerator.Splice do - @moduledoc """ - Static RustQ splice fragments required by generated Rustler decoders. +if Code.ensure_loaded?(RustQ) do + defmodule KiwiCodec.RustlerGenerator.Splice do + @moduledoc """ + Static RustQ splice fragments required by generated Rustler decoders. - Generated schema code targets compact Rust macros so high-level Elixir - generators can stay semantic without expanding large repetitive Rust bodies. - The shared macros and helper functions are authored through RustQ `defrust`, - `defrustmacro`, and type metadata in the helper modules selected here. - """ + Generated schema code targets compact Rust macros so high-level Elixir + generators can stay semantic without expanding large repetitive Rust bodies. + The shared macros and helper functions are authored through RustQ `defrust`, + `defrustmacro`, and type metadata in the helper modules selected here. + """ - alias KiwiCodec.RustlerGenerator.AtomCache - alias KiwiCodec.RustlerGenerator.Decoder - alias KiwiCodec.RustlerGenerator.Full - alias KiwiCodec.RustlerGenerator.Skip - alias KiwiCodec.RustlerGenerator.Sparse - alias KiwiCodec.RustlerGenerator.UnknownFields - alias RustQ.Rust.AST - alias RustQ.Rustler.{Atom, Term} + alias KiwiCodec.RustlerGenerator.AtomCache + alias KiwiCodec.RustlerGenerator.Decoder + alias KiwiCodec.RustlerGenerator.Full + alias KiwiCodec.RustlerGenerator.Skip + alias KiwiCodec.RustlerGenerator.Sparse + alias KiwiCodec.RustlerGenerator.UnknownFields + alias RustQ.Rust.AST + alias RustQ.Rustler.{Atom, Term} - @spec rustler_helpers(keyword()) :: [AST.item()] - def rustler_helpers(opts \\ []) do - features = Keyword.get(opts, :features, [:full]) - decoder_sources = helper_decoder_sources(opts, features) + @spec rustler_helpers(keyword()) :: [AST.item()] + def rustler_helpers(opts \\ []) do + features = Keyword.get(opts, :features, [:full]) + decoder_sources = helper_decoder_sources(opts, features) - decoder_macros(features, decoder_sources, opts) ++ - UnknownFields.fragments(Keyword.get(opts, :unknown_fields, :error)) ++ - Atom.cached([]) ++ - Term.helpers( - include: [ - :cached_struct_keys, - :default_struct_values, - :make_struct_from_arrays - ] - ) - end + decoder_macros(features, decoder_sources, opts) ++ + UnknownFields.fragments(Keyword.get(opts, :unknown_fields, :error)) ++ + Atom.cached([]) ++ + Term.helpers( + include: [ + :cached_struct_keys, + :default_struct_values, + :make_struct_from_arrays + ] + ) + end - defp helper_decoder_sources(opts, _features) do - opts |> Keyword.get(:decoder_sources, []) |> Decoder.sources() - end + defp helper_decoder_sources(opts, _features) do + opts |> Keyword.get(:decoder_sources, []) |> Decoder.sources() + end - defp decoder_macros(features, decoder_sources, opts) do - [ - if(:full in features, do: full_decoder_macro_fragments(), else: []), - if(:skip in features, - do: - skip_decoder_fragments( - decoder_sources, - shared_sparse_skip?(features, opts), - :sparse in features - ), - else: [] - ), - if(:sparse in features or :records in features, do: AtomCache.fragments(), else: []), - if(:sparse in features, - do: sparse_decoder_fragments(decoder_sources, shared_sparse_skip?(features, opts), opts), - else: [] - ) - ] - |> List.flatten() - end + defp decoder_macros(features, decoder_sources, opts) do + [ + if(:full in features, do: full_decoder_macro_fragments(), else: []), + if(:skip in features, + do: + skip_decoder_fragments( + decoder_sources, + shared_sparse_skip?(features, opts), + :sparse in features + ), + else: [] + ), + if(:sparse in features or :records in features, do: AtomCache.fragments(), else: []), + if(:sparse in features, + do: + sparse_decoder_fragments(decoder_sources, shared_sparse_skip?(features, opts), opts), + else: [] + ) + ] + |> List.flatten() + end - defp full_decoder_macro_fragments do - [ - Full.Helpers.fragments([ - :kiwi_enum_decoder, - :kiwi_struct_decoder, - :kiwi_message_decoder - ]), - [] - ] - |> List.flatten() - end + defp full_decoder_macro_fragments do + [ + Full.Helpers.fragments([ + :kiwi_enum_decoder, + :kiwi_struct_decoder, + :kiwi_message_decoder + ]), + [] + ] + |> List.flatten() + end - # Sparse helpers define `kiwi_skip_descriptor_kind` for themselves, so skip decoders bring - # it only without them. A shared sparse and skip descriptor replaces the skip message macro. - defp skip_decoder_fragments(decoder_sources, shared?, sparse?) do - [ - Skip.Helpers.fragments(decoder_sources), - Skip.ValueHelpers.dispatch_fragments(descriptor_kind?: not sparse?), - if(shared?, do: [], else: Skip.ValueHelpers.macro_fragments([:kiwi_skip_message_decoder])) - ] - |> List.flatten() - end + # Sparse helpers define `kiwi_skip_descriptor_kind` for themselves, so skip decoders bring + # it only without them. A shared sparse and skip descriptor replaces the skip message macro. + defp skip_decoder_fragments(decoder_sources, shared?, sparse?) do + [ + Skip.Helpers.fragments(decoder_sources), + Skip.ValueHelpers.dispatch_fragments(descriptor_kind?: not sparse?), + if(shared?, do: [], else: Skip.ValueHelpers.macro_fragments([:kiwi_skip_message_decoder])) + ] + |> List.flatten() + end - defp shared_sparse_skip?(features, opts) do - :sparse in features and :skip in features and - Keyword.get(opts, :sparse_messages, :match) == :descriptor - end + defp shared_sparse_skip?(features, opts) do + :sparse in features and :skip in features and + Keyword.get(opts, :sparse_messages, :match) == :descriptor + end - defp sparse_decoder_fragments(decoder_sources, shared?, _opts) do - Sparse.Helpers.fragments(decoder_sources, macros: sparse_helper_macros(shared?)) - end + defp sparse_decoder_fragments(decoder_sources, shared?, _opts) do + Sparse.Helpers.fragments(decoder_sources, macros: sparse_helper_macros(shared?)) + end - defp sparse_helper_macros(true), - do: [ - :kiwi_sparse_enum_decoder, - :kiwi_sparse_struct_decoder, - :kiwi_sparse_skip_message_descriptor_decoder - ] + defp sparse_helper_macros(true), + do: [ + :kiwi_sparse_enum_decoder, + :kiwi_sparse_struct_decoder, + :kiwi_sparse_skip_message_descriptor_decoder + ] - defp sparse_helper_macros(false), - do: [ - :kiwi_sparse_enum_decoder, - :kiwi_sparse_struct_decoder, - :kiwi_sparse_message_descriptor_decoder - ] + defp sparse_helper_macros(false), + do: [ + :kiwi_sparse_enum_decoder, + :kiwi_sparse_struct_decoder, + :kiwi_sparse_message_descriptor_decoder + ] + end end diff --git a/codegen/kiwi_codec/rustler_generator/unknown_fields.ex b/codegen/kiwi_codec/rustler_generator/unknown_fields.ex index 54c703f..560bfaf 100644 --- a/codegen/kiwi_codec/rustler_generator/unknown_fields.ex +++ b/codegen/kiwi_codec/rustler_generator/unknown_fields.ex @@ -1,50 +1,52 @@ -defmodule KiwiCodec.RustlerGenerator.UnknownFields do - use RustQ.Meta - - @moduledoc """ - Selects how generated Rustler decoders handle message fields the compiled - schema does not define. - - Every generated message decoder calls `kiwi_unknown_field(decoder, definition, - field)` for such a field and continues with the next one when it returns. - - * `:error` rejects the data with `Error::BadArg`. - * `:runtime_schema` skips the field using a `KiwiRuntimeSchema` decoded from - the schema that accompanies the data, which the `Decoder` returns from - `runtime_schema()`; see `KiwiCodec.RustlerGenerator.Decoder`. - - The runtime schema only skips values; decoded output is unchanged. Kiwi - struct layouts cannot be extended compatibly, so a schema that changes a - struct the compiled decoders read is not supported. - """ - - alias RustQ.Meta.AST, as: MetaAST - alias RustQ.Type, as: R - - @runtime_schema_template Path.expand("templates/unknown_field_runtime_schema.rs", __DIR__) - @external_resource @runtime_schema_template - - @type mode :: :error | :runtime_schema - - @doc """ - Returns the Rust items implementing `mode`. - """ - @spec fragments(mode()) :: [RustQ.Rust.Fragment.t()] - def fragments(:error), do: MetaAST.functions!(__MODULE__, [:kiwi_unknown_field]) - - def fragments(:runtime_schema) do - source = @runtime_schema_template |> File.read!() |> String.trim_trailing() - [RustQ.Rust.fragment(:item, source)] - end - - def fragments(mode) do - raise ArgumentError, - "expected :unknown_fields to be :error or :runtime_schema, got: #{inspect(mode)}" - end - - @spec kiwi_unknown_field(R.mut_ref(R.path(:Decoder, R.lifetime(:_))), R.str(), R.u32()) :: - R.nif_result(R.unit()) - defrust kiwi_unknown_field(_decoder, _definition_name, _field_id) do - {:error, badarg()} +if Code.ensure_loaded?(RustQ) do + defmodule KiwiCodec.RustlerGenerator.UnknownFields do + use RustQ.Meta + + @moduledoc """ + Selects how generated Rustler decoders handle message fields the compiled + schema does not define. + + Every generated message decoder calls `kiwi_unknown_field(decoder, definition, + field)` for such a field and continues with the next one when it returns. + + * `:error` rejects the data with `Error::BadArg`. + * `:runtime_schema` skips the field using a `KiwiRuntimeSchema` decoded from + the schema that accompanies the data, which the `Decoder` returns from + `runtime_schema()`; see `KiwiCodec.RustlerGenerator.Decoder`. + + The runtime schema only skips values; decoded output is unchanged. Kiwi + struct layouts cannot be extended compatibly, so a schema that changes a + struct the compiled decoders read is not supported. + """ + + alias RustQ.Meta.AST, as: MetaAST + alias RustQ.Type, as: R + + @runtime_schema_template Path.expand("templates/unknown_field_runtime_schema.rs", __DIR__) + @external_resource @runtime_schema_template + + @type mode :: :error | :runtime_schema + + @doc """ + Returns the Rust items implementing `mode`. + """ + @spec fragments(mode()) :: [RustQ.Rust.Fragment.t()] + def fragments(:error), do: MetaAST.functions!(__MODULE__, [:kiwi_unknown_field]) + + def fragments(:runtime_schema) do + source = @runtime_schema_template |> File.read!() |> String.trim_trailing() + [RustQ.Rust.fragment(:item, source)] + end + + def fragments(mode) do + raise ArgumentError, + "expected :unknown_fields to be :error or :runtime_schema, got: #{inspect(mode)}" + end + + @spec kiwi_unknown_field(R.mut_ref(R.path(:Decoder, R.lifetime(:_))), R.str(), R.u32()) :: + R.nif_result(R.unit()) + defrust kiwi_unknown_field(_decoder, _definition_name, _field_id) do + {:error, badarg()} + end end end diff --git a/guides/rustler-generator-architecture.md b/guides/internals.md similarity index 59% rename from guides/rustler-generator-architecture.md rename to guides/internals.md index 2be4493..1b882ac 100644 --- a/guides/rustler-generator-architecture.md +++ b/guides/internals.md @@ -1,31 +1,50 @@ -# Rustler generator architecture +# Internals -KiwiCodec's Rustler generator is optimized for two goals at the same time: +How KiwiCodec is put together, for people changing it. Using the Rustler +generator is covered in [Native decoders](native-decoders.md). + +## Layout + +- `lib/` is the codec: the schema parser and binary schema format + (`KiwiCodec.Schema.*`), wire primitives (`KiwiCodec.Wire.*`), the DSL and the + modules it generates (`KiwiCodec.DSL`, `KiwiCodec.GeneratedModule.*`), encoding + and decoding of generated structs (`KiwiCodec.Encoder`, `KiwiCodec.Decoder`), + and the schema interpreter. It has no dependency beyond `varint`. +- `codegen/` is the Rustler generator. Every module there is wrapped in + `if Code.ensure_loaded?(RustQ)`, because RustQ is optional: projects that + depend on KiwiCodec get the generator only where they also depend on RustQ. +- Tests mirror the source tree. `test/package/` builds the package and compiles + it in two consumer projects, one without RustQ and one generating Rust. + +## Rustler generator + +The generator serves two goals at once: 1. Keep generator code semantic and maintainable in Elixir. -2. Keep generated Rust source compact enough for large downstream schemas. +2. Keep generated Rust compact enough for large schemas, such as Figma's. For repetitive schema decoders, those goals are served by a compact macro boundary rather than by fully expanding every decoder body. -## Layers +### Layers The generator is organized as a small pipeline: 1. Schema definitions are parsed and selected by `KiwiCodec.RustlerGenerator`. 2. Definition-specific modules (`Definition`, `Sparse`, and `Skip`) derive semantic field metadata and field expressions from the schema. -3. `KiwiCodec.RustlerGenerator.DecoderMacro` lowers that metadata to compact - Rust macro invocations such as `kiwi_message_decoder!`, - `kiwi_sparse_message_decoder!`, and `kiwi_skip_message_decoder!`. +3. `KiwiCodec.RustlerGenerator.DecoderMacro`, for full decoders, and the + `Sparse` and `Skip` modules lower that metadata to compact Rust macro + invocations such as `kiwi_message_decoder!`, `kiwi_sparse_struct_decoder!`, + and `kiwi_skip_message_decoder!`. 4. `KiwiCodec.RustlerGenerator.Splice` selects the shared Rust macro definitions and helper functions consumed by those invocations. The full, sparse, and skip helper modules author those macros with RustQ `defrustmacro`, `defrust`, type metadata, and method metadata read from the `Decoder` source: the reference `decoder.rs` beside `KiwiCodec.RustlerGenerator.Decoder`, or the caller's `decoder_sources:`. -5. `KiwiCodec.RustlerGenerator.Entrypoint` emits Rustler NIF entrypoints through - RustQ `defrust`, where the wrapper control flow is small and readable. +5. `KiwiCodec.RustlerGenerator.Entrypoint` emits Rustler NIF entrypoints with + RustQ `defnif`, where the wrapper control flow is small and readable. This split keeps schema logic in Elixir while avoiding thousands of repeated expanded Rust function bodies. @@ -35,7 +54,7 @@ the record runtime, the runtime schema that skips unknown fields, and the atom cache. They are ordinary Rust APIs, documented for the Rust code that uses them, rather than repetitive decoder shapes. -## Compact RustQ macro boundary +### Compact RustQ macro boundary The macros selected by `Splice` are a compact RustQ-authored boundary. They are shared implementations for highly repetitive decoder shapes, but they should be @@ -59,21 +78,7 @@ When changing full, sparse, or skip decoder generation: A good change should make the Elixir generator clearer without increasing large schema output size. -## Dogfooding size guardrail - -Figler is the main downstream stress test for generated Rust size. It should be -used as a private dogfood check, not updated or published automatically. - -As of the current generator polish, Figler with local KiwiCodec and -`decoder_sources: ["native/src/runtime.rs"]` should generate approximately: - -- about `13,600` lines -- about `585,000` bytes - -Small formatting or schema changes can move this slightly, but a large jump -usually means a compact macro boundary was accidentally expanded. - -## Tests +### Tests `KiwiCodec.RustlerGenerator.NativeTest` builds the generated decoders for every feature with the reference `Decoder`, loads them as NIFs, and checks what they @@ -81,15 +86,13 @@ return against `KiwiCodec.decode/2` and `KiwiCodec.SchemaInterpreter`. A generator change that produces Rust that does not compile, or decodes differently, fails there; tests of the generated text are not needed for it. -## Practical checklist +### Checklist -Before committing Rustler generator changes: +Before committing generator changes: -1. Capture a generated Rust baseline for a schema that exercises full, sparse, - and skip decoders. -2. Make the smallest semantic change. -3. Confirm generated output is unchanged or intentionally smaller. -4. Run KiwiCodec `mix ci` and warning-free docs. -5. Dogfood Figler with a temporary local KiwiCodec dependency. -6. Restore Figler before committing KiwiCodec changes unless a Figler update was - explicitly requested. +1. Generate Rust for a schema that exercises full, sparse, and skip decoders, + before and after the change, and compare the size. +2. Run `mix ci`, which builds and runs the generated Rust. +3. Generate Figler's decoders with this checkout, as Figler's schema is the + largest consumer: set `KIWI_CODEC_PATH` in Figler, run `mix rustq.gen`, and + check that its crate compiles. Restore Figler's generated files afterwards. diff --git a/guides/native-decoders.md b/guides/native-decoders.md new file mode 100644 index 0000000..ab9b16c --- /dev/null +++ b/guides/native-decoders.md @@ -0,0 +1,135 @@ +# Native decoders + +`KiwiCodec.RustlerGenerator.source/2` writes Rust that decodes Kiwi data for a +schema inside a Rustler NIF. It is for data too large or too hot to decode in +Elixir. What it decodes matches `KiwiCodec.decode/2` and +`KiwiCodec.SchemaInterpreter`, and KiwiCodec's tests check that for every +feature. + +## Setup + +The generator needs RustQ, an optional dependency of KiwiCodec. Add it where you +generate, usually only in development and test: + +```elixir +{:rustq, "~> 1.0.0-rc.14", only: [:dev, :test], runtime: false} +``` + +Generate from `rustq.exs`. `mix rustq.gen` writes the file, and +`mix rustq.gen --check`, in CI, fails when it is stale: + +```elixir +use RustQ.Config + +schema = KiwiCodec.parse_schema!(File.read!("priv/schema.kiwi")) + +generate :decoders, "native/my_nif/src/generated.rs" do + content KiwiCodec.RustlerGenerator.source(schema, + entrypoints: ["Node"], + module_prefix: "MyApp.Schema" + ) +end +``` + +Include the generated file in the crate beside a `Decoder`; see +[The Decoder](#the-decoder). + +## Features + +`:features` picks what to generate, `[:full]` by default. Each feature generates +one function per definition, taking the `Decoder` positioned at a value: + +| Feature | Generates | Returns | +| --- | --- | --- | +| `:full` | `decode__from_decoder(env, &mut decoder)` | the struct `KiwiCodec.decode/2` returns | +| `:sparse` | `decode_sparse__from_decoder(env, &mut decoder)` | a map of the fields present, with `__kiwi_module__` | +| `:skip` | `skip__from_decoder(&mut decoder)` | nothing; consumes the value | +| `:records` | `kiwi_decode_record(&mut decoder, definition)` | a `KiwiRecord` Rust code can read and change | + +`` is the definition's name in snake case. Enums have no skip function. + +Full decoders build the structs of the modules `:module_prefix` names, so those +modules should exist in Elixir, generated with `mix kiwi.gen` or +`KiwiCodec.compile_schema!/2`. Sparse maps key fields by atoms and give enum +values as atoms: + +```elixir +%{__kiwi_module__: MyApp.Schema.Node, name: "frame", kind: :ellipse} +``` + +Sparse and skip decoders together let native code read some fields and skip the +rest without building terms for them. `sparse_messages: :descriptor` generates +smaller sparse message decoders that share their field tables with skip +decoders; measure before choosing it for hot schemas. + +Records are decoded values Rust code can work with before deciding what to +return. `kiwi_encode_record(env, &record)` encodes one as the sparse map the +sparse decoder would return. The generated constants name each definition, +field, and enum value; see `KiwiCodec.RustlerGenerator.Records`. + +## Entrypoints + +`:entrypoints` adds NIFs that decode a whole binary with the full decoder and +fail on trailing bytes: + +```rust +#[rustler::nif(schedule = "DirtyCpu")] +fn decode_node<'a>(env: Env<'a>, bytes: Binary<'a>) -> NifResult> +``` + +It takes definition names, which infer `decode_` NIFs, `{nif_name, +definition_name}` pairs, `:all`, a NIF stub module whose one-argument `decode_*` +functions name definitions, or `{:nif_stubs, module}` for a module whose +`stubs/0` lists them. Only the definitions entrypoints need, and their +dependencies, are generated unless `:definitions` names others. + +For the other features, write the NIF yourself. With `features: [:sparse]`, this +one returns the sparse map of a `Node`: + +```rust +#[rustler::nif] +fn decode_sparse_node<'a>(env: Env<'a>, bytes: Binary<'a>) -> NifResult> { + let mut decoder = Decoder::new(bytes.as_slice()); + let term = decode_sparse_node_from_decoder(env, &mut decoder)?; + decoder.finish()?; + Ok(term) +} +``` + +## Unknown fields + +Message decoders reject a field the compiled schema does not define, as +`KiwiCodec.decode/2` does without a schema. Data written with a newer schema +needs that schema to be skipped. With `unknown_fields: :runtime_schema`, the +generated code includes a `KiwiRuntimeSchema` read from a schema in Kiwi's +binary form, and the `Decoder` hands it to the generated decoders: + +```rust +let schema = KiwiRuntimeSchema::decode(schema_bytes.as_slice())?; +let mut decoder = Decoder::with_schema(bytes.as_slice(), Some(&schema)); +``` + +See `KiwiCodec.RustlerGenerator.UnknownFields`. + +## The Decoder + +The generated code reads with a `Decoder<'a>` the crate provides, imported from +`crate::runtime::Decoder` unless `:decoder` names another path. +`KiwiCodec.RustlerGenerator.Decoder` lists the methods each feature uses and +ships `decoder.rs`, a complete one to copy into the crate. + +RustQ reads the `Decoder`'s method signatures to know which reads return a +`NifResult` and need `?`. A `Decoder` whose signatures differ from the reference +one is passed with `:decoder_sources`: + +```elixir +KiwiCodec.RustlerGenerator.source(schema, + entrypoints: ["Node"], + module_prefix: "MyApp.Schema", + decoder_sources: ["native/my_nif/src/runtime.rs"] +) +``` + +## Rustler versions + +The generated code builds with rustler 0.37 to 0.39. diff --git a/integration/generator_consumer/mix.exs b/integration/generator_consumer/mix.exs new file mode 100644 index 0000000..70988ba --- /dev/null +++ b/integration/generator_consumer/mix.exs @@ -0,0 +1,16 @@ +defmodule KiwiCodecGeneratorConsumer.MixProject do + use Mix.Project + + def project do + [app: :kiwi_codec_generator_consumer, version: "0.1.0", elixir: "~> 1.19", deps: deps()] + end + + def application, do: [extra_applications: [:logger]] + + defp deps do + [ + {:kiwi_codec, path: System.fetch_env!("KIWI_CODEC_PACKAGE_PATH")}, + {:rustq, "~> 1.0.0-rc.14", only: [:dev, :test], runtime: false} + ] + end +end diff --git a/integration/generator_consumer/rustq.exs b/integration/generator_consumer/rustq.exs new file mode 100644 index 0000000..cc9284e --- /dev/null +++ b/integration/generator_consumer/rustq.exs @@ -0,0 +1,11 @@ +use RustQ.Config + +schema = KiwiCodec.parse_schema!("message Node { uint id = 1; string name = 2; }") + +generate :decoders, "native/src/generated.rs" do + content KiwiCodec.RustlerGenerator.source(schema, + entrypoints: ["Node"], + features: [:full, :sparse, :skip], + module_prefix: "KiwiCodecGeneratorConsumer.Schema" + ) +end diff --git a/integration/generator_consumer/test/consumer_test.exs b/integration/generator_consumer/test/consumer_test.exs new file mode 100644 index 0000000..3521675 --- /dev/null +++ b/integration/generator_consumer/test/consumer_test.exs @@ -0,0 +1,13 @@ +defmodule KiwiCodecGeneratorConsumerTest do + use ExUnit.Case, async: true + + test "mix rustq.gen wrote Rustler decoders" do + functions = + "native/src/generated.rs" + |> RustQ.Syn.parse_file!() + |> RustQ.Syn.functions() + |> Enum.map(& &1.name) + + assert "decode_node" in functions + end +end diff --git a/integration/generator_consumer/test/test_helper.exs b/integration/generator_consumer/test/test_helper.exs new file mode 100644 index 0000000..869559e --- /dev/null +++ b/integration/generator_consumer/test/test_helper.exs @@ -0,0 +1 @@ +ExUnit.start() diff --git a/lib/kiwi_codec.ex b/lib/kiwi_codec.ex index ce81880..93c9c4e 100644 --- a/lib/kiwi_codec.ex +++ b/lib/kiwi_codec.ex @@ -1,10 +1,15 @@ defmodule KiwiCodec do @moduledoc """ - Pure Elixir codec for Kiwi schema binary messages. + Encodes and decodes Kiwi messages as Elixir structs. - Kiwi is a compact schema-driven format with message, struct, enum, and primitive - encodings that differ from Protocol Buffers. This package is intentionally - generic; product-specific schemas can live in companion packages. + `use KiwiCodec, kind: :message`, `:struct`, or `:enum` defines a module for a + schema definition with the `KiwiCodec.DSL` macros, and `mix kiwi.gen` or + `compile_schema!/2` generates such modules from a schema. `encode/1` and + `decode/3` convert their structs to and from Kiwi binaries. + + Data written with a newer schema decodes when `decode/3` is given that schema: + fields the modules do not define are skipped. `KiwiCodec.SchemaInterpreter` + works without modules, for schemas loaded at runtime. """ defmacro __using__(opts) do diff --git a/mix.exs b/mix.exs index 1f96928..4890d9b 100644 --- a/mix.exs +++ b/mix.exs @@ -29,9 +29,10 @@ defmodule KiwiCodec.MixProject do [extra_applications: [:logger]] end + # The Rustler generator under codegen/ compiles only where RustQ, an optional dependency, + # is available, such as a consumer's development and test builds. defp elixirc_paths(:test), do: ["lib", "codegen", "test/support"] - defp elixirc_paths(:dev), do: ["lib", "codegen"] - defp elixirc_paths(_env), do: ["lib"] + defp elixirc_paths(_env), do: ["lib", "codegen"] defp aliases do [ @@ -61,7 +62,7 @@ defmodule KiwiCodec.MixProject do end defp rustq_dep do - {:rustq, "~> 1.0.0-rc.14", only: [:dev, :test], runtime: false} + {:rustq, "~> 1.0.0-rc.14", optional: true, runtime: false} end defp package do @@ -77,8 +78,9 @@ defmodule KiwiCodec.MixProject do main: "readme", extras: [ "README.md", - "CHANGELOG.md", - "guides/rustler-generator-architecture.md" + "guides/native-decoders.md", + "guides/internals.md", + "CHANGELOG.md" ], groups_for_modules: docs_groups_for_modules(), source_ref: "v#{@version}", @@ -90,15 +92,13 @@ defmodule KiwiCodec.MixProject do [ Codec: [ KiwiCodec, + KiwiCodec.DSL, + KiwiCodec.TransformModule, KiwiCodec.Container, - KiwiCodec.Decoder, - KiwiCodec.Encoder, KiwiCodec.DecodeError, - KiwiCodec.EncodeError, - KiwiCodec.TransformModule, - KiwiCodec.Inspect + KiwiCodec.EncodeError ], - Schema: [ + Schemas: [ KiwiCodec.Schema, KiwiCodec.Schema.Enum, KiwiCodec.Schema.EnumVariant, @@ -106,10 +106,10 @@ defmodule KiwiCodec.MixProject do KiwiCodec.Schema.Message, KiwiCodec.Schema.Struct, KiwiCodec.Schema.Binary, - KiwiCodec.Metadata, - KiwiCodec.Metadata.Field, KiwiCodec.SchemaInterpreter, KiwiCodec.SchemaInterpreter.Prepared, + KiwiCodec.Metadata, + KiwiCodec.Metadata.Field, KiwiCodec.PrimitiveType ], "Wire encoding": [ @@ -118,21 +118,30 @@ defmodule KiwiCodec.MixProject do KiwiCodec.Wire.Varint ], "Elixir generation": [ - KiwiCodec.ModuleGenerator, KiwiCodec.ModuleCompiler, - KiwiCodec.FileGenerator, - KiwiCodec.DSL, - KiwiCodec.GeneratedModule.Metadata, - KiwiCodec.GeneratedModule.Shape, - KiwiCodec.GeneratedModule.TypeSpec + KiwiCodec.ModuleGenerator, + KiwiCodec.FileGenerator ], "Rustler generation": [ KiwiCodec.RustlerGenerator, + KiwiCodec.RustlerGenerator.Decoder, KiwiCodec.RustlerGenerator.Records, - KiwiCodec.RustlerGenerator.UnknownFields, - KiwiCodec.RustlerGenerator.Skip, - KiwiCodec.RustlerGenerator.Skip.Kind, - KiwiCodec.RustlerGenerator.Sparse + KiwiCodec.RustlerGenerator.UnknownFields + ], + "Codec internals": [ + KiwiCodec.Decoder, + KiwiCodec.Encoder, + KiwiCodec.Inspect, + KiwiCodec.GeneratedModule.Metadata, + KiwiCodec.GeneratedModule.Shape, + KiwiCodec.GeneratedModule.TypeSpec + ], + "Schema internals": [ + KiwiCodec.Schema.Binary.TypeIndex, + KiwiCodec.Schema.Parser, + KiwiCodec.Schema.Token, + KiwiCodec.Schema.Tokenizer, + KiwiCodec.Schema.Validator ], "Rustler generation internals": [ KiwiCodec.RustlerGenerator.AtomCache, @@ -143,18 +152,14 @@ defmodule KiwiCodec.MixProject do KiwiCodec.RustlerGenerator.Name, KiwiCodec.RustlerGenerator.RustExpr, KiwiCodec.RustlerGenerator.Selection, + KiwiCodec.RustlerGenerator.Skip, KiwiCodec.RustlerGenerator.Skip.Descriptor, KiwiCodec.RustlerGenerator.Skip.Helpers, + KiwiCodec.RustlerGenerator.Skip.Kind, KiwiCodec.RustlerGenerator.Skip.ValueHelpers, + KiwiCodec.RustlerGenerator.Sparse, KiwiCodec.RustlerGenerator.Sparse.Helpers, KiwiCodec.RustlerGenerator.Splice - ], - "Schema internals": [ - KiwiCodec.Schema.Binary.TypeIndex, - KiwiCodec.Schema.Parser, - KiwiCodec.Schema.Token, - KiwiCodec.Schema.Tokenizer, - KiwiCodec.Schema.Validator ] ] end diff --git a/test/package/consumer_test.exs b/test/package/consumer_test.exs index 16c5787..200409f 100644 --- a/test/package/consumer_test.exs +++ b/test/package/consumer_test.exs @@ -3,21 +3,24 @@ defmodule KiwiCodec.PackageConsumerTest do @moduletag timeout: 300_000 - test "unpacked package compiles as a runtime-only consumer" do + setup_all do root = File.cwd!() workspace = Path.join(System.tmp_dir!(), "kiwi-codec-package-#{System.unique_integer([:positive])}") package = Path.join(workspace, "package") - consumer = Path.join(workspace, "runtime_consumer") File.mkdir_p!(workspace) on_exit(fn -> File.rm_rf!(workspace) end) run!(root, "mix", ["hex.build", "--unpack", "--output", package]) - File.cp_r!(Path.join([root, "integration", "runtime_consumer"]), consumer) - env = [{"KIWI_CODEC_PACKAGE_PATH", package}, {"MIX_ENV", "prod"}] + %{root: root, workspace: workspace, package: package} + end + + test "unpacked package compiles as a runtime-only consumer", context do + consumer = copy_fixture!(context, "runtime_consumer") + env = [{"KIWI_CODEC_PACKAGE_PATH", context.package}, {"MIX_ENV", "prod"}] run!(consumer, "mix", ["deps.get"], env) lock = File.read!(Path.join(consumer, "mix.lock")) @@ -27,6 +30,23 @@ defmodule KiwiCodec.PackageConsumerTest do run!(consumer, "mix", ["test", "--warnings-as-errors"], env) end + test "unpacked package generates Rust where the consumer depends on RustQ", context do + consumer = copy_fixture!(context, "generator_consumer") + env = [{"KIWI_CODEC_PACKAGE_PATH", context.package}] + run!(consumer, "mix", ["deps.get"], env) + run!(consumer, "mix", ["rustq.gen"], env) + run!(consumer, "mix", ["test", "--warnings-as-errors"], env) + + # RustQ is a development dependency there, so production builds leave the generator out. + run!(consumer, "mix", ["compile", "--warnings-as-errors"], [{"MIX_ENV", "prod"} | env]) + end + + defp copy_fixture!(context, name) do + consumer = Path.join(context.workspace, name) + File.cp_r!(Path.join([context.root, "integration", name]), consumer) + consumer + end + defp run!(directory, command, args, env \\ []) do {output, status} = System.cmd(command, args, diff --git a/test/package/package_test.exs b/test/package/package_test.exs index ed897d4..9fa068f 100644 --- a/test/package/package_test.exs +++ b/test/package/package_test.exs @@ -9,13 +9,13 @@ defmodule KiwiCodec.PackageTest do assert "guides" in files end - test "limits RustQ to development and test compilation" do + test "makes RustQ optional, for consumers that generate Rust" do {_app, _requirement, opts} = Mix.Project.config() |> Keyword.fetch!(:deps) |> Enum.find(&(elem(&1, 0) == :rustq)) - assert Keyword.fetch!(opts, :only) == [:dev, :test] + assert Keyword.fetch!(opts, :optional) refute Keyword.get(opts, :runtime, true) end end