feat: Pulse protocol with quantization toolchain#449
Open
kuruk-mm wants to merge 6 commits into
Open
Conversation
* Add Quantization as a protobuf plugin * pulse_comms.proto Signed-off-by: Mikhail Agapov <mikhail.agapov@decentraland.org> * add server message & handshake response * add player state full/delta * add player joined message * add player joined into server message * add player state into client message * Update protobufjs to 7.5.4 to resolve the problem with namespaces * proto/google/ is no longer created or published, so both buf (CI) and protoc (consumers like unity-explorer) resolve descriptor.proto from their own built-in includes — which all have the correct csharp_namespace Signed-off-by: Mikhail Agapov <mikhail.agapov@decentraland.org> * fix: exclude proto/google from npm package to fix C# codegen namespace Revert protobufjs to 7.2.4 (7.5.4 breaks buf and proto-compatibility-tool with newer reserved syntax) and restore the make install copy step for local tooling. Change "files" from "proto" to "proto/decentraland" so the stripped google/protobuf/descriptor.proto (missing csharp_namespace option) is no longer published. Consumers' protoc resolves descriptor.proto from its own built-in includes which have the correct option csharp_namespace = "Google.Protobuf.Reflection". Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Adjust states encoding * Add quantization example for PlayerStateDelta Signed-off-by: Mikhail Agapov <mikhail.agapov@decentraland.org> * Copy protoc-gen-bitwise to the output package * Make Quantize.cs compitable with Unity Signed-off-by: Mikhail Agapov <mikhail.agapov@decentraland.org> * Isolate PlayerState as a reusable component Signed-off-by: Mikhail Agapov <mikhail.agapov@decentraland.org> * Add requried quntizationfor delta Signed-off-by: Mikhail Agapov <mikhail.agapov@decentraland.org> * Add RESYNC_REQUEST Signed-off-by: Mikhail Agapov <mikhail.agapov@decentraland.org> * ProfileVersionAnnouncement - parcel_index change uint32 -> int32 Signed-off-by: Mikhail Agapov <mikhail.agapov@decentraland.org> * add emote messages * add duration on EmoteStart for one shot emotes * Fix MovementBlend Range Signed-off-by: Mikhail Agapov <mikhail.agapov@decentraland.org> * add teleport messages * Add "jump_count" Signed-off-by: Mikhail Agapov <mikhail.agapov@decentraland.org> * add teleport ClientMessage & ServerMessage * Add BaselineSeq to delta Signed-off-by: Mikhail Agapov <mikhail.agapov@decentraland.org> * add parcel index & player state in teleport * add head yaw and head pitch to state flags * Add `sequence` to `EmoteStarted` Signed-off-by: Mikhail Agapov <mikhail.agapov@decentraland.org> * add player state into EmoteStart * Add seq and PlayerState to `EmoteStopped` Signed-off-by: Mikhail Agapov <mikhail.agapov@decentraland.org> * Add "realm" field to "TeleportRequest" Signed-off-by: Mikhail Agapov <mikhail.agapov@decentraland.org> * Split pulse_comms into separate files Signed-off-by: Mikhail Agapov <mikhail.agapov@decentraland.org> * Add Initial State to HandshakeRequest - It's needed to properly authorize reconnection in the middle of the session Signed-off-by: Mikhail Agapov <mikhail.agapov@decentraland.org> * Add support for "pointing at" Signed-off-by: Mikhail Agapov <mikhail.agapov@decentraland.org> * Change "Point At" to the absolute value Signed-off-by: Mikhail Agapov <mikhail.agapov@decentraland.org> * Add "realm" to the initial state Signed-off-by: Mikhail Agapov <mikhail.agapov@decentraland.org> * Change head_pitch max from 180 to 360 Signed-off-by: Mikhail Agapov <mikhail.agapov@decentraland.org> * emote mask --------- Signed-off-by: Mikhail Agapov <mikhail.agapov@decentraland.org> Co-authored-by: Nicolas Lorusso <nalorusso@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Rewrite the protoc-gen-bitwise plugin from Python to a dependency-free
Node implementation, so protocol generation needs only `node` on PATH —
no Python interpreter and no `protobuf` pip package.
- plugin.js stdin/stdout protoc plugin entry; advertises
FEATURE_PROTO3_OPTIONAL
- wire.js self-contained protobuf wire codec for
CodeGeneratorRequest/Response (no runtime deps)
- options.js parser for the custom quantized / bit_packed options
- generator_csharp.js C# partial-class generator with faithful %g float
formatting; output is byte-for-byte identical to the
Python plugin
- test/ + `npm run gen:test` golden-fixture parity test
- remove plugin.py, generator_csharp.py, options_pb2.py
- package.json ship only the .js + C# runtime (exclude test fixtures)
- README.md, CLAUDE.md Python -> Node prerequisites and invocation
Verified byte-identical regeneration three ways: protoc end-to-end, a
Pulse clean-room rebuild (all 16 generated files), and unity-explorer
build-protocol. Consumers invoke the plugin via a tiny `node plugin.js`
wrapper (.cmd on Windows, shell script elsewhere).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#431) * feat: power-law quantized float option; apply to Pulse velocity deltas Add a new `(decentraland.common.quantized_power)` field option for uint32 fields: an `(bits-1)`-bit linear unorm magnitude `u` (high bits) plus a sign (LSB), decoded as `sign * max * u^pow`. Unlike the linear `quantized` option, zero is exactly representable (`u=0` -> `0`) and `pow > 1` concentrates resolution near zero. Putting the sign in the LSB (`(magnitude << 1) | sign`) makes the varint cost track magnitude rather than direction — a small `|value|` of either sign stays in one varint byte, and a stopped axis canonicalizes to 0 (omitted by proto3). `PlayerStateDeltaTier0.velocity_x/y/z` switch from `quantized{min:-50,max:50,bits:8}` to `quantized_power{max:50,pow:2,bits:8}`. The linear quantizer could not represent an exact 0 over [-50,50] in 8 bits (±0.196 residual), which drove a steady drift on stopped foreign avatars; the power curve fixes that and spends its bits on the low speeds where avatars actually move. Implemented end to end against the Node protoc-gen-bitwise plugin: options.proto message + extension (50003), the options.js parser, generator_csharp.js dispatch, the C# runtime (Quantize.EncodePower/DecodePower), golden-fixture coverage (test helper + regenerated PulseServer/QuantizationExample goldens), plus the README, CLAUDE.md and quantization_example.proto docs. Breaking wire change for velocity_x/y/z: server (pulse), Unity, godot and bevy must regenerate and deploy in lockstep. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: robtfm <50659922+robtfm@users.noreply.github.com> * Quantize PlayerState identically to PlayerStateDeltaTier0 Signed-off-by: Mikhail Agapov <mikhail.agapov@decentraland.org> * Add "step" const to the quantization gen Signed-off-by: Mikhail Agapov <mikhail.agapov@decentraland.org> --------- Signed-off-by: robtfm <50659922+robtfm@users.noreply.github.com> Signed-off-by: Mikhail Agapov <mikhail.agapov@decentraland.org> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Mikhail Agapov <mikhail.agapov@decentraland.org>
* Quantize TeleportRequest * Add Quantized Values Validation Signed-off-by: Mikhail Agapov <mikhail.agapov@decentraland.org> * Import "decentraland/common/vectors.proto" is unused Signed-off-by: Mikhail Agapov <mikhail.agapov@decentraland.org> * protoc-gen-bitwise: add range check, drop decode cache Emit AreQuantizedFieldsInRange() (per-field integer bounds check, used by the server to reject out-of-range client codes) and remove the per-field decoded- float cache — decode on get, encode on set. Regenerate goldens; add UPDATE_GOLDEN mode to the golden test so future generator changes are one command. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Signed-off-by: Mikhail Agapov <mikhail.agapov@decentraland.org> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* docs: describe protoc-gen-bitwise as it actually works; drop dead BitReader/BitWriter CLAUDE.md still documented the original bit-stream design (BitWriter/ BitReader classes, float fields, "68 bits = 9 bytes on the wire"). The plugin actually emits quantized-accessor partials (*.Bitwise.cs) over plain uint32 fields sent as standard protobuf varints, backed only by Quantize.cs; runtime BitReader.cs/BitWriter.cs were referenced by nothing and are removed. Also sync README with current generator output (QuantizedStep consts, AreQuantizedFieldsInRange, EncodePower/ DecodePower, per-proto-file output naming) and drop the incorrect "cached" accessor wording. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: normalize CRLF when comparing gen:test golden fixtures With core.autocrlf=true (and no .gitattributes) git materializes the golden .cs fixtures with CRLF on Windows while the generator always emits LF, so the strict byte comparison failed on any fresh Windows checkout. Normalize line endings when reading the goldens. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: add realm in pulse PlayerJoined message * added realm to teleport performed message --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Lorenzo Ranciaffi <lorenzo.ranciaffi@decentraland.org>
Test this pull request
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Ports the complete Pulse feature from
experimentaltomain— and only Pulse. Six commits cherry-picked with original authorship preserved:proto/decentraland/pulse/, custom field options, protoc-gen-bitwise plugin)pulse_client.proto)Included
proto/decentraland/pulse/—pulse_client.proto,pulse_server.proto,pulse_shared.protoproto/decentraland/common/options.proto+quantization_example.proto— customquantized/quantized_power/bit_packedfield options with a fully worked referenceprotoc-gen-bitwise/— Node protoc plugin, C#Quantizeruntime, golden-fixture testsREADME.md/CLAUDE.md— Pulse & quantization docspackage.json—gen:testscript, published-files listDeliberately excluded (not Pulse)
@protobuf-ts/protoc^2.8.1 → ^2.11.0bump — belongs to Voice chat (feat: Voice chat protocol #265)experimentalhistory (verified identical viagit patch-id)fix/realm-data-in-pulse-player-joined— not yet merged intoexperimentalWhy
Pulse protos and the quantization toolchain are stable in
experimental; this makes them available frommainwithout pulling in unrelated experimental features.Test plan
npm run gen:test— golden-fixture parity for the bitwise generator passesnpm test— proto-compatibility-tool against publishedmainpasses (Pulse protos detected as new, no breaking changes)experimental(git diff feat/pulse experimental -- <pulse paths>empty except the intentionally excluded protoc bump)