Skip to content

Add BE key decoder for sysio KV support#8

Merged
heifner merged 3 commits into
masterfrom
feature/be-key-decoder
Apr 20, 2026
Merged

Add BE key decoder for sysio KV support#8
heifner merged 3 commits into
masterfrom
feature/be-key-decoder

Conversation

@heifner

@heifner heifner commented Mar 24, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds abieos_be_key_hex_to_json() — a C API function that decodes the big-endian-encoded key field of a SHiP contract_row_kv_v0 delta to JSON, using the key_names/key_types arrays from the contract's ABI table_def.

This is a standalone utility for any C/C++ SHiP delta consumer that needs to interpret KV table keys without re-implementing CDT's be_key_stream encoding by hand. (hyperion-history-api doesn't use this — it has its own decoder in TypeScript at workers/deserializer.ts:decodeBEKey. The function is here for future or third-party consumers.)

Encoding spec

Mirrors sysio::kv::be_key_stream in wire-cdt:libraries/sysiolib/contracts/sysio/kv_utils.hpp — the encoder is designed so a lexicographic memcmp on the encoded bytes matches the natural ordering of the underlying values, which is why signed integers and floats get sign-flipped:

Type Encoding
uint8/16/32/64 big-endian raw
int8/16/32/64 big-endian XOR 1<<(N-1)
uint128/int128 big-endian (high uint64 then low uint64); int128 XOR 1<<127
name big-endian raw uint64
bool 1 byte
float (32-bit) big-endian; positive XOR 1<<31, negative bitwise NOT
double (64-bit) big-endian; positive XOR 1<<63, negative bitwise NOT
string / bytes NUL-escaped: each 0x000x00 0x01, terminated by 0x00 0x00
checksum160 20 raw bytes (no transform)
checksum256 32 raw bytes
checksum512 64 raw bytes

checksum* types reach be_key_stream via a generic template<DataStream, size_t> operator<<(DataStream&, const fixed_bytes<Size>&) in wire-cdt:libraries/sysiolib/core/sysio/fixed_bytes.hpp that writes the raw bytes via be_key_stream::write(const char*, size_t) — no sign flip, no length prefix.

C API

// Decode big-endian encoded KV key bytes to JSON.
// key_names: JSON array of field names, e.g. '["region","id"]'
// key_types: JSON array of type names, e.g. '["string","uint64"]'
// hex: hex-encoded key bytes
// Returns JSON object, e.g. '{"region":"us-east","id":"42"}'.
// Returns null on error; use abieos_get_error to retrieve error.
const char* abieos_be_key_hex_to_json(abieos_context* context,
                                      const char* key_names,
                                      const char* key_types,
                                      const char* hex);

JSON output conventions

Output matches the rest of abieos:

  • uint64/int64 → JSON string ("100") so JS callers don't lose precision via Number
  • uint128/int1280x-prefixed lowercase hex string
  • checksum* → lowercase hex string (no 0x prefix)
  • bytes → lowercase hex string (no 0x prefix)
  • Strings get standard JSON escapes for ", \, \n, \r, \t, \b, \f, plus \uXXXX for any other control byte

Robustness improvements over the initial draft

  • Replaced the hand-rolled parse_json_string_array with sysio::from_json<std::vector<std::string>>, the same JSON parser the rest of abieos already uses for ABI loading. No more brittle character-by-character state machine.
  • Trailing-byte detection: if any bytes remain in the input after all declared fields are decoded, the decoder errors out ("be_key: N trailing byte(s) after decoding declared fields"). Silently dropping bytes hides caller bugs where key_types doesn't match the encoded key.
  • Truncated NUL-escape detection: explicit error messages for string (truncated escape), string (no terminator), and string (bad escape) instead of producing garbage.
  • Full string escaping in JSON output, including \uXXXX for control bytes — necessary for any string field that legitimately contains binary data.

Test coverage

15 new test cases in src/test.cpp/check_types covering every supported type plus error paths:

  • All integer widths and sign variants (uint8/16/32/64, int8/16/32/64)
  • uint128/int128 sign-flip round-trip
  • name (round-tripped through abieos_string_to_name)
  • bool (true/false)
  • checksum160, checksum256
  • string (simple, embedded NUL via \u0000 escape, empty)
  • Composite key (scope:name + sym_code:uint64 — sysio.token's accounts table layout)
  • Error: trailing bytes, truncated input, unknown type, mismatched key_names/key_types lengths, empty arrays

All four existing abieos C++ tests still pass.

heifner added 2 commits March 24, 2026 15:07
New C API function that decodes big-endian encoded KV key bytes to JSON
using key_names/key_types arrays from the contract ABI.

Supports: uint8-64, int8-64, uint128, name, float64/double (sign-flipped),
string (null-terminated), bool. Matches the encoding produced by CDT's
kv::map be_key_stream.

Usage:
  abieos_be_key_hex_to_json(ctx,
    '["region","id"]',     // key_names JSON array
    '["string","uint64"]', // key_types JSON array
    "75732d6561737400...")  // hex-encoded BE key bytes
  => '{"region":"us-east","id":"42"}'
Wire's KV contracts use a number of secondary index key types that the
original PR #8 decoder didn't handle: int128 (sign-flipped, 16 bytes),
float32 (sign-flipped, 4 bytes), and the fixed_bytes-backed checksum
types (checksum160, checksum256, checksum512 — raw bytes, no
transform). Without those, sysio.roa, sysio.authex, sysio.system, and
sysio.token would all hit "unsupported type" errors when a SHiP
client tried to decode their secondary index keys.

This commit brings the decoder to full parity with CDT's
sysio::kv::be_key_stream encoder in
libraries/sysiolib/contracts/sysio/kv_utils.hpp:

  uint8/16/32/64    big-endian raw
  int8/16/32/64     big-endian XOR 1<<(N-1)
  uint128/int128    big-endian (high uint64 then low uint64);
                    int128 XOR 1<<127
  name              big-endian raw uint64
  bool              1 byte
  float (32-bit)    big-endian, positive XOR 1<<31, negative bitwise NOT
  double (64-bit)   big-endian, positive XOR 1<<63, negative bitwise NOT
  string / bytes    NUL-escaped: each 0x00 → 0x00 0x01,
                    terminated by 0x00 0x00
  checksum160       20 raw bytes
  checksum256       32 raw bytes
  checksum512       64 raw bytes

The fixed_bytes / checksum types reach be_key_stream via a generic
template<DataStream, size_t> operator<<(DataStream&, fixed_bytes<Size>)
in libraries/sysiolib/core/sysio/fixed_bytes.hpp that calls
be_key_stream::write(const char*, size_t) directly — no sign flip, no
length prefix, just raw bytes.

Other improvements:

- Replaced the brittle hand-rolled parse_json_string_array with
  sysio::from_json<std::vector<std::string>>, which is the same JSON
  parser the rest of abieos already uses for JSON ABI loading.
- JSON output now properly escapes control characters in strings
  (\b, \f, and \uXXXX for other <0x20 bytes), not just the obvious
  \n/\r/\t/\"/\\.
- decode_be_field detects truncated NUL-escape sequences explicitly
  ("string (truncated escape)" / "string (no terminator)" /
  "string (bad escape)") instead of silently producing garbage.
- The wrapping function rejects trailing bytes after all declared
  fields are decoded — that's almost always a sign that the caller's
  key_types didn't match the encoded key, and silently dropping bytes
  hides bugs.
- uint64 / int64 are emitted as JSON strings (e.g. "100"), matching
  the rest of abieos which represents 64-bit integers as strings to
  avoid JS Number precision loss.
- uint128 / int128 are emitted as 0x-prefixed lowercase hex strings,
  also matching abieos convention.

15 new test cases in test.cpp/check_types covering every supported
type plus error paths (trailing bytes, truncated input, unknown type,
mismatched key_names/key_types lengths, empty arrays). All four
existing C++ tests still pass.
@heifner heifner changed the title Add BE key decoder for KV map support Add BE key decoder for sysio KV support Apr 10, 2026
@heifner
heifner requested a review from dtaghavi April 16, 2026 19:20

@dtaghavi dtaghavi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: Add BE key decoder for sysio KV support

Overview

This PR adds abieos_be_key_hex_to_json() — a well-structured C API function that decodes CDT's be_key_stream-encoded KV table keys to JSON. The implementation reuses the existing sysio::from_json parser for input handling and includes solid error detection for trailing bytes, truncated input, and bad NUL escapes. Test coverage is broad across integer/name/bool/checksum/string types.


Issues

1. Header doc comment contradicts the implementation (abieos.h)

The example in the header says:
```c
// Returns JSON object, e.g. '{"region":"us-east","id":42}'.
```
But uint64 values are emitted as JSON strings, so the correct example is:
```c
// Returns JSON object, e.g. '{"region":"us-east","id":"42"}'.
```
The PR description gets this right ("uint64/int64 → JSON string"); the header comment needs to match.


2. No test coverage for float32/float64

The sign-flip inversion is the most subtle logic in the decoder:
```cpp
// float32
if (bits >> 31) bits ^= (uint32_t(1) << 31);
else bits = ~bits;
```
There are zero test cases for float32 or float64. This is a gap — a bug here is silent since std::to_string(v) won't throw on wrong bit patterns. Suggested cases: a positive value, a negative value, zero, ±infinity, and the boundary where the sign-flip changes branch (+0.0 vs -0.0).


3. Truncated-read errors surface internal read width, not the field type

read_be64 always passes "uint64" to be_key_overrun, so decoding a name or int64 field that's truncated produces:

be_key: unexpected end of data reading uint64

…instead of reading name. Same for checksum* types via read_raw_hex. The error is accurate but confusing for API callers. Consider threading the field type through to the error rather than the internal read width.


4. Minor: hex_chars is defined twice

"0123456789abcdef" appears as a static const char array in both read_raw_hex and the bytes branch of decode_be_field. A single file-scope constant would remove the duplication.


5. Minor: misleading comment in the composite-key test

The composite-key test uses 1397703940 with the comment "SYS" symbol code. 1397703940 = 0x534E5344, which doesn't correspond to "SYS" in ASCII. The round-trip logic is still correct (it exercises a uint64 in a composite key), but the comment is misleading and worth fixing.


Strengths

  • Replacing the hand-rolled JSON-array parser with sysio::from_json<std::vector<std::string>> is a clear win for correctness and maintainability.
  • Trailing-byte detection prevents silently wrong output when key_types doesn't match the actual encoded key — this is the kind of caller bug that would otherwise be very hard to track down.
  • std::memcpy for float/double bit-casting is correct and strict-aliasing safe.
  • The anonymous namespace properly scopes all helpers.
  • NUL-escape error messages (truncated escape, no terminator, bad escape) are precise and actionable.
  • The PR description's encoding table is excellent reference material; it should stay in a comment or doc somewhere even after merge.

Verdict

Request changes on: fix the header doc comment (#1) and add float32/float64 tests (#2) — both are straightforward. Issues #3–5 are non-blocking but worth a follow-up.

Address review comments on the BE key decoder PR:

1. Header doc example (abieos.h): the example previously showed
   `"id":42`; uint64 is actually emitted as a JSON string, so fix
   to `"id":"42"` and note the precision-safety rationale.

2. Thread the caller's declared field type through truncation
   errors. Previously read_be* always reported the internal width
   ("uint64"), so truncating a name/int64/float64/checksum field
   produced a confusing "reading uint64" message. Each reader now
   takes a type_label and forwards it to be_key_overrun, so errors
   name the user-declared type.

3. Dedupe the lowercase hex alphabet constant. Hoisted to a single
   namespace-scoped `be_key_hex_chars` used by both read_raw_hex
   and the bytes-field branch.

4. Fix misleading composite-key comment: 1397703940 is not the SYS
   symbol code. Replaced with the actual packed ASCII value
   0x535953 = 5462355, computed inline so the test documents the
   encoding.

5. Add float32 and float64 test coverage -- previously absent.
   Covers positive, negative, +0.0 vs -0.0 (which exercise both
   branches of the sign-flip inversion), and an aliased "float" /
   "double" type name.

6. Add truncation-error tests that assert the error message names
   the caller's field type (name, int64, float64, checksum256).
@heifner
heifner requested a review from dtaghavi April 16, 2026 21:08

@dtaghavi dtaghavi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All five issues from the previous review have been addressed — thanks for the thorough follow-through.

What changed:

  • Header doc comment — fixed to "id":"42" with an explanatory note about JS precision. Clear and accurate.
  • float32/float64 tests — comprehensive: +0.0, -0.0 (branch boundary), ±1.5, ±2.5, plus alias tests for float/double. Verified the encoded bytes independently — all decode to the expected values.
  • Truncated-read error messagestype_label is now threaded through all read_be* calls. The new error tests explicitly assert "reading name", "reading int64", "reading float64", "reading checksum256" — exactly the right coverage.
  • Duplicate hex_chars — consolidated into a single constexpr char be_key_hex_chars[] at namespace scope. Clean.
  • sym_code comment — now correctly derives 5462355 from 'S'|('Y'<<8)|('S'<<16) with an inline explanation of the ASCII packing. No magic numbers.

Approved.

@heifner
heifner merged commit d1e45b3 into master Apr 20, 2026
13 of 14 checks passed
@heifner
heifner deleted the feature/be-key-decoder branch April 20, 2026 19:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants