Add BE key decoder for sysio KV support#8
Conversation
New C API function that decodes big-endian encoded KV key bytes to JSON
using key_names/key_types arrays from the contract ABI.
Supports: uint8-64, int8-64, uint128, name, float64/double (sign-flipped),
string (null-terminated), bool. Matches the encoding produced by CDT's
kv::map be_key_stream.
Usage:
abieos_be_key_hex_to_json(ctx,
'["region","id"]', // key_names JSON array
'["string","uint64"]', // key_types JSON array
"75732d6561737400...") // hex-encoded BE key bytes
=> '{"region":"us-east","id":"42"}'
Wire's KV contracts use a number of secondary index key types that the original PR #8 decoder didn't handle: int128 (sign-flipped, 16 bytes), float32 (sign-flipped, 4 bytes), and the fixed_bytes-backed checksum types (checksum160, checksum256, checksum512 — raw bytes, no transform). Without those, sysio.roa, sysio.authex, sysio.system, and sysio.token would all hit "unsupported type" errors when a SHiP client tried to decode their secondary index keys. This commit brings the decoder to full parity with CDT's sysio::kv::be_key_stream encoder in libraries/sysiolib/contracts/sysio/kv_utils.hpp: uint8/16/32/64 big-endian raw int8/16/32/64 big-endian XOR 1<<(N-1) uint128/int128 big-endian (high uint64 then low uint64); int128 XOR 1<<127 name big-endian raw uint64 bool 1 byte float (32-bit) big-endian, positive XOR 1<<31, negative bitwise NOT double (64-bit) big-endian, positive XOR 1<<63, negative bitwise NOT string / bytes NUL-escaped: each 0x00 → 0x00 0x01, terminated by 0x00 0x00 checksum160 20 raw bytes checksum256 32 raw bytes checksum512 64 raw bytes The fixed_bytes / checksum types reach be_key_stream via a generic template<DataStream, size_t> operator<<(DataStream&, fixed_bytes<Size>) in libraries/sysiolib/core/sysio/fixed_bytes.hpp that calls be_key_stream::write(const char*, size_t) directly — no sign flip, no length prefix, just raw bytes. Other improvements: - Replaced the brittle hand-rolled parse_json_string_array with sysio::from_json<std::vector<std::string>>, which is the same JSON parser the rest of abieos already uses for JSON ABI loading. - JSON output now properly escapes control characters in strings (\b, \f, and \uXXXX for other <0x20 bytes), not just the obvious \n/\r/\t/\"/\\. - decode_be_field detects truncated NUL-escape sequences explicitly ("string (truncated escape)" / "string (no terminator)" / "string (bad escape)") instead of silently producing garbage. - The wrapping function rejects trailing bytes after all declared fields are decoded — that's almost always a sign that the caller's key_types didn't match the encoded key, and silently dropping bytes hides bugs. - uint64 / int64 are emitted as JSON strings (e.g. "100"), matching the rest of abieos which represents 64-bit integers as strings to avoid JS Number precision loss. - uint128 / int128 are emitted as 0x-prefixed lowercase hex strings, also matching abieos convention. 15 new test cases in test.cpp/check_types covering every supported type plus error paths (trailing bytes, truncated input, unknown type, mismatched key_names/key_types lengths, empty arrays). All four existing C++ tests still pass.
dtaghavi
left a comment
There was a problem hiding this comment.
Review: Add BE key decoder for sysio KV support
Overview
This PR adds abieos_be_key_hex_to_json() — a well-structured C API function that decodes CDT's be_key_stream-encoded KV table keys to JSON. The implementation reuses the existing sysio::from_json parser for input handling and includes solid error detection for trailing bytes, truncated input, and bad NUL escapes. Test coverage is broad across integer/name/bool/checksum/string types.
Issues
1. Header doc comment contradicts the implementation (abieos.h)
The example in the header says:
```c
// Returns JSON object, e.g. '{"region":"us-east","id":42}'.
```
But uint64 values are emitted as JSON strings, so the correct example is:
```c
// Returns JSON object, e.g. '{"region":"us-east","id":"42"}'.
```
The PR description gets this right ("uint64/int64 → JSON string"); the header comment needs to match.
2. No test coverage for float32/float64
The sign-flip inversion is the most subtle logic in the decoder:
```cpp
// float32
if (bits >> 31) bits ^= (uint32_t(1) << 31);
else bits = ~bits;
```
There are zero test cases for float32 or float64. This is a gap — a bug here is silent since std::to_string(v) won't throw on wrong bit patterns. Suggested cases: a positive value, a negative value, zero, ±infinity, and the boundary where the sign-flip changes branch (+0.0 vs -0.0).
3. Truncated-read errors surface internal read width, not the field type
read_be64 always passes "uint64" to be_key_overrun, so decoding a name or int64 field that's truncated produces:
be_key: unexpected end of data reading uint64
…instead of reading name. Same for checksum* types via read_raw_hex. The error is accurate but confusing for API callers. Consider threading the field type through to the error rather than the internal read width.
4. Minor: hex_chars is defined twice
"0123456789abcdef" appears as a static const char array in both read_raw_hex and the bytes branch of decode_be_field. A single file-scope constant would remove the duplication.
5. Minor: misleading comment in the composite-key test
The composite-key test uses 1397703940 with the comment "SYS" symbol code. 1397703940 = 0x534E5344, which doesn't correspond to "SYS" in ASCII. The round-trip logic is still correct (it exercises a uint64 in a composite key), but the comment is misleading and worth fixing.
Strengths
- Replacing the hand-rolled JSON-array parser with
sysio::from_json<std::vector<std::string>>is a clear win for correctness and maintainability. - Trailing-byte detection prevents silently wrong output when
key_typesdoesn't match the actual encoded key — this is the kind of caller bug that would otherwise be very hard to track down. std::memcpyfor float/double bit-casting is correct and strict-aliasing safe.- The anonymous namespace properly scopes all helpers.
- NUL-escape error messages (
truncated escape,no terminator,bad escape) are precise and actionable. - The PR description's encoding table is excellent reference material; it should stay in a comment or doc somewhere even after merge.
Verdict
Request changes on: fix the header doc comment (#1) and add float32/float64 tests (#2) — both are straightforward. Issues #3–5 are non-blocking but worth a follow-up.
Address review comments on the BE key decoder PR:
1. Header doc example (abieos.h): the example previously showed
`"id":42`; uint64 is actually emitted as a JSON string, so fix
to `"id":"42"` and note the precision-safety rationale.
2. Thread the caller's declared field type through truncation
errors. Previously read_be* always reported the internal width
("uint64"), so truncating a name/int64/float64/checksum field
produced a confusing "reading uint64" message. Each reader now
takes a type_label and forwards it to be_key_overrun, so errors
name the user-declared type.
3. Dedupe the lowercase hex alphabet constant. Hoisted to a single
namespace-scoped `be_key_hex_chars` used by both read_raw_hex
and the bytes-field branch.
4. Fix misleading composite-key comment: 1397703940 is not the SYS
symbol code. Replaced with the actual packed ASCII value
0x535953 = 5462355, computed inline so the test documents the
encoding.
5. Add float32 and float64 test coverage -- previously absent.
Covers positive, negative, +0.0 vs -0.0 (which exercise both
branches of the sign-flip inversion), and an aliased "float" /
"double" type name.
6. Add truncation-error tests that assert the error message names
the caller's field type (name, int64, float64, checksum256).
dtaghavi
left a comment
There was a problem hiding this comment.
All five issues from the previous review have been addressed — thanks for the thorough follow-through.
What changed:
- Header doc comment — fixed to
"id":"42"with an explanatory note about JS precision. Clear and accurate. - float32/float64 tests — comprehensive: +0.0, -0.0 (branch boundary), ±1.5, ±2.5, plus alias tests for
float/double. Verified the encoded bytes independently — all decode to the expected values. - Truncated-read error messages —
type_labelis now threaded through allread_be*calls. The new error tests explicitly assert"reading name","reading int64","reading float64","reading checksum256"— exactly the right coverage. - Duplicate
hex_chars— consolidated into a singleconstexpr char be_key_hex_chars[]at namespace scope. Clean. - sym_code comment — now correctly derives
5462355from'S'|('Y'<<8)|('S'<<16)with an inline explanation of the ASCII packing. No magic numbers.
Approved.
Summary
Adds
abieos_be_key_hex_to_json()— a C API function that decodes the big-endian-encodedkeyfield of a SHiPcontract_row_kv_v0delta to JSON, using thekey_names/key_typesarrays from the contract's ABItable_def.This is a standalone utility for any C/C++ SHiP delta consumer that needs to interpret KV table keys without re-implementing CDT's
be_key_streamencoding by hand. (hyperion-history-apidoesn't use this — it has its own decoder in TypeScript atworkers/deserializer.ts:decodeBEKey. The function is here for future or third-party consumers.)Encoding spec
Mirrors
sysio::kv::be_key_streaminwire-cdt:libraries/sysiolib/contracts/sysio/kv_utils.hpp— the encoder is designed so a lexicographicmemcmpon the encoded bytes matches the natural ordering of the underlying values, which is why signed integers and floats get sign-flipped:uint8/16/32/64int8/16/32/641<<(N-1)uint128/int128int128XOR1<<127nameboolfloat(32-bit)1<<31, negative bitwise NOTdouble(64-bit)1<<63, negative bitwise NOTstring/bytes0x00→0x00 0x01, terminated by0x00 0x00checksum160checksum256checksum512checksum*types reachbe_key_streamvia a generictemplate<DataStream, size_t> operator<<(DataStream&, const fixed_bytes<Size>&)inwire-cdt:libraries/sysiolib/core/sysio/fixed_bytes.hppthat writes the raw bytes viabe_key_stream::write(const char*, size_t)— no sign flip, no length prefix.C API
JSON output conventions
Output matches the rest of abieos:
uint64/int64→ JSON string ("100") so JS callers don't lose precision viaNumberuint128/int128→0x-prefixed lowercase hex stringchecksum*→ lowercase hex string (no0xprefix)bytes→ lowercase hex string (no0xprefix)",\,\n,\r,\t,\b,\f, plus\uXXXXfor any other control byteRobustness improvements over the initial draft
parse_json_string_arraywithsysio::from_json<std::vector<std::string>>, the same JSON parser the rest of abieos already uses for ABI loading. No more brittle character-by-character state machine."be_key: N trailing byte(s) after decoding declared fields"). Silently dropping bytes hides caller bugs wherekey_typesdoesn't match the encoded key.string (truncated escape),string (no terminator), andstring (bad escape)instead of producing garbage.\uXXXXfor control bytes — necessary for any string field that legitimately contains binary data.Test coverage
15 new test cases in
src/test.cpp/check_typescovering every supported type plus error paths:abieos_string_to_name)\u0000escape, empty)scope:name + sym_code:uint64— sysio.token'saccountstable layout)key_names/key_typeslengths, empty arraysAll four existing abieos C++ tests still pass.