Documentation: Should surrealcbor be documented?

[nickchomey]

Description

I just stumbled upon the new surrealcbor package while trying to figure out some CustomDateTime errors. I dont see any mention of it in the readme, nor a readme in the package itself. Should something be added to alert people about its existence and usage?

Is there an existing issue for this?

• I have searched the existing issues

Code of Conduct

• I agree to follow this project's Code of Conduct

[nickchomey]

I also dont really see many (any?) examples on how to encode using surrealcbor - everything is about decoding. In particular, I'm having difficulty with durations. And, specifically when I set a duration from a time.Duration of 0

[mumoshu]

Hi @nickchomey! Thank you for your feedback, as always!

For encoding, it should work similarly to the existing fxamacker/cbor-based encoder because... under the hood, it just calls it.
I was wondering, but did you read the PR's description? #304 I think it should include everything one needs to use surrealcbor (from my perspective).

Re doc, if we are to add a doc about surrealcbor, a short version of the PR #304 description should suffice, as it should work almost identically.

Re CustomDateTime(although you should just use time.Time in surrealcbor) and time.Duration, it should just work in both encoding and decoding. I'd appreciate it if you could submit two separate issues on surrealcbor so that I can try to reproduce and fix!

[mumoshu]

Also, to give you a little more confidence in surrealcbor, we are running the testable examples using both the previous cbor implementation and the surrealcbor. They are passing the same set of testable examples:

surrealdb.go/.github/workflows/test.yml

Lines 33 to 49 in 513e9cd

	# v2.3.7 with Go 1.25.0, http, surrealcbor
	- surrealdb-version: 'v2.3.7'
	go-version: '1.25.0'
	connection-type: 'http'
	surrealdb-url: 'http://localhost:8000'
	surrealdb-cbor-impl: 'surrealcbor'
	# v2.3.7 with Go 1.25.0, gorilla/websocket
	- surrealdb-version: 'v2.3.7'
	go-version: '1.25.0'
	connection-type: 'ws'
	surrealdb-url: 'ws://localhost:8000/rpc'
	# v2.3.7 with Go 1.25.0, gorilla/websocket, surrealcbor
	- surrealdb-version: 'v2.3.7'
	go-version: '1.25.0'
	connection-type: 'ws'
	surrealdb-url: 'ws://localhost:8000/rpc'
	surrealdb-cbor-impl: 'surrealcbor'

In other words, surrealcbor produces the same encoding and decoding results as far as we can see from the testable examples.

[nickchomey]

Thanks for the information.

I dug a bit deeper and it seems like my issue is, indeed, with using surrealcbor with customduration with surrealcbor. Its hard to explain what the issue is, so here's a sample app that shows it.

https://github.com/nickchomey/surrealcbortest

Just define your connection settings in the main function.

It first doesnt use surrealcbor then creates a table and create, update, upsert, insert, insert bulk via RPC methods. Then the same with the query rpc method.

Then it does the same with surrealcbor and everything except the query create return errors. HOWEVER, the writes actually succeed in the db - the error comes with parsing the response. Something to do with an array length of 1 when 2 are expected. No idea what that's about - perhaps the CBOR Tags?

it was all supposed to be drop-in, so hopefully it can get resolved.

surrealdb.go/surrealcbor/decode_tag.go

Lines 242 to 250 in 1455fc3

	func (d *decoder) decodeCustomDurationTag(v reflect.Value) error {
	// CustomDuration is encoded as [seconds, nanoseconds]
	var arr []any
	if err := d.decodeValue(reflect.ValueOf(&arr).Elem()); err != nil {
	return fmt.Errorf("failed to decode duration array: %w", err)
	}
	if len(arr) != 2 {
	return fmt.Errorf("invalid duration array length: %d", len(arr))
	}

[mumoshu]

@nickchomey Much appreciated!

Realized that I missed implementing the special case where the array length for the duration can be one or even zero, according to https://surrealdb.com/docs/surrealdb/integration/cbor#tag-14. So, your analysis was totally correct. Great job...!

I'll fix it along with tests for round-trips of various durations so that we can cover all those edge cases.

[nickchomey]

Thanks! Could you also provide a bit of documentation about CBOR tags? What are they, when/why would I use them, how do I use them, do we even truly need to use them, are there performance advantages or tradeoffs to using them, etc?

The example provided above does not use them explicitly - does it happen automatically behind the scenes?

And, likewise, it would be helpful to document when/why would we use surrealcbor rather than the default cbor - what issues is it addressing, does it have any tradeoffs etc? Lots of this is in the PR (#304), but that's surely not the place for documentation...

The requested documentation could probably just be a few paragraphs in the main README, perhaps elaborate a bit more on surrealcbor-specific things in its own readme?

---

And, to make it really concrete, I'm currently still working on my own CDC/ETL mechanism to move data from mysql to surrealdb, via Debezium. Debezium can emit json or avro schemas, and adds some more granular debezium and/or Kafka Connect type annotations.

I currently decode messages from avro binary to normal go types/values, then pass the values through a converter that checks for an associated debezium type annotation, or otherwise just pass its existing go type and value through to surrealdb sdk which automatically handles CBOR conversion/serialization.

I only handle io.debezium.time.MicroTime -> models.CustomDuration
And io.debezium.time.ZonedTimestamp and similar just get some conversions between milliseconds and nanoseconds, then on to sdk.

My question, I suppose, is whether I could be adding cbor tags when I know the exact type, to make CBOR marshaling more efficient than, I presume, whatever reflection is probably happening in the go sdk?

e.g. I see in various tests, you use cbor.Tag to specify exactly which type it is.

Or is the whole point of the surrealcbor package to allow us to forget about all of this and just use normal Go types/values and let the sdk handle the rest? If that's the case, again, are there any performance tradeoffs as compared to doing things more explicitly, with the cbor tags or otherwise? Are any of the sdk's custom types still needed - eg models.DecimalString with surrealcbor?

The most ideal would be if there could be a table in a doc that shows all of the types and how we should use/handle them with and without surrealcbor.

I hope this makes sense!

[mumoshu]

@nickchomey Thank you for your feedback, as always!

Thanks! Could you also provide a bit of documentation about CBOR tags? What are they, when/why would I use them, how do I use them, do we even truly need to use them, are there performance advantages or tradeoffs to using them, etc?

It's a CBOR terminology, and SurrealDB uses those tags so that it can give specific types to some data! https://surrealdb.com/docs/surrealdb/integration/cbor

Without that, you have no way to know the exact type of data encoded in CBOR.

And, likewise, it would be helpful to document when/why would we use surrealcbor rather than the default cbor - what issues is it addressing, does it have any tradeoffs etc? Lots of this is in the PR (#304), but that's surely not the place for documentation...

OK, I'll write that down in a README.md of the surrealcbor package and link from the top-level doc so that it is discoverable. Does that sound good to you?

My question, I suppose, is whether I could be adding cbor tags when I know the exact type, to make CBOR marshaling more efficient than, I presume, whatever reflection is probably happening in the go sdk?

e.g. I see in various tests, you use cbor.Tag to specify exactly which type it is.

That cbor.Tag is from fxamacker/cbor and surrealcbor does not support unmarshaling anything to cbor.Tag struct! Although it would be easy to add support for that.

I'd definitely consider enabling surrealcbor to unmarshal cbor data to the cbor.Tag or its equivalent, if necessary.

And regarding your efficiency question, I think whether you use a custom struct or something that represents 'a CBOR value wrapped in a CBOR tag,like thecbor.Tag` struct, doesn't make much difference, because in the end, you need reflection.
If you prioritize theoretically max efficiency, you'd need a marshaler/unmarshaler using a code generator instead of reflection. That's a general rule of thumb for Go, I suppose.

Or is the whole point of the surrealcbor package to allow us to forget about all of this and just use normal Go types/values and let the sdk handle the rest?

It's more like that, although it is not yet fixed. When I decided to write surrealcbor, I tried to make it so that you can use various Go standard library types, such as time.Duration and time.Time where they make sense, so that you can forget about cbor tags in many use-cases.

If that's the case, again, are there any performance tradeoffs as compared to doing things more explicitly, with the cbor tags or otherwise? Are any of the sdk's custom types still needed - eg models.DecimalString with surrealcbor?

Custom types in the models package are necessary when you need to be more specific about which CBOR tag to use, although I'm not 100% sure for what use case you need those or not. If you don't care about the CBOR tag and your goal is to save something in SurrealDB, you should be able to use types like time.Duration and time.Time and various primitive types in many cases. You may still need to use models.RecordID because that's the only way to marshal anything as SurrealDB record IDs.

The most ideal would be if there could be a table in a doc that shows all of the types and how we should use/handle them with and without surrealcbor.

I'd appreciate it if you could clarify this a bit more!

FWIW, we have these docs:

For our standard fxamacker/cbor-based impl: https://pkg.go.dev/github.com/surrealdb/surrealdb.go@v0.10.0/pkg/models#section-readme

surrealcbor: https://pkg.go.dev/github.com/surrealdb/surrealdb.go@v0.10.0/surrealcbor

For me, it looks clear which CBOR tag is for which Go type 🤔

[nickchomey]

Thanks for the details! Though we still seem to be talking past each other - I thought I was already reasonably specific with the debezium details and doc link!

I'll write up a template doc that would address my confusion, and perhaps you color fill in/improve it?

[nickchomey]

Here's a sample doc to get started. (It was generated by Copilot - I'm SURE it is full of errors. But I'm not particularly qualified to correct it).

---

Working with types and CBOR in surrealdb.go

CBOR in the SDK (what and why)

• CBOR is a compact binary format (RFC 8949) used by SurrealDB’s RPC to encode requests and responses efficiently.
• SurrealDB extends CBOR with custom tags to represent SurrealDB Data Types (Record IDs, UUIDs, durations, geometry, etc.). See the CBOR Protocol doc for the authoritative list and on‑wire shapes:

Two CBOR stacks in the SDK

• Default (used by connection.NewConfig by default):
  • pkg/models CborMarshaler + CborUnmarshaler (both built on fxamacker/cbor).
  • Tags are emitted when you use the SDK’s wrapper types; decode is fxamacker-style for the most part.
• Optional surrealcbor:
  • A different decoder with SurrealDB‑specific behavior and performance optimizations. It still uses fxamacker for encoding.
  • Opt-in by overriding the Unmarshaler in connection.Config:
    • cfg := connection.NewConfig(u); cfg.Unmarshaler = surrealcbor.New()

When to use which

• Stick with the default unless you need one or more of:
  • NONE (tag 6) decoding to Go nil for pointers/interfaces/maps/slices.
  • Decoding to time.Time from SurrealDB’s compact datetime (tag 12).
  • Decoding to time.Duration from Duration tags (13 string, 14 compact).
  • Drop-in cbor.RawMessage support for preserving nested raw CBOR.
  • Faster/lower-allocation decode on large payloads.
• For encoding, both stacks behave the same: use SDK wrapper types to get the correct tags on the wire.

Type mapping (send and receive)

Legend:

• “Send (Go)” = what you pass to SDK to get the intended on-wire representation.
• “Decode (default)” = default CborUnmarshaler behavior.
• “Decode (surrealcbor)” = surrealcbor decoder behavior.

SurrealDB type	Send (Go)	Decode (default)	Decode (surrealcbor)
Null	nil	nil	nil
NONE (tag 6)	models.None	Not mapped to nil	Mapped to nil for ptr/interface/map/slice
Bool	bool	bool	bool
Number	int/int64/float64	Numeric types (target-dependent)	Same
String	string	string	string
Bytes	[]byte	[]byte	[]byte
Datetime	time.Time (encodes tag 0) or models.CustomDateTime (encodes tag 12)	time.Time from tag 0; tag 12 requires targeting CustomDateTime	time.Time from tag 0 and tag 12; CustomDateTime when targeted
Duration	models.CustomDuration (tag 14) or models.CustomDurationString (tag 13), or string literal (“1h2m3s”). time.Duration encodes as integer (ns).	time.Duration only if integer on wire; tags 13/14 require wrapper targets	time.Duration from tags 13/14; wrappers decode when targeted; pointers support NONE→nil
UUID	models.UUID (tag 37) or models.UUIDString (tag 9)	Wrapper targets decode; otherwise []byte/string	Wrapper targets decode; otherwise []byte/string
Decimal	models.DecimalString (tag 10)	Wrapper target or string	Wrapper target or string
Table	models.Table (tag 7)	Wrapper target or string	Wrapper target or string
Record ID	models.RecordID (tag 8)	Wrapper target; generic container if targeted broadly	Wrapper target; generic container if targeted broadly
Array	[]T / []any	Slice	Slice
Object/Map	struct / map[string]any	Struct fields / map	Struct fields / map (plus NONE→nil for fields)
Future	models.Future (tag 15)	Wrapper target or string	Wrapper target or string
Range/Bounds	Wrapper types when available; else SurrealQL expressions	Wrapper target or generic	Wrapper target or generic
Geometry (88–94)	Geometry wrappers	Wrapper target or generic arrays	Wrapper target or generic arrays

Notes:

• SurrealDB prefers returning: datetime tag 12, duration tag 14, UUID tag 37.
• To send a true SurrealDB NONE, use models.None. Go nil encodes as CBOR null/undefined (different semantics).

One annotated struct example

// Person shows preferred types for SurrealDB CBOR tags and how decode behaves.
// - Marshal (both stacks): uses fxamacker/cbor with SurrealDB tag registrations.
// - Unmarshal (default): fxamacker-style.
// - Unmarshal (surrealcbor): Surreal-aware (see field comments).
type Person struct {
  // Core identity -------------------------------------------------------------

  ID   models.RecordID   `json:"id"`
  // Sends tag 8: [table, id]. Validated on encode. Decodes to RecordID in both stacks.

  TableName models.Table  `json:"table_name,omitempty"`
  // Sends tag 7. Decodes to models.Table (or string if targeted as such).

  UUIDBin   models.UUID        `json:"uuid_bin,omitempty"`
  // Sends tag 37 (preferred). Decodes to models.UUID (or []byte if targeting []byte).

  UUIDStr   models.UUIDString  `json:"uuid_str,omitempty"`
  // Sends tag 9. Decodes to models.UUIDString (or string if targeting string).

  // Text, numbers, bytes ------------------------------------------------------

  Name  string   `json:"name"`
  // Simple text string.

  Score float64  `json:"score,omitempty"`
  // Simple number.

  Avatar []byte  `json:"avatar,omitempty"`
  // CBOR byte string.

  // Datetime and duration -----------------------------------------------------

  JoinedAt time.Time   `json:"joined_at"`
  // Encodes tag 0 (RFC3339 time). Server may return tag 12.
  // Default decode: time.Time from tag 0; tag 12 requires targeting CustomDateTime.
  // surrealcbor: time.Time from tag 0 and tag 12; CustomDateTime when targeted.

  LastSeen *time.Time  `json:"last_seen,omitempty"`
  // Pointer-friendly: with surrealcbor, NONE (tag 6) -> nil.

  TTLBin   models.CustomDuration        `json:"ttl_bin,omitempty"`
  // Sends tag 14 (compact). Both stacks decode to wrapper when targeted.
  // surrealcbor can also decode tag 14 into time.Duration if field type is time.Duration.

  TTLStr   models.CustomDurationString  `json:"ttl_str,omitempty"`
  // Sends tag 13 (string). Both stacks decode to wrapper; surrealcbor can decode to time.Duration if targeted.

  TTLAsDuration *time.Duration          `json:"ttl_as_duration,omitempty"`
  // Read-friendly field: with surrealcbor, tags 13/14 -> time.Duration and NONE -> nil.
  // If you set this when sending, it will encode as an integer (ns), not a Surreal Duration tag.

  // Decimal -------------------------------------------------------------------

  Credit models.DecimalString `json:"credit,omitempty"`
  // Sends tag 10 (decimal-as-string). Decodes to wrapper or string.

  // Optional fields and NONE --------------------------------------------------

  Email *string `json:"email,omitempty"`
  // With surrealcbor, NONE (tag 6) decodes to nil for pointers.
  // To SEND a NONE (remove a field), use models.None in a map payload (e.g., Merge/Update).

  // Future --------------------------------------------------------------------

  Expr models.Future `json:"expr,omitempty"`
  // Sends tag 15. Decodes to wrapper or string (depending on target).

  // Geometry (88–94) ----------------------------------------------------------

  Home         *models.GeometryPoint        `json:"home,omitempty"`
  Path         *models.GeometryLine         `json:"path,omitempty"`
  Area         *models.GeometryPolygon      `json:"area,omitempty"`
  MultiPoints  *models.GeometryMultiPoint   `json:"multi_points,omitempty"`
  MultiLines   *models.GeometryMultiLine    `json:"multi_lines,omitempty"`
  MultiPolys   *models.GeometryMultiPolygon `json:"multi_polys,omitempty"`
  Geoms        *models.GeometryCollection   `json:"geoms,omitempty"`
  // All send their respective geometry tags (88–94). Decode to wrappers or generic arrays.

  // Raw CBOR (deferred parsing) ----------------------------------------------

  Raw cbor.RawMessage `json:"raw,omitempty"`
  // surrealcbor: decodes a nested CBOR item as-is (raw bytes). Useful to defer parsing.
}

Typical usage

// Encode/Decode with defaults:
u, _ := url.Parse("ws://localhost:8000")
cfg := connection.NewConfig(u) // Default: models.CborMarshaler + models.CborUnmarshaler
conn := gorillaws.New(cfg)
db, _ := surrealdb.FromConnection(ctx, conn)

// Opt-in surrealcbor for decoding:
cfg2 := connection.NewConfig(u)
cfg2.Unmarshaler = surrealcbor.New() // only change the decoder
conn2 := gorillaws.New(cfg2)
db2, _ := surrealdb.FromConnection(ctx, conn2)

Consolidated guidance: behavior, ergonomics, performance

• Preferred types
  • Use SDK wrappers for SurrealDB-tagged types (RecordID, Table, UUID, Decimal, Duration, Datetime-compact when needed, Geometry, Future, NONE).
  • Use idiomatic Go for the rest (string, numbers, []byte, maps/slices, time.Time). Only use models.CustomDateTime if you specifically need to send tag 12.
• Sending NONE vs null
  • models.None sends SurrealDB NONE (tag 6). Go nil sends CBOR null/undefined (different semantics).
• Decoding
  • Default decoder is conservative and schema-agnostic (no NONE→nil).
  • surrealcbor is Surreal-aware: NONE→nil (ptr/interface/map/slice), time.Time from tags 0 and 12, time.Duration from tags 13/14 (supports pointers), cbor.RawMessage passthrough, plus full coverage for UUID/Decimal/Table/RecordID/Range/Geometry/Future.
  • surrealcbor also reduces allocations and improves throughput on large/complex payloads.
• Manual CBOR tags
  • You can craft cbor.Tag or implement MarshalCBOR for ultimate control, but you usually shouldn’t:
    • It isn’t meaningfully faster than using wrappers (reflection is still used to walk your struct).
    • You lose wrapper safety (e.g., RecordID validation).
  • Prefer wrappers and surrealcbor when you need Surreal semantics and performance on decode.

[nickchomey]

oh my god, check this out:
https://deepwiki.com/surrealdb/surrealdb.go/3.3-data-types-and-serialization
https://deepwiki.com/surrealdb/surrealdb.go/7-data-handling
https://deepwiki.com/surrealdb/surrealdb.go/7.1-data-models
https://deepwiki.com/surrealdb/surrealdb.go/7.2-cbor-serialization

I just stumbled upon a deepwiki for debezium, and thought the page was part of docs for some other tool. Turns out deepwiki creates docs for github repos. I'm sure there's mistakes, but its pretty impressive. Take a look through that entire surrealdb.go wiki/docsite - you might get some inspiration for things to add to the official docs!

[mumoshu]

@nickchomey Thank you very much for sharing all those! New to deepwiki, looks good overall! (Although it conflates test-specific things with the main SDK code and somehow incorporates #336 into a diagram, it does serve as a source of inspiration, indeed!

FWIW, my idea was to enhance https://pkg.go.dev/github.com/surrealdb/surrealdb.go#hdr-Data_Models so that it has all the links to relevant CBOR and custom type-related docs. From there, you can see the models doc which covers CBOR. I can add another link to surrealcbor so that everyone can discover it and see what it is/how to use it/etc.

Would you prefer having a comprehensive document, rather than having advanced information structured/scattered across testable examples, and go doc, and so on?

From what I could see from deepwiki and your ChatGPT-generated draft doc, everything stems from what I wrote in tests, testable examples, and package/function/struct/field go doc comments. So, if you follow the pkg.go.dev from the beginning, you can get all those ideas. So the problem could be more about discoverability, rather than the lack of a specific doc? 🤔

For code examples, I prefer having all those in testable examples, because in practice, ones written in markdown documents can quickly become stale and not consistently updated, which can confuse SDK users, even though they may look great at first glance.

Did you know that pkg.go.dev can automatically generate a comprehensive documentation from all the code comments and testable examples?

For example, here's the documentation for the Query function. It has many sections and includes various code examples generated from the testable examples(example*test.go).

https://pkg.go.dev/github.com/surrealdb/surrealdb.go#Query

If we enhanced go doc comments as we see issues, shouldn't it be better than writing a comprehensive markdown or HTML documents that duplicate the info found in godoc, because the godoc version should be easier to maintain/be updated along with code updates?

Hope this makes sense! Thank you again for your continuous support 🙏

[mumoshu]

@nickchomey Re: your use-case and make encoding efficient, I have an alternative idea- so what if we had a lower-level SDK API where you can "build any CBOR data" interactively without converting anything into another go map[string]any or another custom struct(s)?

When you have source data (like a map of keys and values, or a CSV or a table row from the data source), the most efficient way to encode it into CBOR for SurrealDB consumption would be to encode it directly to CBOR, WITHOUT reflection.

That would be possible with a kind of streaming encoder API with various CBOR writing helpers.

surrealcbor already have a streaming encoder like this:

surrealdb.go/surrealcbor/cbor_stream_test.go

Lines 16 to 30 in 1455fc3

	// Create encoder
	enc := NewEncoder(buf)
	// Encode multiple values
	values := []any{
	"hello",
	42,
	map[string]any{"key": "value"},
	models.None,
	}
	for _, v := range values {
	err := enc.Encode(v)
	require.NoErrorf(t, err, "Encode failed for %v", v)
	}

And what's missing is CBOR writing helpers.

The idea is that, if you have access to the source data, you should be able to iterate over fields(columsn, key-values pair, or whatever you call it), switch on the type of the source value, encode by calling a specific CBOR encoding function:

Imaginary code example:

var w io.Writer
encode := surrealcbor.NewStreamingEncoder(w)
for _, row := range sourceTable {
  err = encode.CBORMapTag()
  // err check
  for col, val := range row {
    switch val.(type) :{
    case int:
      err = encode.CBORInt(val)
      // errcheck
    case string:
      err = encode.CBORString(val)
      // errcheck
    case time.Duration:
      err = encode.CBORDurationCompact(val)
      // errcheck
    case yourCustomStruct:
      err = encode.CBORMapTag(val)
      // errcheck
      for k, v := range val {
        // additional switch-case to conditionally call necesasry encoding function without reflection
      }
// snip

WDYT? I can quickly build something like this if you find it useful.

[mumoshu]

Hi @nickchomey! Some update to surrealcbor vs the previous CBOR implementation, fyi- I reconsidered it and I'm going to enable surrealcbor by default soon and restructure the documentation as such.
That way, you don't need to know about the differences between the two, and can focus on how to use the SDK with its default CBOR impl.