avro

A library for working with Apache Avro encoded data in Go.

Packages

Package	Description
github.com/z5labs/avro-go	Binary encoding and decoding of Avro primitives
github.com/z5labs/avro-go/canonical	Parsing Canonical Form schema types with JSON marshaling
github.com/z5labs/avro-go/idl	Avro IDL tokenizer, parser, and printer

---

avro — Binary Encoding

Marshaling

Implement BinaryMarshaler on your type and call MarshalBinary to encode it:

type Message struct {
    Content string
}

func (m Message) MarshalAvroBinary(w *avro.BinaryWriter) error {
    return w.WriteString(m.Content)
}

var buf bytes.Buffer
err := avro.MarshalBinary(&buf, Message{Content: "hello"})

BinaryWriter exposes one write method per Avro primitive:

Method	Avro type
WriteBool(bool)	boolean
WriteInt(int32)	int
WriteLong(int64)	long
WriteFloat(float32)	float
WriteDouble(float64)	double
WriteBytes([]byte)	bytes
WriteFixed([]byte)	fixed
WriteString(string)	string

Unmarshaling

Implement BinaryUnmarshaler on your type and call UnmarshalBinary to decode it:

func (m *Message) UnmarshalAvroBinary(r *avro.BinaryReader) error {
    var err error
    m.Content, err = r.ReadString()
    return err
}

var msg Message
err := avro.UnmarshalBinary(bytes.NewReader(data), &msg)

BinaryReader mirrors BinaryWriter with corresponding Read* methods.

Streaming Arrays

An Avro array is encoded as a series of blocks — a long item count followed by that many items, terminated by a zero count — so an array whose encoding is far larger than memory can be read and written an item at a time. ArrayReader and ArrayWriter own that block framing; neither holds more than a single item.

ArrayReader.Next decodes into a destination you supply, so reusing one value across the whole array or keeping every item is your choice:

r := avro.NewArrayReader(avro.NewBinaryReader(in))

var msg Message
for {
    ok, err := r.Next(&msg)
    if err != nil {
        return err
    }
    if !ok {
        break
    }
    // use msg before the next call overwrites it
}

WriteArray writes items incrementally and terminates the array for you:

err := avro.WriteArray(avro.NewBinaryWriter(out), func(w *avro.ArrayWriter) error {
    for _, msg := range messages {
        if err := w.Write(msg); err != nil {
            return err
        }
    }
    return nil
})

Use NewArrayWriter directly when the array outlives a single function, but note that an array is only complete once its terminating block is written: an ArrayWriter that is never Closed produces a truncated array, which ArrayReader reports as ErrTruncatedArray rather than a short read.

By default each item is written straight through as its own block, buffering nothing. WithSizedBlocks(n) instead batches items into blocks that record their encoded size, bounding the buffer at n bytes plus one item:

w := avro.NewArrayWriter(avro.NewBinaryWriter(out), avro.WithSizedBlocks(64<<10))

A sized block can be discarded without decoding any of its items, which is what ArrayReader.SkipBlock is for. Because emitting sizes is optional for a writer, SkipBlock reports which path it took:

Result	Meaning
SkipSized	The block declared its size and was discarded straight from the reader; no item was decoded
SkipUnsized	The block declared no size, so nothing was consumed — drain it with Next instead
SkipNone	The array's terminating block has been reached

Single-Object Encoding

The Avro single-object encoding prepends a 2-byte magic header and an 8-byte schema fingerprint to the binary payload, allowing readers to identify the schema at runtime.

Implement SingleObjectMarshaler (embeds BinaryMarshaler plus a Fingerprint() [8]byte method) and call MarshalSingleObject:

func (m Message) Fingerprint() [8]byte {
    var fp [8]byte
    binary.LittleEndian.PutUint64(fp[:], avro.Fingerprint64([]byte(`"string"`)))
    return fp
}

var buf bytes.Buffer
err := avro.MarshalSingleObject(&buf, msg)

Decode with SingleObjectUnmarshaler and UnmarshalSingleObject:

var msg Message
err := avro.UnmarshalSingleObject(r, &msg)

UnmarshalSingleObject returns ErrBadMagic when the header is invalid and ErrFingerprintMismatch when the schema fingerprint in the payload does not match the one returned by Fingerprint().

Schema Fingerprinting

Fingerprint64 computes the 64-bit Rabin fingerprint (CRC-64-AVRO) of a schema JSON string, as defined in the Avro specification:

fp := avro.Fingerprint64([]byte(`"string"`))

---

canonical — Parsing Canonical Form

The canonical package provides typed Go representations of Avro schemas in Parsing Canonical Form. The top-level Schema type implements json.Marshaler and json.Unmarshaler, producing canonical JSON with correct field ordering and no extra whitespace.

Creating schemas

Use the constructor functions to build schemas:

s := canonical.RecordSchema(canonical.Record{
    Name: "com.example.Person",
    Fields: []canonical.Field{
        {Name: "name", Type: canonical.PrimitiveSchema(canonical.String)},
        {Name: "age", Type: canonical.PrimitiveSchema(canonical.Int)},
    },
})

Primitive constants are provided for all Avro primitives: Null, Boolean, Int, Long, Float, Double, Bytes, String.

Marshaling to canonical JSON

b, err := json.Marshal(s)
// {"name":"com.example.Person","type":"record","fields":[{"name":"name","type":"string"},{"name":"age","type":"int"}]}

Unmarshaling from canonical JSON

var s canonical.Schema
err := json.Unmarshal(data, &s)

r, ok := s.Record()
if ok {
    fmt.Println(r.Name) // com.example.Person
}

Accessor methods (Primitive(), Record(), Enum(), Array(), Map(), Union(), Fixed()) provide type-safe access to the underlying concrete type.

Converting from IDL

SchemaFrom converts a parsed idl.Schema into canonical form:

f, err := idl.Parse(strings.NewReader(`
    namespace com.example;
    schema int;
    record Person {
        string name;
        int    age;
    }
`))

schemas, err := canonical.SchemaFrom(f.Schema)

b, err := json.Marshal(schemas[1])
// {"name":"com.example.Person","type":"record","fields":[{"name":"name","type":"string"},{"name":"age","type":"int"}]}

The function returns a slice because an IDL schema can define multiple named types. Namespace qualification, type references, and all structural schema information are preserved; non-canonical attributes (doc comments, aliases, defaults) are stripped.

---

idl — Avro IDL

The idl package parses Avro IDL source files into an AST and can print an AST back to IDL text.

Parsing

Parse reads an Avro IDL source from any io.Reader and returns a *File AST:

f, err := idl.Parse(strings.NewReader(`
    namespace com.example;
    schema record User {
        string name;
        int    age;
    }
`))

The File struct contains either a *Schema or a *Protocol. A *Schema holds the top-level named types (Record, Enum, Fixed) and primitive type identifiers.

Printing

Print formats a *File AST back to Avro IDL text:

var buf bytes.Buffer
err := idl.Print(&buf, f)
fmt.Println(buf.String())

Tokenizing

Tokenize exposes the low-level lexer as an iter.Seq2[Token, error] iterator (Go 1.23+):

for tok, err := range idl.Tokenize(r) {
    if err != nil {
        // handle error
    }
    fmt.Println(tok)
}

Token types include TokenComment, TokenDocComment, TokenIdentifier, TokenSymbol, TokenString, TokenNumber, and TokenAnnotation.

---

License

Released under the MIT License.