1.0.7: ANC packets start at byte 6, as MXL stores RFC 8331; C bit follows the data stream

CHANGELOG.md

@@ -1,5 +1,12 @@
 # Changelog
 
+## 1.0.7
+
+- **The ANC grain's packets start at byte 6, as MXL stores RFC 8331.** MXL (docs/Architecture.md, `video/smpte291`) keeps the RFC 8331 payload from the Length field on: Length (16 bits), ANC_Count (8), F (2) and reserved (22) are 6 bytes, and the first ANC packet follows at byte 6. The writer padded that header to 8 bytes, so every packet sat 2 bytes late and a reader that follows MXL (mxl-st2110-gateway ANC egress, MXL's `mxl-data-probe`, gst-mxl-rs) read the 2 padding bytes as the start of the first packet. The padding is gone; each packet is still padded to a multiple of 32 bits (RFC 8331 `word_align`), now counted from the packet's own start. Length counts the bytes after the 6-byte header. The F bits are unchanged. The decoder (tests) reads the new layout.
+- **The RFC 8331 C bit now reflects the data stream; it was 1 for every packet with a known line.** C = 1 means the colour-difference (chroma) data stream. It now follows the SDK's `GetDataStreamIndex`: 0 (luma, Y; the only stream in SD) gives C = 0, 1 (chroma, HD when the luma stream is full) gives C = 1, and any other value counts as luma (C = 0). That index was also written to S and StreamNum, which RFC 8331 keeps for the link number of a multi-stream interface; S and StreamNum are now 0. The decoder (tests) maps C back to the data stream index.
+- New unit tests: the encoded bytes of two packets (the gateway's AFD golden packet with C = 0, and a chroma-stream packet with C = 1; field 1) equal a reference vector from mxl-st2110-gateway's ANC codec (`codec::serialiseGrain`); the C bit per data stream index. Checked once against the gateway's parser: no parity or checksum errors.
+- The Kubernetes example uses the `1.0.7` image.
+
 ## 1.0.6
 
 - Outputs at 29.97 and 59.94 frames/s (progressive and interlaced) play their audio. The audio read position was the frame index × floor(48000 / frame rate) samples (1601 instead of 1601.6 at 29.97, 800 instead of 800.8 at 59.94), which at today's TAI frame indexes is about 7.8 days (29.97) or 20.7 days (59.94) behind the sample index the audio sources write, far outside the audio ring, so these outputs played silence for good. Now each frame's audio block runs from the 48 kHz sample index of the frame's TAI start to that of the next frame (`mxlTimestampToIndex(48000, mxlIndexToTimestamp(frame rate, k))`), which gives the 1602/1601 cadence at 29.97 and 801/800 at 59.94 (8008 resp. 4004 samples per 5 frames). The PCM scratch buffers are sized for the largest block at bring-up, and the read-offset cap uses the largest block. Whole rates are unchanged: at 25, 50, 24, 30, 60 and 23.98 frames/s the blocks are exactly the frame index × 1920, 960, 2000, 1600, 800 and 2002 samples, as before.

CMakeLists.txt

@@ -1,7 +1,7 @@
 cmake_minimum_required(VERSION 3.24 FATAL_ERROR)
 
 project(mxl-decklink
-    VERSION 1.0.6
+    VERSION 1.0.7
     DESCRIPTION "MXL MediaFunction container for Blackmagic DeckLink"
     LANGUAGES CXX
 )

IMPLEMENTATION_PLAN.md

@@ -112,7 +112,7 @@ The MXL v1.1 sources were reviewed before design; the implementation is built on
 | Readers keep reading a flow | A writer that re-creates a flow under the same id gives it a new data file. A reader of the old file gets `MXL_ERR_FLOW_INVALID` from a read past the old head (`PosixDiscreteFlowReader.cpp:96-104`, `244-254` at `218ddaa`; same for continuous flows and `WaitForDataAt`). `Instance::getFlowReader` returns the instance's cached reader of a flow id while any holder keeps a reference (`Instance.cpp:121-143`) | The output callbacks take an invalid reader out of the sync group and stop reading it (`VideoReader::flowInvalid` / `AudioReader::flowInvalid` find it after a failed group wait with a non-blocking read past the head). The supervisor releases it and re-opens it (`ReopenSchedule`: at once, 100 ms, 200 ms, 500 ms, then 1 s). Readers live on per-channel instances, one per domain the channel reads (video and audio may come from different domains), so the release drops the last reference (1.0.4) |
 | A source that stops writing keeps its flow | A read past the head waits until its timeout and returns `MXL_ERR_OUT_OF_RANGE_TOO_EARLY`; `WaitForDataAt` likewise (`FlowSynchronizationGroup.cpp:86-112` at `218ddaa`); the runtime `headIndex` stays where the writer stopped | A read that finds nothing from a reader whose head has not moved for two frame periods is stalled: out of the sync group, read without waiting, last frame repeated / its audio silent, until the head moves (1.0.4). Reads in the DeckLink callbacks wait at most until the frames the card holds run out (§2.4 read deadline), so a stall never makes the card late |
 | Audio sample index semantics | `mxlFlowWriterOpenSamples(writer, index, count, …)` addresses the `count` samples **ending at** `index` (same convention as `mxlFlowReaderGetSamples`) | `audiowriter` computes `index = mxlTimestampToIndex(48000/1, tai_first_sample) + count` |
-| ANC flow "per RFC 8331 §2 from the length field onward" | MXL only fixes the container: `format: data`, `media_type: video/smpte291`, 4096-byte grains, 1-byte slices | `anc.cpp` serializes `Length(16) ANC_Count(8) F(2) reserved(22)` followed by 10-bit-packed ANC data packets (C, Line, HO, S, StreamNum, DID, SDID, DC, UDW, CS, word-aligned) — the RFC 8331 payload after the RTP-specific fields |
+| ANC flow "per RFC 8331 §2 from the length field onward" | MXL only fixes the container: `format: data`, `media_type: video/smpte291`, 4096-byte grains, 1-byte slices | `anc.cpp` serializes `Length(16) ANC_Count(8) F(2) reserved(22)` (6 bytes, no padding; 1.0.7) followed by 10-bit-packed ANC data packets (C from the SDK data stream index, Line, HO, S = 0, StreamNum = 0, DID, SDID, DC, UDW, CS, word-aligned) — the RFC 8331 payload after the RTP-specific fields |
 | Timestamps via `CLOCK_TAI` | `mxlGetTime()` **is** `CLOCK_TAI`-based; all index math uses 128-bit rounding | The container uses `mxlGetTime`/`mxlTimestampToIndex` for all index math so container and readers can never disagree; hardware calibration (§3.5) is an offset applied before index conversion, recalibrated every 60 s with a 1 ms sanity gate. Deviation from §3.5: a second delta above the gate in the same direction is applied (steady drift of a card without PTP lock), otherwise the offset never recovers |
 
 Everything else follows the spec directly:

SPECIFICATION.md

@@ -73,7 +73,7 @@ Then the MXL sequence runs: `mxlFlowWriterOpenGrain(inst, writer, grainIndex, &g
 
 In parallel, `audioPacket->GetBytes(&pcm)` is read. Because DeckLink delivers interleaved PCM (`bmdAudioSampleType32bitInteger`, 48 kHz) and MXL expects `audio/float32` deinterleaved per channel in a `channels` blob, the bridge converts in flight: for each sample, `float32 = int32 / 2147483648.0f`, deinterleaving into an `mxlMutableWrappedMultiBufferSlice`. The slice structure addresses ring-buffer wraparound via two fragments. `mxlFlowWriterOpenSamples` / `CommitSamples` open and finalize the batch.
 
-For ANC data, `IDeckLinkVideoFrameAncillaryPackets` is iterated and written into a `video/smpte291` flow per RFC 8331 §2 (from the length field onward) when the channel has ANC enabled (one grain per field for interlaced modes, §3.2.2). The callback must complete well under the frame interval (at 1080p50: <20 ms) — therefore no blocking operations inside it.
+For ANC data, `IDeckLinkVideoFrameAncillaryPackets` is iterated and written into a `video/smpte291` flow per RFC 8331 §2 (from the length field onward, as MXL docs/Architecture.md stores it: Length at byte 0, ANC_Count, F, reserved, the first packet at byte 6, each packet a multiple of 32 bits; the C bit is 1 only for the SDK's chroma data stream, index 1, and S and StreamNum are 0) when the channel has ANC enabled (one grain per field for interlaced modes, §3.2.2). The callback must complete well under the frame interval (at 1080p50: <20 ms) — therefore no blocking operations inside it.
 
 ### 2.4 Data Flow, Output Path (MXL → DeckLink)
 

deploy/mxl-decklink.yaml

@@ -30,7 +30,7 @@ spec:
         # feature.node.kubernetes.io/pci-1edb.present: "true"
       containers:
         - name: mxl-decklink
-          image: ghcr.io/leeo86/mxl-decklink:1.0.6
+          image: ghcr.io/leeo86/mxl-decklink:1.0.7
           resources:
             # §6.4: Guaranteed QoS — requests == limits, integer CPUs.
             # Sizing per §6.3 (Quad 2, 8× HD half duplex).

src/util/anc.cpp

@@ -21,9 +21,10 @@ namespace mxldl::util
                 }
             }
 
-            void alignTo32()
+            /// Pads with zero bits to the next 32-bit boundary counted from bit `origin`.
+            void alignTo32(std::size_t origin)
             {
-                while (_bitPos % 32 != 0)
+                while ((_bitPos - origin) % 32 != 0)
                 {
                     putBit(0);
                 }
@@ -75,9 +76,15 @@ namespace mxldl::util
                 return v;
             }
 
-            void alignTo32()
+            /// Skips to the next 32-bit boundary counted from bit `origin`.
+            void alignTo32(std::size_t origin)
             {
-                _bitPos = (_bitPos + 31) / 32 * 32;
+                _bitPos = origin + (_bitPos - origin + 31) / 32 * 32;
+            }
+
+            [[nodiscard]] std::size_t bitPos() const
+            {
+                return _bitPos;
             }
 
             [[nodiscard]] bool exhausted() const
@@ -91,6 +98,11 @@ namespace mxldl::util
             std::size_t _bitPos = 0;
         };
 
+        /// RFC 8331 header from the Length field on: Length(16) ANC_Count(8) F(2)
+        /// reserved(22). MXL stores the payload from Length on (docs/Architecture.md),
+        /// so the packets start at byte 6.
+        constexpr std::size_t kHeaderBytes = 6;
+
         /// Bits needed for one encoded packet, before 32-bit alignment.
         std::size_t packetBits(AncPacket const& p)
         {
@@ -113,15 +125,15 @@ namespace mxldl::util
     AncEncodeResult encodeRfc8331(std::vector<AncPacket> const& packets, std::size_t maxBytes, std::uint8_t field)
     {
         AncEncodeResult result;
-        if (maxBytes < 8)
+        if (maxBytes < kHeaderBytes)
         {
             result.packetsDropped = packets.size();
             return result;
         }
 
-        // Select the packets that fit; header is 8 bytes (Length + ANC_Count +
-        // F + reserved), each packet is padded to a 32-bit boundary.
-        std::size_t availableBits = (maxBytes - 8) * 8;
+        // Select the packets that fit; the header is 6 bytes (Length + ANC_Count +
+        // F + reserved), each packet is padded to a multiple of 32 bits.
+        std::size_t availableBits = (maxBytes - kHeaderBytes) * 8;
         std::vector<AncPacket const*> selected;
         for (auto const& p : packets)
         {
@@ -148,15 +160,20 @@ namespace mxldl::util
         w.put(static_cast<std::uint32_t>(selected.size()), 8);
         w.put(field & 0x3U, 2); // F: 00 progressive / unspecified, 10 field 1, 11 field 2
         w.put(0, 22); // reserved
-        w.alignTo32(); // header padded to 8 bytes so packets stay word-aligned
 
         for (auto const* p : selected)
         {
-            w.put(p->lineNumber != 0 ? 1U : 0U, 1); // C: 1 when line/offset meaningful
+            std::size_t const origin = w.bitPos();
+            // C: 1 = colour-difference (chroma) data stream. The SDK's data stream
+            // index is 0 for luma (Y, and the only stream in SD) and 1 for chroma
+            // (HD, when the luma stream is full); anything else counts as luma.
+            w.put(p->dataStreamIndex == 1 ? 1U : 0U, 1);
             w.put(p->lineNumber & 0x7ffU, 11);
             w.put(0, 12); // horizontal offset: unknown from the SDK
-            w.put(p->dataStreamIndex != 0 ? 1U : 0U, 1); // S
-            w.put(p->dataStreamIndex & 0x7fU, 7);
+            // S = 0, StreamNum = 0: no multi-stream link number (ST 425-3 etc.) is known;
+            // the Y/C stream is carried by C, not by StreamNum.
+            w.put(0, 1);
+            w.put(0, 7);
 
             std::uint16_t const did10 = addParity(p->did);
             std::uint16_t const sdid10 = addParity(p->sdid);
@@ -176,13 +193,13 @@ namespace mxldl::util
             std::uint16_t const cs9 = static_cast<std::uint16_t>(sum & 0x1ffU);
             std::uint16_t const cs10 = static_cast<std::uint16_t>(((~cs9 & 0x100U) << 1) | cs9);
             w.put(cs10, 10);
-            w.alignTo32();
+            w.alignTo32(origin); // word_align: the packet is a multiple of 32 bits
         }
 
         // Back-patch Length: size in bytes of the data section (everything
-        // after the 8-byte header), per RFC 8331 §2.1.
+        // after the 6-byte header), per RFC 8331 §2.1.
         std::size_t const totalBytes = result.payload.size();
-        std::size_t const dataBytes = totalBytes > 8 ? totalBytes - 8 : 0;
+        std::size_t const dataBytes = totalBytes - kHeaderBytes;
         result.payload[0] = static_cast<std::uint8_t>(dataBytes >> 8);
         result.payload[1] = static_cast<std::uint8_t>(dataBytes & 0xff);
 
@@ -193,7 +210,7 @@ namespace mxldl::util
     std::vector<AncPacket> decodeRfc8331(std::uint8_t const* data, std::size_t size)
     {
         std::vector<AncPacket> out;
-        if (size < 8)
+        if (size < kHeaderBytes)
         {
             return out;
         }
@@ -202,16 +219,16 @@ namespace mxldl::util
         std::uint32_t const count = r.get(8);
         r.get(2); // F
         r.get(22); // reserved
-        r.alignTo32(); // header padding (see encodeRfc8331)
 
         for (std::uint32_t i = 0; i < count && !r.exhausted(); ++i)
         {
+            std::size_t const origin = r.bitPos();
             AncPacket p;
-            r.get(1); // C
+            p.dataStreamIndex = r.get(1) != 0 ? 1 : 0; // C: chroma stream = index 1
             p.lineNumber = r.get(11);
             r.get(12); // horizontal offset
             r.get(1); // S
-            p.dataStreamIndex = static_cast<std::uint8_t>(r.get(7));
+            r.get(7); // StreamNum
             p.did = static_cast<std::uint8_t>(r.get(10) & 0xff);
             p.sdid = static_cast<std::uint8_t>(r.get(10) & 0xff);
             std::uint32_t const dc = r.get(10) & 0xff;
@@ -221,7 +238,7 @@ namespace mxldl::util
                 p.userData.push_back(static_cast<std::uint8_t>(r.get(10) & 0xff));
             }
             r.get(10); // checksum
-            r.alignTo32();
+            r.alignTo32(origin);
             out.push_back(std::move(p));
         }
         return out;

src/util/anc.hpp

@@ -16,16 +16,16 @@ namespace mxldl::util
         std::uint8_t did = 0;
         std::uint8_t sdid = 0;
         std::uint32_t lineNumber = 0;
-        std::uint8_t dataStreamIndex = 0;
+        std::uint8_t dataStreamIndex = 0; // SDK data stream: 0 luma (Y; SD), 1 chroma (C) → RFC 8331 C bit
         std::vector<std::uint8_t> userData; // UDW payload, 8-bit values
     };
 
     /// Serializes packets into the RFC 8331 §2 payload from the Length field
-    /// onward (the container stores no RTP header):
-    ///   Length(16) | ANC_Count(8) | F(2) | reserved(22) |   (F = `field`)
-    ///   per packet: C(1) LineNumber(11) HorizOffset(12) S(1) StreamNum(7)
+    /// onward, as MXL stores it (no RTP header, no extended sequence number):
+    ///   Length(16) | ANC_Count(8) | F(2) | reserved(22) |   (F = `field`; 6 bytes)
+    ///   per packet, from byte 6 on: C(1) LineNumber(11) HorizOffset(12) S(1) StreamNum(7)
     ///               DID(10) SDID(10) DataCount(10) UDW(10 × n) Checksum(10)
-    ///               word-aligned padding
+    ///               word_align (zero bits to a multiple of 32 bits per packet)
     /// DID/SDID/DC/UDW 10-bit words carry the SMPTE 291 parity bits (b8 =
     /// even parity of b0..7, b9 = !b8).
     /// \param maxBytes grain capacity; packets that would overflow are dropped

src/version.hpp

@@ -3,5 +3,5 @@
 
 namespace mxldl
 {
-    inline constexpr char const* kVersion = "1.0.6";
+    inline constexpr char const* kVersion = "1.0.7";
 }