feat(storage): add zstd replay compression

CHANGELOG.md

@@ -33,6 +33,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - 2026-04-30: Modrinth publishing metadata now includes Purpur, Spigot, and Bukkit loaders
 - 2026-05-14: Modrinth uploads now publish the matching release changelog on `main` and the `[Unreleased]` section for `dev` alpha builds
 - 2026-06-04: README content was reorganized into overview sections with dedicated Architecture, Configuration, and Commands documents under `docs/`
+- 2026-06-23: New binary `.br` archives now compress timeline payloads and chunk payloads with Zstd level 1, declare `payloadCompression: "zstd"`, bump the binary replay compatibility floor to `1.5.0-alpha.12`, and keep LZ4 replay/chunk archives readable through metadata or frame-magic compatibility fallback
 
 ### Removed
 - 2026-04-28: `General.Enable-Benchmark-Command`; `/replay benchmark` is now always permission-gated through `replay.benchmark`

README.md

@@ -11,7 +11,7 @@ It records player and nearby entity activity on the server, stores that timeline
 - Uses FoliaLib for scheduler and teleport compatibility
 - Supports file and MySQL storage backends
 - Uses a short-lived replay-list cache for saved replay commands and API listing calls
-- New saves use finalized binary `.br` archives; legacy JSON replays remain readable during the migration window, but pre-`v2` alpha `.br` inventory archives are intentionally unsupported
+- New saves use finalized binary `.br` archives with Zstd-compressed payloads; legacy JSON replays and older LZ4 `.br` archives remain readable during the migration window, but pre-`v2` alpha `.br` inventory archives are intentionally unsupported
 
 ## How this differs from client-side replay mods
 
@@ -119,6 +119,7 @@ Binary replay note:
 - Finalized `.br` archives store the recording start wall-clock timestamp in `manifest.json` as `recordingStartedAtEpochMillis`.
 - Active temp append logs also write a fixed file header carrying the same timestamp so final saves can preserve it after crash-safe recovery.
 - Current `.br` archives use format version `2` and store equipment-state and storage-inventory payloads as separate raw item-byte records.
+- New `.br` archives write `replay.bin` and chunk payloads with Zstd level 1 while preserving read compatibility with existing LZ4 archives; Zstd archives require the current binary compatibility floor.
 - Chunk-enabled `.br` archives may also include `chunks/` region entries containing palette-compressed chunk baselines that are decoded lazily during playback.
 
 ## Changelog

docs/ARCHITECTURE.md

@@ -50,6 +50,7 @@ Important recording characteristics:
 
 - Held-item swaps, equipment snapshots, and storage inventory snapshots are captured as dedicated event types.
 - New binary archives store equipment and storage payloads as separate raw-byte records.
+- New binary archives compress finalized timeline and chunk payloads with Zstd level 1; older LZ4 payloads remain readable through manifest metadata or frame magic detection.
 - Legacy JSON replays can still be read during the migration window, but new saves are finalized as `.br` archives.
 - If the server crashes while recording, startup recovery can resume and finalize orphaned append logs instead of silently losing them.
 
@@ -111,12 +112,14 @@ BetterReplay keeps storage backend selection behind the [ReplayStorage.java](../
 - `ReplayStorageCodec` and `ReplayFormatDetector` allow the loader to distinguish legacy JSON payloads from finalized binary archives.
 - New saves are written as binary `.br` archives with a manifest, payload entries, and optional chunk regions.
 - Current binary archives use replay format `v2`.
+- The `.br` ZIP container still uses stored entries; compression is applied inside `replay.bin` and each independently compressed chunk payload.
 - Replay-name and replay-summary listings flow through a shared 5-second cache in `ReplayCache`; stale manager reads refresh the active storage backend and update the cache for commands, tab completion, and API callers.
 
 Compatibility notes:
 
 - Legacy JSON replay loading is temporary compatibility support.
 - Older alpha `.br` archives that predate the `v2` inventory/event split are intentionally unsupported by current builds.
+- New Zstd-compressed `.br` archives stamp the maintained binary replay compatibility floor so older plugin builds reject them before attempting LZ4 decode.
 - Binary payload storage in MySQL requires a `LONGBLOB` data column; initialization widens the column automatically when needed.
 
 ## Commands, API, and admin tooling

docs/ARCHIVE_MANIFEST_SCHEMA.md

@@ -29,6 +29,7 @@ at the root of the `.br` archive.
 | `recordingStartedAtEpochMillis` | integer | Yes | Wall-clock recording start time in Unix epoch milliseconds |
 | `payloadChecksum` | string | Yes | Whole-payload checksum for `replay.bin` |
 | `payloadChecksumAlgorithm` | string | Yes | Name of the checksum algorithm used for `payloadChecksum` |
+| `payloadCompression` | string | No | Compression codec for `replay.bin`; omitted legacy manifests fall back to frame magic detection |
 
 ## Optional Chunk Metadata Fields
 
@@ -187,6 +188,34 @@ Example:
 "payloadChecksumAlgorithm": "CRC32C"
 ```
 
+### `payloadCompression`
+
+Type:
+
+- string
+
+Meaning:
+
+- identifies the compression codec used for the stored `replay.bin` bytes
+- lets readers select the decoder before inspecting the decompressed binary replay payload
+
+Supported values:
+
+- `zstd` for Zstd-compressed replay payloads; new archive default
+- `lz4_frame` for legacy LZ4 frame-compressed replay payloads
+
+Compatibility:
+
+- new archives should write this field
+- legacy archives may omit it; readers should then fall back to frame magic detection using `04 22 4D 18` for LZ4 frame and `28 B5 2F FD` for Zstd
+- unsupported values are a hard replay load failure
+
+Example:
+
+```json
+"payloadCompression": "zstd"
+```
+
 ### `hasChunkData`
 
 Type:
@@ -335,11 +364,12 @@ Example:
 ```json
 {
   "formatVersion": 1,
-  "recordedWithVersion": "1.5.0-SNAPSHOT",
-  "minimumViewerVersion": "1.4.0",
+  "recordedWithVersion": "1.5.0-alpha.14",
+  "minimumViewerVersion": "1.5.0-alpha.12",
   "recordingStartedAtEpochMillis": 1700000000000,
   "payloadChecksum": "7d8f8f2b",
   "payloadChecksumAlgorithm": "CRC32C",
+  "payloadCompression": "zstd",
   "hasChunkData": true,
   "chunkRegionEntryCount": 3,
   "chunkEntryCount": 418,
@@ -364,7 +394,8 @@ Recommended order:
 7. read `replay.bin`
 8. validate chunk metadata field semantics
 9. validate `payloadChecksum` using `payloadChecksumAlgorithm`
-10. if `hasChunkData` is true, build or validate the `chunks/` entry inventory
+10. validate `payloadCompression` when present, or detect compression from `replay.bin` frame magic when absent
+11. if `hasChunkData` is true, build or validate the `chunks/` entry inventory
 
 If any required replay-core step fails, the replay must not proceed to playback.
 
@@ -442,6 +473,16 @@ Reason:
 
 - the payload integrity cannot be validated reliably
 
+### Unsupported `payloadCompression`
+
+Result:
+
+- hard failure
+
+Reason:
+
+- the reader cannot select a safe decoder for `replay.bin`
+
 ### Checksum mismatch
 
 Result:
@@ -464,6 +505,7 @@ Recommended logged values:
 - `minimumViewerVersion`
 - `recordingStartedAtEpochMillis`
 - `payloadChecksumAlgorithm`
+- `payloadCompression`
 - the reason validation failed
 
 ## Reserved Future Fields

docs/BINARY_FORMAT_SPEC.md

@@ -47,11 +47,11 @@ The `manifest.json` entry provides replay metadata and compatibility information
 
 The `replay.bin` entry contains the finalized replay payload:
 
-- compressed as a single LZ4-compressed payload
+- compressed as a single payload frame, using Zstd level 1 for new archives
 - decompressed fully into a heap `byte[]` when loaded for playback
 - decoded lazily from that in-memory byte array as playback advances
 
-The v1 implementation stores `replay.bin` as a single LZ4 frame.
+Legacy archives may store `replay.bin` as a single LZ4 frame and remain readable.
 
 During active recording, BetterReplay first writes an append-only temp file under `replays/.tmp/`.
 That append-log has its own fixed file header so metadata needed during crash recovery is persisted before any framed records are appended.
@@ -65,7 +65,20 @@ The `.br` file is a ZIP-style archive whose entries are stored using `STORE` rat
 | Entry name | Required | Purpose |
 |-----------|----------|---------|
 | `manifest.json` | Yes | Replay metadata, versioning, checksum, and compatibility gate |
-| `replay.bin` | Yes | LZ4-compressed finalized replay payload |
+| `replay.bin` | Yes | Compressed finalized replay payload |
+
+### Timeline payload compression
+
+New archives declare `payloadCompression` in `manifest.json` and write `replay.bin` with Zstd level 1.
+
+Supported timeline payload codecs:
+
+| Manifest value | Frame magic | Meaning |
+|----------------|-------------|---------|
+| `lz4_frame` | `04 22 4D 18` | Legacy LZ4 frame-compressed replay payload |
+| `zstd` | `28 B5 2F FD` | Zstd-compressed replay payload; new archive default |
+
+Readers must use `payloadCompression` when it is present. If the field is absent, readers may fall back to frame magic detection for compatibility with existing archives.
 
 ### Optional chunk-enabled entries
 
@@ -104,11 +117,12 @@ Examples:
 
 The finalized chunk region format and the temp region append-log both use a one-byte codec identifier.
 
-v1 freezes exactly one supported codec:
+Supported finalized chunk payload codecs:
 
 | Codec ID | Name | Meaning |
 |----------|------|---------|
 | `0x01` | `LZ4_FRAME` | Payload bytes are one standalone LZ4 frame for a single chunk snapshot payload |
+| `0x02` | `ZSTD` | Payload bytes are one standalone Zstd frame for a single chunk snapshot payload; new archive default |
 
 Unknown chunk codec identifiers are a hard parse failure for the affected entry.
 
@@ -131,7 +145,7 @@ Rules:
 
 ## Legacy Chunk Baseline Payload (`BRCS`)
 
-The uncompressed bytes inside each chunk payload encode one full chunk baseline snapshot before the outer `LZ4_FRAME` compression is applied.
+The uncompressed bytes inside each chunk payload encode one full chunk baseline snapshot before the per-chunk compression frame is applied.
 
 v1 uses this layout:
 
@@ -275,7 +289,7 @@ Rows are written in deterministic lexicographic order by `(localChunkX, localChu
 |-------|-------|----------|-------|
 | `localChunkX` | 1 byte | unsigned byte | `0-31` within the region |
 | `localChunkZ` | 1 byte | unsigned byte | `0-31` within the region |
-| `codecId` | 1 byte | unsigned byte | v1 supports only `0x01` (`LZ4_FRAME`) |
+| `codecId` | 1 byte | unsigned byte | `0x01` (`LZ4_FRAME`) or `0x02` (`ZSTD`) |
 | reserved | 1 byte | zero-filled | must be `0x00` in v1 |
 | `payloadOffset` | 4 bytes | little-endian signed int32, non-negative in valid files | offset relative to the start of the payload area, not the file start |
 | `compressedLength` | 4 bytes | little-endian signed int32, positive in valid files | compressed payload byte count |
@@ -319,7 +333,7 @@ Each appended chunk snapshot record uses this layout:
 |-------|-------|----------|-------|
 | `localChunkX` | 1 byte | unsigned byte | `0-31` within the region |
 | `localChunkZ` | 1 byte | unsigned byte | `0-31` within the region |
-| `codecId` | 1 byte | unsigned byte | v1 supports only `0x01` (`LZ4_FRAME`) |
+| `codecId` | 1 byte | unsigned byte | `0x01` (`LZ4_FRAME`) or `0x02` (`ZSTD`) |
 | flags | 1 byte | unsigned byte | must be `0x00` in v1 |
 | `uncompressedLength` | 4 bytes | little-endian signed int32, positive in valid files | original payload length |
 | `compressedLength` | 4 bytes | little-endian signed int32, positive in valid files | stored payload byte count |
@@ -380,7 +394,7 @@ Total append-log header size: `16` bytes.
 
 ## Replay Payload Model
 
-After LZ4 decompression, `replay.bin` is treated as a single binary payload with three logical parts:
+After decompression, `replay.bin` is treated as a single binary payload with three logical parts:
 
 1. payload header
 2. event stream
@@ -766,4 +780,4 @@ Reserved future areas include:
 - additional event tags
 - more advanced offline tooling
 
-Those future additions should still preserve the v1 principle that unsupported or ambiguous data fails clearly rather than being partially interpreted.
+Those future additions should still preserve the v1 principle that unsupported or ambiguous data fails clearly rather than being partially interpreted.

pom.xml

@@ -75,6 +75,11 @@
             <artifactId>lz4-java</artifactId>
             <version>1.8.0</version>
         </dependency>
+        <dependency>
+            <groupId>com.github.luben</groupId>
+            <artifactId>zstd-jni</artifactId>
+            <version>1.5.6-9</version>
+        </dependency>
         <dependency>
             <groupId>org.geysermc.floodgate</groupId>
             <artifactId>api</artifactId>

src/main/java/me/justindevb/replay/playback/ReplayChunkPlaybackCache.java

@@ -7,9 +7,7 @@
 import me.justindevb.replay.storage.binary.BinaryChunkRegionCodec;
 import me.justindevb.replay.storage.binary.BinaryChunkRegionEntry;
 import me.justindevb.replay.storage.binary.BinaryReplayFormat;
-import net.jpountz.lz4.LZ4FrameInputStream;
 
-import java.io.ByteArrayInputStream;
 import java.io.IOException;
 import java.util.HashMap;
 import java.util.Map;
@@ -115,7 +113,7 @@ private Optional<ReplayChunkSnapshot> decodeChunk(ChunkCoordinate coordinate) {
             });
             for (BinaryChunkRegionEntry entry : decodedRegion.entries()) {
                 if (entry.localChunkX() == coordinate.localChunkX() && entry.localChunkZ() == coordinate.localChunkZ()) {
-                    byte[] payload = decompress(entry.compressedPayload());
+                    byte[] payload = entry.compression().decompress(entry.compressedPayload());
                     return Optional.of(switch (chunkData.metadata().payloadFormat()) {
                         case BRCS -> new ReplayChunkSnapshot.LegacyBlockStateSnapshot(legacyPayloadCodec.decode(payload));
                         case BRCP -> new ReplayChunkSnapshot.PacketFriendlySnapshot(packetFriendlyPayloadCodec.decode(payload));
@@ -131,9 +129,4 @@ private Optional<ReplayChunkSnapshot> decodeChunk(ChunkCoordinate coordinate) {
         }
     }
 
-    private static byte[] decompress(byte[] compressedPayload) throws IOException {
-        try (LZ4FrameInputStream lz4 = new LZ4FrameInputStream(new ByteArrayInputStream(compressedPayload))) {
-            return lz4.readAllBytes();
-        }
-    }
-}
+}

src/main/java/me/justindevb/replay/storage/binary/BinaryChunkCompression.java

@@ -6,18 +6,31 @@
  * Frozen per-chunk payload codecs for chunk baseline storage.
  */
 public enum BinaryChunkCompression {
-    LZ4_FRAME(1);
+    LZ4_FRAME(1, BinaryReplayPayloadCompression.LZ4_FRAME),
+    ZSTD(2, BinaryReplayPayloadCompression.ZSTD);
+
+    public static final BinaryChunkCompression DEFAULT = ZSTD;
 
     private final int codecId;
+    private final BinaryReplayPayloadCompression payloadCompression;
 
-    BinaryChunkCompression(int codecId) {
+    BinaryChunkCompression(int codecId, BinaryReplayPayloadCompression payloadCompression) {
         this.codecId = codecId;
+        this.payloadCompression = payloadCompression;
     }
 
     public int codecId() {
         return codecId;
     }
 
+    public byte[] compress(byte[] payload) throws IOException {
+        return payloadCompression.compress(payload);
+    }
+
+    public byte[] decompress(byte[] compressedPayload) throws IOException {
+        return payloadCompression.decompress(compressedPayload);
+    }
+
     public static BinaryChunkCompression fromCodecId(int codecId) throws IOException {
         for (BinaryChunkCompression compression : values()) {
             if (compression.codecId == codecId) {
@@ -26,4 +39,4 @@ public static BinaryChunkCompression fromCodecId(int codecId) throws IOException
         }
         throw new IOException("Unsupported chunk payload codec id: " + codecId);
     }
-}
+}

src/main/java/me/justindevb/replay/storage/binary/BinaryChunkTempRegionFileWriter.java

@@ -4,9 +4,7 @@
 import me.justindevb.replay.chunk.ChunkRecordingArtifacts;
 import me.justindevb.replay.chunk.ChunkRegionKey;
 import me.justindevb.replay.chunk.ChunkTempRegionWriter;
-import net.jpountz.lz4.LZ4FrameOutputStream;
 
-import java.io.ByteArrayOutputStream;
 import java.io.IOException;
 import java.io.OutputStream;
 import java.nio.file.Files;
@@ -48,12 +46,13 @@ public void append(CapturedChunkBaseline baseline) throws IOException {
         Files.createDirectories(regionPath.getParent());
         boolean writeHeader = Files.notExists(regionPath);
 
-        byte[] compressedPayload = compress(baseline.payloadBytes());
+        BinaryChunkCompression compression = BinaryChunkCompression.DEFAULT;
+        byte[] compressedPayload = compression.compress(baseline.payloadBytes());
         BinaryChunkTempRegionAppendRecord record = new BinaryChunkTempRegionAppendRecord(
                 baseline.coordinate().localChunkX(),
                 baseline.coordinate().localChunkZ(),
                 baseline.payloadBytes().length,
-                BinaryChunkCompression.LZ4_FRAME,
+                compression,
                 compressedPayload);
 
         try (OutputStream outputStream = Files.newOutputStream(
@@ -92,11 +91,4 @@ private Path resolveRegionPath(ChunkRegionKey regionKey) {
                 .resolve("r." + regionKey.regionX() + "." + regionKey.regionZ() + TEMP_REGION_EXTENSION);
     }
 
-    private static byte[] compress(byte[] payloadBytes) throws IOException {
-        ByteArrayOutputStream out = new ByteArrayOutputStream();
-        try (LZ4FrameOutputStream lz4 = new LZ4FrameOutputStream(out)) {
-            lz4.write(payloadBytes);
-        }
-        return out.toByteArray();
-    }
-}
+}

src/main/java/me/justindevb/replay/storage/binary/BinaryReplayArchiveFinalizer.java

@@ -5,7 +5,6 @@
 import me.justindevb.replay.recording.TimelineEvent;
 import me.justindevb.replay.storage.ReplayFinalizer;
 import me.justindevb.replay.util.VersionUtil;
-import net.jpountz.lz4.LZ4FrameOutputStream;
 
 import java.io.ByteArrayOutputStream;
 import java.io.IOException;
@@ -66,14 +65,16 @@ public byte[] finalizeRecoveredReplay(
             ReplayChunkData chunkData
     ) throws IOException {
         byte[] finalizedPayload = buildFinalizedPayload(recovery.records(), recovery.timeline(), recovery.stringTable());
-        byte[] compressedPayload = compress(finalizedPayload);
+        BinaryReplayPayloadCompression payloadCompression = BinaryReplayPayloadCompression.DEFAULT;
+        byte[] compressedPayload = payloadCompression.compress(finalizedPayload);
         ReplayChunkData effectiveChunkData = chunkData != null ? chunkData : ReplayChunkData.NONE;
         BinaryReplayManifest manifest = BinaryReplayManifest.createV1(
                 pluginVersion,
                 VersionUtil.MIN_RECORDING_VERSION,
                 resolveRecordingStartedAtEpochMillis(recovery),
                 crc32cHex(compressedPayload),
-                effectiveChunkData.metadata());
+                effectiveChunkData.metadata(),
+                payloadCompression);
         byte[] manifestBytes = gson.toJson(manifest).getBytes(StandardCharsets.UTF_8);
         return buildArchive(manifestBytes, compressedPayload, effectiveChunkData);
     }
@@ -181,14 +182,6 @@ private static byte[] buildIndexSection(List<String> stringTable, List<BinaryTic
         return out.toByteArray();
     }
 
-    private static byte[] compress(byte[] payload) throws IOException {
-        ByteArrayOutputStream out = new ByteArrayOutputStream();
-        try (LZ4FrameOutputStream lz4 = new LZ4FrameOutputStream(out)) {
-            lz4.write(payload);
-        }
-        return out.toByteArray();
-    }
-
     private static byte[] buildArchive(byte[] manifestBytes, byte[] replayBytes, ReplayChunkData chunkData) throws IOException {
         ByteArrayOutputStream out = new ByteArrayOutputStream();
         try (ZipOutputStream zip = new ZipOutputStream(out)) {