diff --git a/CHANGELOG.md b/CHANGELOG.md index 2908de5..ceef80f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -33,6 +33,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - 2026-04-30: Modrinth publishing metadata now includes Purpur, Spigot, and Bukkit loaders - 2026-05-14: Modrinth uploads now publish the matching release changelog on `main` and the `[Unreleased]` section for `dev` alpha builds - 2026-06-04: README content was reorganized into overview sections with dedicated Architecture, Configuration, and Commands documents under `docs/` +- 2026-06-23: New binary `.br` archives now compress timeline payloads and chunk payloads with Zstd level 1, declare `payloadCompression: "zstd"`, bump the binary replay compatibility floor to `1.5.0-alpha.12`, and keep LZ4 replay/chunk archives readable through metadata or frame-magic compatibility fallback ### Removed - 2026-04-28: `General.Enable-Benchmark-Command`; `/replay benchmark` is now always permission-gated through `replay.benchmark` diff --git a/README.md b/README.md index a30aa8a..54b4e7a 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ It records player and nearby entity activity on the server, stores that timeline - Uses FoliaLib for scheduler and teleport compatibility - Supports file and MySQL storage backends - Uses a short-lived replay-list cache for saved replay commands and API listing calls -- New saves use finalized binary `.br` archives; legacy JSON replays remain readable during the migration window, but pre-`v2` alpha `.br` inventory archives are intentionally unsupported +- New saves use finalized binary `.br` archives with Zstd-compressed payloads; legacy JSON replays and older LZ4 `.br` archives remain readable during the migration window, but pre-`v2` alpha `.br` inventory archives are intentionally unsupported ## How this differs from client-side replay mods @@ -119,6 +119,7 @@ Binary replay note: - Finalized `.br` archives store the recording start wall-clock timestamp in `manifest.json` as `recordingStartedAtEpochMillis`. - Active temp append logs also write a fixed file header carrying the same timestamp so final saves can preserve it after crash-safe recovery. - Current `.br` archives use format version `2` and store equipment-state and storage-inventory payloads as separate raw item-byte records. +- New `.br` archives write `replay.bin` and chunk payloads with Zstd level 1 while preserving read compatibility with existing LZ4 archives; Zstd archives require the current binary compatibility floor. - Chunk-enabled `.br` archives may also include `chunks/` region entries containing palette-compressed chunk baselines that are decoded lazily during playback. ## Changelog diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index ff05053..793fa46 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -50,6 +50,7 @@ Important recording characteristics: - Held-item swaps, equipment snapshots, and storage inventory snapshots are captured as dedicated event types. - New binary archives store equipment and storage payloads as separate raw-byte records. +- New binary archives compress finalized timeline and chunk payloads with Zstd level 1; older LZ4 payloads remain readable through manifest metadata or frame magic detection. - Legacy JSON replays can still be read during the migration window, but new saves are finalized as `.br` archives. - If the server crashes while recording, startup recovery can resume and finalize orphaned append logs instead of silently losing them. @@ -111,12 +112,14 @@ BetterReplay keeps storage backend selection behind the [ReplayStorage.java](../ - `ReplayStorageCodec` and `ReplayFormatDetector` allow the loader to distinguish legacy JSON payloads from finalized binary archives. - New saves are written as binary `.br` archives with a manifest, payload entries, and optional chunk regions. - Current binary archives use replay format `v2`. +- The `.br` ZIP container still uses stored entries; compression is applied inside `replay.bin` and each independently compressed chunk payload. - Replay-name and replay-summary listings flow through a shared 5-second cache in `ReplayCache`; stale manager reads refresh the active storage backend and update the cache for commands, tab completion, and API callers. Compatibility notes: - Legacy JSON replay loading is temporary compatibility support. - Older alpha `.br` archives that predate the `v2` inventory/event split are intentionally unsupported by current builds. +- New Zstd-compressed `.br` archives stamp the maintained binary replay compatibility floor so older plugin builds reject them before attempting LZ4 decode. - Binary payload storage in MySQL requires a `LONGBLOB` data column; initialization widens the column automatically when needed. ## Commands, API, and admin tooling diff --git a/docs/ARCHIVE_MANIFEST_SCHEMA.md b/docs/ARCHIVE_MANIFEST_SCHEMA.md index 3fd4a07..e2b70f3 100644 --- a/docs/ARCHIVE_MANIFEST_SCHEMA.md +++ b/docs/ARCHIVE_MANIFEST_SCHEMA.md @@ -29,6 +29,7 @@ at the root of the `.br` archive. | `recordingStartedAtEpochMillis` | integer | Yes | Wall-clock recording start time in Unix epoch milliseconds | | `payloadChecksum` | string | Yes | Whole-payload checksum for `replay.bin` | | `payloadChecksumAlgorithm` | string | Yes | Name of the checksum algorithm used for `payloadChecksum` | +| `payloadCompression` | string | No | Compression codec for `replay.bin`; omitted legacy manifests fall back to frame magic detection | ## Optional Chunk Metadata Fields @@ -187,6 +188,34 @@ Example: "payloadChecksumAlgorithm": "CRC32C" ``` +### `payloadCompression` + +Type: + +- string + +Meaning: + +- identifies the compression codec used for the stored `replay.bin` bytes +- lets readers select the decoder before inspecting the decompressed binary replay payload + +Supported values: + +- `zstd` for Zstd-compressed replay payloads; new archive default +- `lz4_frame` for legacy LZ4 frame-compressed replay payloads + +Compatibility: + +- new archives should write this field +- legacy archives may omit it; readers should then fall back to frame magic detection using `04 22 4D 18` for LZ4 frame and `28 B5 2F FD` for Zstd +- unsupported values are a hard replay load failure + +Example: + +```json +"payloadCompression": "zstd" +``` + ### `hasChunkData` Type: @@ -335,11 +364,12 @@ Example: ```json { "formatVersion": 1, - "recordedWithVersion": "1.5.0-SNAPSHOT", - "minimumViewerVersion": "1.4.0", + "recordedWithVersion": "1.5.0-alpha.14", + "minimumViewerVersion": "1.5.0-alpha.12", "recordingStartedAtEpochMillis": 1700000000000, "payloadChecksum": "7d8f8f2b", "payloadChecksumAlgorithm": "CRC32C", + "payloadCompression": "zstd", "hasChunkData": true, "chunkRegionEntryCount": 3, "chunkEntryCount": 418, @@ -364,7 +394,8 @@ Recommended order: 7. read `replay.bin` 8. validate chunk metadata field semantics 9. validate `payloadChecksum` using `payloadChecksumAlgorithm` -10. if `hasChunkData` is true, build or validate the `chunks/` entry inventory +10. validate `payloadCompression` when present, or detect compression from `replay.bin` frame magic when absent +11. if `hasChunkData` is true, build or validate the `chunks/` entry inventory If any required replay-core step fails, the replay must not proceed to playback. @@ -442,6 +473,16 @@ Reason: - the payload integrity cannot be validated reliably +### Unsupported `payloadCompression` + +Result: + +- hard failure + +Reason: + +- the reader cannot select a safe decoder for `replay.bin` + ### Checksum mismatch Result: @@ -464,6 +505,7 @@ Recommended logged values: - `minimumViewerVersion` - `recordingStartedAtEpochMillis` - `payloadChecksumAlgorithm` +- `payloadCompression` - the reason validation failed ## Reserved Future Fields diff --git a/docs/BINARY_FORMAT_SPEC.md b/docs/BINARY_FORMAT_SPEC.md index b260ad8..bbd6de5 100644 --- a/docs/BINARY_FORMAT_SPEC.md +++ b/docs/BINARY_FORMAT_SPEC.md @@ -47,11 +47,11 @@ The `manifest.json` entry provides replay metadata and compatibility information The `replay.bin` entry contains the finalized replay payload: -- compressed as a single LZ4-compressed payload +- compressed as a single payload frame, using Zstd level 1 for new archives - decompressed fully into a heap `byte[]` when loaded for playback - decoded lazily from that in-memory byte array as playback advances -The v1 implementation stores `replay.bin` as a single LZ4 frame. +Legacy archives may store `replay.bin` as a single LZ4 frame and remain readable. During active recording, BetterReplay first writes an append-only temp file under `replays/.tmp/`. That append-log has its own fixed file header so metadata needed during crash recovery is persisted before any framed records are appended. @@ -65,7 +65,20 @@ The `.br` file is a ZIP-style archive whose entries are stored using `STORE` rat | Entry name | Required | Purpose | |-----------|----------|---------| | `manifest.json` | Yes | Replay metadata, versioning, checksum, and compatibility gate | -| `replay.bin` | Yes | LZ4-compressed finalized replay payload | +| `replay.bin` | Yes | Compressed finalized replay payload | + +### Timeline payload compression + +New archives declare `payloadCompression` in `manifest.json` and write `replay.bin` with Zstd level 1. + +Supported timeline payload codecs: + +| Manifest value | Frame magic | Meaning | +|----------------|-------------|---------| +| `lz4_frame` | `04 22 4D 18` | Legacy LZ4 frame-compressed replay payload | +| `zstd` | `28 B5 2F FD` | Zstd-compressed replay payload; new archive default | + +Readers must use `payloadCompression` when it is present. If the field is absent, readers may fall back to frame magic detection for compatibility with existing archives. ### Optional chunk-enabled entries @@ -104,11 +117,12 @@ Examples: The finalized chunk region format and the temp region append-log both use a one-byte codec identifier. -v1 freezes exactly one supported codec: +Supported finalized chunk payload codecs: | Codec ID | Name | Meaning | |----------|------|---------| | `0x01` | `LZ4_FRAME` | Payload bytes are one standalone LZ4 frame for a single chunk snapshot payload | +| `0x02` | `ZSTD` | Payload bytes are one standalone Zstd frame for a single chunk snapshot payload; new archive default | Unknown chunk codec identifiers are a hard parse failure for the affected entry. @@ -131,7 +145,7 @@ Rules: ## Legacy Chunk Baseline Payload (`BRCS`) -The uncompressed bytes inside each chunk payload encode one full chunk baseline snapshot before the outer `LZ4_FRAME` compression is applied. +The uncompressed bytes inside each chunk payload encode one full chunk baseline snapshot before the per-chunk compression frame is applied. v1 uses this layout: @@ -275,7 +289,7 @@ Rows are written in deterministic lexicographic order by `(localChunkX, localChu |-------|-------|----------|-------| | `localChunkX` | 1 byte | unsigned byte | `0-31` within the region | | `localChunkZ` | 1 byte | unsigned byte | `0-31` within the region | -| `codecId` | 1 byte | unsigned byte | v1 supports only `0x01` (`LZ4_FRAME`) | +| `codecId` | 1 byte | unsigned byte | `0x01` (`LZ4_FRAME`) or `0x02` (`ZSTD`) | | reserved | 1 byte | zero-filled | must be `0x00` in v1 | | `payloadOffset` | 4 bytes | little-endian signed int32, non-negative in valid files | offset relative to the start of the payload area, not the file start | | `compressedLength` | 4 bytes | little-endian signed int32, positive in valid files | compressed payload byte count | @@ -319,7 +333,7 @@ Each appended chunk snapshot record uses this layout: |-------|-------|----------|-------| | `localChunkX` | 1 byte | unsigned byte | `0-31` within the region | | `localChunkZ` | 1 byte | unsigned byte | `0-31` within the region | -| `codecId` | 1 byte | unsigned byte | v1 supports only `0x01` (`LZ4_FRAME`) | +| `codecId` | 1 byte | unsigned byte | `0x01` (`LZ4_FRAME`) or `0x02` (`ZSTD`) | | flags | 1 byte | unsigned byte | must be `0x00` in v1 | | `uncompressedLength` | 4 bytes | little-endian signed int32, positive in valid files | original payload length | | `compressedLength` | 4 bytes | little-endian signed int32, positive in valid files | stored payload byte count | @@ -380,7 +394,7 @@ Total append-log header size: `16` bytes. ## Replay Payload Model -After LZ4 decompression, `replay.bin` is treated as a single binary payload with three logical parts: +After decompression, `replay.bin` is treated as a single binary payload with three logical parts: 1. payload header 2. event stream @@ -766,4 +780,4 @@ Reserved future areas include: - additional event tags - more advanced offline tooling -Those future additions should still preserve the v1 principle that unsupported or ambiguous data fails clearly rather than being partially interpreted. \ No newline at end of file +Those future additions should still preserve the v1 principle that unsupported or ambiguous data fails clearly rather than being partially interpreted. diff --git a/pom.xml b/pom.xml index f4d0613..f6b8ccd 100644 --- a/pom.xml +++ b/pom.xml @@ -75,6 +75,11 @@ lz4-java 1.8.0 + + com.github.luben + zstd-jni + 1.5.6-9 + org.geysermc.floodgate api diff --git a/src/main/java/me/justindevb/replay/playback/ReplayChunkPlaybackCache.java b/src/main/java/me/justindevb/replay/playback/ReplayChunkPlaybackCache.java index bf62bb3..80a6b18 100644 --- a/src/main/java/me/justindevb/replay/playback/ReplayChunkPlaybackCache.java +++ b/src/main/java/me/justindevb/replay/playback/ReplayChunkPlaybackCache.java @@ -7,9 +7,7 @@ import me.justindevb.replay.storage.binary.BinaryChunkRegionCodec; import me.justindevb.replay.storage.binary.BinaryChunkRegionEntry; import me.justindevb.replay.storage.binary.BinaryReplayFormat; -import net.jpountz.lz4.LZ4FrameInputStream; -import java.io.ByteArrayInputStream; import java.io.IOException; import java.util.HashMap; import java.util.Map; @@ -115,7 +113,7 @@ private Optional decodeChunk(ChunkCoordinate coordinate) { }); for (BinaryChunkRegionEntry entry : decodedRegion.entries()) { if (entry.localChunkX() == coordinate.localChunkX() && entry.localChunkZ() == coordinate.localChunkZ()) { - byte[] payload = decompress(entry.compressedPayload()); + byte[] payload = entry.compression().decompress(entry.compressedPayload()); return Optional.of(switch (chunkData.metadata().payloadFormat()) { case BRCS -> new ReplayChunkSnapshot.LegacyBlockStateSnapshot(legacyPayloadCodec.decode(payload)); case BRCP -> new ReplayChunkSnapshot.PacketFriendlySnapshot(packetFriendlyPayloadCodec.decode(payload)); @@ -131,9 +129,4 @@ private Optional decodeChunk(ChunkCoordinate coordinate) { } } - private static byte[] decompress(byte[] compressedPayload) throws IOException { - try (LZ4FrameInputStream lz4 = new LZ4FrameInputStream(new ByteArrayInputStream(compressedPayload))) { - return lz4.readAllBytes(); - } - } -} \ No newline at end of file +} diff --git a/src/main/java/me/justindevb/replay/storage/binary/BinaryChunkCompression.java b/src/main/java/me/justindevb/replay/storage/binary/BinaryChunkCompression.java index 6cf46c7..df48dea 100644 --- a/src/main/java/me/justindevb/replay/storage/binary/BinaryChunkCompression.java +++ b/src/main/java/me/justindevb/replay/storage/binary/BinaryChunkCompression.java @@ -6,18 +6,31 @@ * Frozen per-chunk payload codecs for chunk baseline storage. */ public enum BinaryChunkCompression { - LZ4_FRAME(1); + LZ4_FRAME(1, BinaryReplayPayloadCompression.LZ4_FRAME), + ZSTD(2, BinaryReplayPayloadCompression.ZSTD); + + public static final BinaryChunkCompression DEFAULT = ZSTD; private final int codecId; + private final BinaryReplayPayloadCompression payloadCompression; - BinaryChunkCompression(int codecId) { + BinaryChunkCompression(int codecId, BinaryReplayPayloadCompression payloadCompression) { this.codecId = codecId; + this.payloadCompression = payloadCompression; } public int codecId() { return codecId; } + public byte[] compress(byte[] payload) throws IOException { + return payloadCompression.compress(payload); + } + + public byte[] decompress(byte[] compressedPayload) throws IOException { + return payloadCompression.decompress(compressedPayload); + } + public static BinaryChunkCompression fromCodecId(int codecId) throws IOException { for (BinaryChunkCompression compression : values()) { if (compression.codecId == codecId) { @@ -26,4 +39,4 @@ public static BinaryChunkCompression fromCodecId(int codecId) throws IOException } throw new IOException("Unsupported chunk payload codec id: " + codecId); } -} \ No newline at end of file +} diff --git a/src/main/java/me/justindevb/replay/storage/binary/BinaryChunkTempRegionFileWriter.java b/src/main/java/me/justindevb/replay/storage/binary/BinaryChunkTempRegionFileWriter.java index 07f85c5..bf0ff07 100644 --- a/src/main/java/me/justindevb/replay/storage/binary/BinaryChunkTempRegionFileWriter.java +++ b/src/main/java/me/justindevb/replay/storage/binary/BinaryChunkTempRegionFileWriter.java @@ -4,9 +4,7 @@ import me.justindevb.replay.chunk.ChunkRecordingArtifacts; import me.justindevb.replay.chunk.ChunkRegionKey; import me.justindevb.replay.chunk.ChunkTempRegionWriter; -import net.jpountz.lz4.LZ4FrameOutputStream; -import java.io.ByteArrayOutputStream; import java.io.IOException; import java.io.OutputStream; import java.nio.file.Files; @@ -48,12 +46,13 @@ public void append(CapturedChunkBaseline baseline) throws IOException { Files.createDirectories(regionPath.getParent()); boolean writeHeader = Files.notExists(regionPath); - byte[] compressedPayload = compress(baseline.payloadBytes()); + BinaryChunkCompression compression = BinaryChunkCompression.DEFAULT; + byte[] compressedPayload = compression.compress(baseline.payloadBytes()); BinaryChunkTempRegionAppendRecord record = new BinaryChunkTempRegionAppendRecord( baseline.coordinate().localChunkX(), baseline.coordinate().localChunkZ(), baseline.payloadBytes().length, - BinaryChunkCompression.LZ4_FRAME, + compression, compressedPayload); try (OutputStream outputStream = Files.newOutputStream( @@ -92,11 +91,4 @@ private Path resolveRegionPath(ChunkRegionKey regionKey) { .resolve("r." + regionKey.regionX() + "." + regionKey.regionZ() + TEMP_REGION_EXTENSION); } - private static byte[] compress(byte[] payloadBytes) throws IOException { - ByteArrayOutputStream out = new ByteArrayOutputStream(); - try (LZ4FrameOutputStream lz4 = new LZ4FrameOutputStream(out)) { - lz4.write(payloadBytes); - } - return out.toByteArray(); - } -} \ No newline at end of file +} diff --git a/src/main/java/me/justindevb/replay/storage/binary/BinaryReplayArchiveFinalizer.java b/src/main/java/me/justindevb/replay/storage/binary/BinaryReplayArchiveFinalizer.java index 587a728..1e9f539 100644 --- a/src/main/java/me/justindevb/replay/storage/binary/BinaryReplayArchiveFinalizer.java +++ b/src/main/java/me/justindevb/replay/storage/binary/BinaryReplayArchiveFinalizer.java @@ -5,7 +5,6 @@ import me.justindevb.replay.recording.TimelineEvent; import me.justindevb.replay.storage.ReplayFinalizer; import me.justindevb.replay.util.VersionUtil; -import net.jpountz.lz4.LZ4FrameOutputStream; import java.io.ByteArrayOutputStream; import java.io.IOException; @@ -66,14 +65,16 @@ public byte[] finalizeRecoveredReplay( ReplayChunkData chunkData ) throws IOException { byte[] finalizedPayload = buildFinalizedPayload(recovery.records(), recovery.timeline(), recovery.stringTable()); - byte[] compressedPayload = compress(finalizedPayload); + BinaryReplayPayloadCompression payloadCompression = BinaryReplayPayloadCompression.DEFAULT; + byte[] compressedPayload = payloadCompression.compress(finalizedPayload); ReplayChunkData effectiveChunkData = chunkData != null ? chunkData : ReplayChunkData.NONE; BinaryReplayManifest manifest = BinaryReplayManifest.createV1( pluginVersion, VersionUtil.MIN_RECORDING_VERSION, resolveRecordingStartedAtEpochMillis(recovery), crc32cHex(compressedPayload), - effectiveChunkData.metadata()); + effectiveChunkData.metadata(), + payloadCompression); byte[] manifestBytes = gson.toJson(manifest).getBytes(StandardCharsets.UTF_8); return buildArchive(manifestBytes, compressedPayload, effectiveChunkData); } @@ -181,14 +182,6 @@ private static byte[] buildIndexSection(List stringTable, List { + try (LZ4FrameOutputStream lz4 = new LZ4FrameOutputStream(out)) { + lz4.write(payload); + } + } + case ZSTD -> { + try (ZstdOutputStream zstd = new ZstdOutputStream(out, ZSTD_LEVEL)) { + zstd.write(payload); + } + } + } + return out.toByteArray(); + } + + public byte[] decompress(byte[] compressedPayload) throws IOException { + Objects.requireNonNull(compressedPayload, "compressedPayload"); + return switch (this) { + case LZ4_FRAME -> { + try (LZ4FrameInputStream lz4 = new LZ4FrameInputStream(new ByteArrayInputStream(compressedPayload))) { + yield lz4.readAllBytes(); + } + } + case ZSTD -> { + try (ZstdInputStream zstd = new ZstdInputStream(new ByteArrayInputStream(compressedPayload))) { + yield zstd.readAllBytes(); + } + } + }; + } + + public static BinaryReplayPayloadCompression fromManifestValue(String value) throws IOException { + if (value == null || value.isBlank()) { + throw new IOException("Binary replay payload compression is missing"); + } + String normalized = value.toLowerCase(Locale.ROOT); + for (BinaryReplayPayloadCompression compression : values()) { + if (compression.manifestValue.equals(normalized)) { + return compression; + } + } + throw new IOException("Unsupported binary replay payload compression: " + value); + } + + public static BinaryReplayPayloadCompression detect(byte[] compressedPayload) throws IOException { + Objects.requireNonNull(compressedPayload, "compressedPayload"); + for (BinaryReplayPayloadCompression compression : values()) { + if (startsWith(compressedPayload, compression.magicBytes)) { + return compression; + } + } + throw new IOException("Unsupported binary replay payload compression magic"); + } + + private static boolean startsWith(byte[] bytes, byte[] prefix) { + return bytes.length >= prefix.length && Arrays.equals(Arrays.copyOf(bytes, prefix.length), prefix); + } +} diff --git a/src/main/java/me/justindevb/replay/storage/binary/BinaryReplayStorageCodec.java b/src/main/java/me/justindevb/replay/storage/binary/BinaryReplayStorageCodec.java index ec5ea0c..68bfcb7 100644 --- a/src/main/java/me/justindevb/replay/storage/binary/BinaryReplayStorageCodec.java +++ b/src/main/java/me/justindevb/replay/storage/binary/BinaryReplayStorageCodec.java @@ -12,7 +12,6 @@ import me.justindevb.replay.storage.ReplaySaveRequest; import me.justindevb.replay.storage.ReplayStorageCodec; import me.justindevb.replay.util.VersionUtil; -import net.jpountz.lz4.LZ4FrameInputStream; import java.io.ByteArrayInputStream; import java.io.File; @@ -160,7 +159,7 @@ public ReplayInspection inspectReplay(String replayName, byte[] storedBytes, Str 0); } - byte[] payload = decompress(archiveEntries.replayBytes()); + byte[] payload = decompress(manifest, archiveEntries.replayBytes()); validatePayloadHeader(payload); ParsedPayload parsedPayload = parsePayload(payload); LazyTimeline timeline = new LazyTimeline(payload, parsedPayload.events(), parsedPayload.stringTable(), parsedPayload.tickIndex()); @@ -197,7 +196,7 @@ ParsedBinaryReplay openReplay(byte[] storedBytes, String runningVersion) throws BinaryReplayManifest manifest = parseManifest(archiveEntries.manifestBytes()); validateManifest(manifest, archiveEntries.replayBytes(), runningVersion); - byte[] payload = decompress(archiveEntries.replayBytes()); + byte[] payload = decompress(manifest, archiveEntries.replayBytes()); validatePayloadHeader(payload); ParsedPayload parsedPayload = parsePayload(payload); @@ -326,10 +325,11 @@ private void validateManifest(BinaryReplayManifest manifest, byte[] replayBytes, } } - private static byte[] decompress(byte[] replayBytes) throws IOException { - try (LZ4FrameInputStream lz4 = new LZ4FrameInputStream(new ByteArrayInputStream(replayBytes))) { - return lz4.readAllBytes(); - } + private static byte[] decompress(BinaryReplayManifest manifest, byte[] replayBytes) throws IOException { + BinaryReplayPayloadCompression compression = manifest.payloadCompression() != null + ? BinaryReplayPayloadCompression.fromManifestValue(manifest.payloadCompression()) + : BinaryReplayPayloadCompression.detect(replayBytes); + return compression.decompress(replayBytes); } private static void validatePayloadHeader(byte[] payload) throws IOException { diff --git a/src/main/java/me/justindevb/replay/util/VersionUtil.java b/src/main/java/me/justindevb/replay/util/VersionUtil.java index 8135b33..e664887 100644 --- a/src/main/java/me/justindevb/replay/util/VersionUtil.java +++ b/src/main/java/me/justindevb/replay/util/VersionUtil.java @@ -8,7 +8,7 @@ public final class VersionUtil { /** Minimum plugin version required to read recordings produced by this build. */ - public static final String MIN_RECORDING_VERSION = "1.5.0-alpha.1"; + public static final String MIN_RECORDING_VERSION = "1.5.0-alpha.12"; private VersionUtil() {} diff --git a/src/test/java/me/justindevb/replay/playback/ReplayChunkPlaybackCacheTest.java b/src/test/java/me/justindevb/replay/playback/ReplayChunkPlaybackCacheTest.java index 57c40f0..71a60dc 100644 --- a/src/test/java/me/justindevb/replay/playback/ReplayChunkPlaybackCacheTest.java +++ b/src/test/java/me/justindevb/replay/playback/ReplayChunkPlaybackCacheTest.java @@ -9,10 +9,8 @@ import me.justindevb.replay.storage.binary.BinaryChunkRegionCodec; import me.justindevb.replay.storage.binary.BinaryChunkRegionEntry; import me.justindevb.replay.storage.binary.BinaryReplayChunkMetadata; -import net.jpountz.lz4.LZ4FrameOutputStream; import org.junit.jupiter.api.Test; -import java.io.ByteArrayOutputStream; import java.util.List; import java.util.Map; import java.util.Optional; @@ -51,6 +49,23 @@ void loadChunk_decodesStoredChunkPayloadOnDemand() throws Exception { assertEquals(16 * 16, snapshot.payload().stateIndexes().length); } + @Test + void loadChunk_decodesZstdStoredChunkPayloadOnDemand() throws Exception { + byte[] payload = payloadCodec.encode(0, 1, List.of("minecraft:stone"), new short[16 * 16]); + byte[] compressedPayload = BinaryChunkCompression.ZSTD.compress(payload); + byte[] regionBytes = regionCodec.encode(List.of(new BinaryChunkRegionEntry(0, 0, payload.length, BinaryChunkCompression.ZSTD, compressedPayload))); + ReplayChunkData chunkData = new ReplayChunkData( + BinaryReplayChunkMetadata.present(1, 1, "abcd"), + Map.of("chunks/world/r.0.0.brregion", regionBytes)); + + ReplayChunkPlaybackCache cache = new ReplayChunkPlaybackCache(chunkData); + Optional decoded = cache.loadChunk(new ChunkCoordinate("world", 0, 0)); + + assertTrue(decoded.isPresent()); + ReplayChunkSnapshot.LegacyBlockStateSnapshot snapshot = (ReplayChunkSnapshot.LegacyBlockStateSnapshot) decoded.get(); + assertEquals("minecraft:stone", snapshot.payload().palette().getFirst()); + } + @Test void loadChunkWithDiagnostics_reportsCacheHitAfterInitialDecode() throws Exception { byte[] payload = payloadCodec.encode(0, 1, List.of("minecraft:stone"), new short[16 * 16]); @@ -120,11 +135,7 @@ void loadChunk_decodesPacketFriendlySnapshotWhenArchiveUsesBrcp() throws Excepti } private static byte[] compress(byte[] payload) throws Exception { - ByteArrayOutputStream out = new ByteArrayOutputStream(); - try (LZ4FrameOutputStream lz4 = new LZ4FrameOutputStream(out)) { - lz4.write(payload); - } - return out.toByteArray(); + return BinaryChunkCompression.LZ4_FRAME.compress(payload); } private static Logger testLogger(TestLogHandler handler) { @@ -157,4 +168,4 @@ boolean contains(Level level, String messageFragment) { && record.getMessage().contains(messageFragment)); } } -} \ No newline at end of file +} diff --git a/src/test/java/me/justindevb/replay/storage/binary/BinaryChunkRegionCodecTest.java b/src/test/java/me/justindevb/replay/storage/binary/BinaryChunkRegionCodecTest.java index abc4943..f2f0dca 100644 --- a/src/test/java/me/justindevb/replay/storage/binary/BinaryChunkRegionCodecTest.java +++ b/src/test/java/me/justindevb/replay/storage/binary/BinaryChunkRegionCodecTest.java @@ -79,6 +79,18 @@ void rejectsOverlappingPayloadRanges() { assertThrows(IOException.class, () -> codec.decode(regionBytes)); } + @Test + void rejectsUnsupportedCompressionCodecId() { + byte[] regionBytes = codec.encode(List.of( + new BinaryChunkRegionEntry(1, 1, 8, BinaryChunkCompression.LZ4_FRAME, new byte[] {0x01}) + )); + regionBytes[18] = 99; + + IOException ex = assertThrows(IOException.class, () -> codec.decode(regionBytes)); + + assertEquals("Unsupported chunk payload codec id: 99", ex.getMessage()); + } + private static int littleEndianInt(byte[] bytes, int offset) { return ByteBuffer.wrap(bytes, offset, Integer.BYTES) .order(BinaryReplayFormat.PRIMITIVE_BYTE_ORDER) @@ -90,4 +102,4 @@ private static byte[] slice(byte[] bytes, int start, int end) { System.arraycopy(bytes, start, slice, 0, slice.length); return slice; } -} \ No newline at end of file +} diff --git a/src/test/java/me/justindevb/replay/storage/binary/BinaryReplayArchiveFinalizerTest.java b/src/test/java/me/justindevb/replay/storage/binary/BinaryReplayArchiveFinalizerTest.java index daa64d8..8af9983 100644 --- a/src/test/java/me/justindevb/replay/storage/binary/BinaryReplayArchiveFinalizerTest.java +++ b/src/test/java/me/justindevb/replay/storage/binary/BinaryReplayArchiveFinalizerTest.java @@ -2,7 +2,6 @@ import com.google.gson.Gson; import me.justindevb.replay.recording.TimelineEvent; -import net.jpountz.lz4.LZ4FrameInputStream; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.io.TempDir; @@ -138,9 +137,7 @@ private static Map readArchiveEntries(byte[] archiveBytes) throw } private static byte[] decompress(byte[] compressed) throws IOException { - try (LZ4FrameInputStream in = new LZ4FrameInputStream(new ByteArrayInputStream(compressed))) { - return in.readAllBytes(); - } + return BinaryReplayPayloadCompression.detect(compressed).decompress(compressed); } private static ParsedPayload parsePayload(byte[] payload) { @@ -201,4 +198,4 @@ private record ParsedPayload(List stringTable, List entries = readArchiveEntries(archive); + BinaryReplayManifest manifest = gson.fromJson( + new String(entries.get(BinaryReplayFormat.MANIFEST_ENTRY_NAME), StandardCharsets.UTF_8), + BinaryReplayManifest.class); + + assertEquals(BinaryReplayPayloadCompression.ZSTD.manifestValue(), manifest.payloadCompression()); + assertEquals(sampleTimeline(), codec.decodeTimeline(archive, VersionUtil.MIN_RECORDING_VERSION)); + } + + @Test + void lz4TimelineArchive_stillDecodes() throws Exception { + byte[] archive = codec.finalizeReplay("lz4", sampleTimeline(), "1.5.0", RECORDING_STARTED_AT); + Map entries = readArchiveEntries(archive); + byte[] payload = decompress(entries.get(BinaryReplayFormat.REPLAY_ENTRY_NAME)); + entries.put(BinaryReplayFormat.REPLAY_ENTRY_NAME, BinaryReplayPayloadCompression.LZ4_FRAME.compress(payload)); + replaceManifestCompression(entries, BinaryReplayPayloadCompression.LZ4_FRAME.manifestValue()); + updateManifestChecksum(entries); + + assertEquals(sampleTimeline(), codec.decodeTimeline(writeArchive(entries), VersionUtil.MIN_RECORDING_VERSION)); + } + + @Test + void missingPayloadCompression_fallsBackToMagicByteDetection() throws Exception { + byte[] archive = codec.finalizeReplay("legacy-manifest-zstd", sampleTimeline(), "1.5.0", RECORDING_STARTED_AT); + Map entries = readArchiveEntries(archive); + replaceManifestCompression(entries, null); + + assertEquals(sampleTimeline(), codec.decodeTimeline(writeArchive(entries), VersionUtil.MIN_RECORDING_VERSION)); + } + + @Test + void unsupportedPayloadCompression_failsClearly() throws Exception { + byte[] archive = codec.finalizeReplay("bad-compression", sampleTimeline(), "1.5.0", RECORDING_STARTED_AT); + Map entries = readArchiveEntries(archive); + replaceManifestCompression(entries, "brotli"); + + IOException ex = assertThrows(IOException.class, + () -> codec.decodeTimeline(writeArchive(entries), VersionUtil.MIN_RECORDING_VERSION)); + + assertTrue(ex.getMessage().contains("Unsupported binary replay payload compression: brotli")); + } + @Test void rejectsReplaysThatRequireNewerViewerVersion() throws Exception { byte[] archive = codec.finalizeReplay("versioned", sampleTimeline(), "1.4.0", RECORDING_STARTED_AT); @@ -200,6 +246,9 @@ void finalizeReplay_withChunkArtifacts_includesChunkEntriesInArchive() throws Ex assertEquals(1, manifest.chunkRegionEntryCount()); assertEquals(1, manifest.chunkEntryCount()); assertTrue(entries.containsKey("chunks/world/r.0.0.brregion")); + BinaryChunkRegionCodec.DecodedBinaryChunkRegion region = new BinaryChunkRegionCodec() + .decode(entries.get("chunks/world/r.0.0.brregion")); + assertEquals(BinaryChunkCompression.ZSTD, region.indexEntries().getFirst().compression()); } } @@ -383,17 +432,22 @@ private static void writeStoredEntry(ZipOutputStream zip, String name, byte[] co } private static byte[] decompress(byte[] replayBytes) throws IOException { - try (LZ4FrameInputStream lz4 = new LZ4FrameInputStream(new ByteArrayInputStream(replayBytes))) { - return lz4.readAllBytes(); - } + return BinaryReplayPayloadCompression.detect(replayBytes).decompress(replayBytes); } private static byte[] compress(byte[] payload) throws IOException { - ByteArrayOutputStream out = new ByteArrayOutputStream(); - try (LZ4FrameOutputStream lz4 = new LZ4FrameOutputStream(out)) { - lz4.write(payload); + return BinaryReplayPayloadCompression.DEFAULT.compress(payload); + } + + private void replaceManifestCompression(Map entries, String compression) { + JsonObject manifest = JsonParser.parseString( + new String(entries.get(BinaryReplayFormat.MANIFEST_ENTRY_NAME), StandardCharsets.UTF_8)).getAsJsonObject(); + if (compression == null) { + manifest.remove("payloadCompression"); + } else { + manifest.addProperty("payloadCompression", compression); } - return out.toByteArray(); + entries.put(BinaryReplayFormat.MANIFEST_ENTRY_NAME, gson.toJson(manifest).getBytes(StandardCharsets.UTF_8)); } private void updateManifestChecksum(Map entries) { @@ -414,7 +468,8 @@ private void updateManifestChecksum(Map entries) { manifest.chunkEntryCount(), manifest.chunkCoordinateHash(), manifest.chunkPayloadFormat(), - manifest.chunkPayloadVersion()); + manifest.chunkPayloadVersion(), + manifest.payloadCompression()); entries.put(BinaryReplayFormat.MANIFEST_ENTRY_NAME, gson.toJson(updated).getBytes(StandardCharsets.UTF_8)); } }