Skip to content

Commit 0b2f5cb

Browse files
committed
fix(cli/backup): stream pg_dump to disk via Node pipeline, drop gzip subprocess
Backup creation OOM-killed Node a few seconds into pg_dump on the production-sized openmapx database: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory Cause: pgDumpToFile spawned `docker compose exec … pg_dump` via execa and piped it into a sibling `gzip -c` execa with `input: sub.stdout`. execa v9 buffers each child's stdout into the resolved result object by default — so even though gzip was consuming the bytes via the input stream, execa was simultaneously collecting the entire dump into memory for the result. Switch to a real Node stream pipeline: pg_dump (execa, buffer.stdout=false) → createGzip() → createWriteStream(outFile) via stream/promises pipeline(). Bonus: drops the separate gzip subprocess (zlib.createGzip handles it in-process). Existing 117 CLI tests still pass.
1 parent 57e1893 commit 0b2f5cb

1 file changed

Lines changed: 23 additions & 14 deletions

File tree

packages/cli/src/commands/backup.ts

Lines changed: 23 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
import {
2+
createWriteStream,
23
existsSync,
34
mkdirSync,
45
readdirSync,
@@ -8,6 +9,8 @@ import {
89
writeFileSync,
910
} from "node:fs";
1011
import { join, resolve } from "node:path";
12+
import { pipeline } from "node:stream/promises";
13+
import { createGzip } from "node:zlib";
1114
import { PLATFORM_VERSION } from "@openmapx/core";
1215
import { services as coreServices } from "@openmapx/core/server";
1316
import type { Command } from "commander";
@@ -476,9 +479,15 @@ async function pgDumpToFile(
476479
db: string,
477480
outFile: string,
478481
): Promise<void> {
479-
// `docker compose exec -T <svc> pg_dump -U <user> <db>` → gzip → outFile.
480-
// We pipe the dump through gzip ourselves rather than relying on `sh -c`
481-
// inside the container (avoids an extra shell + escaping).
482+
// `docker compose exec -T <svc> pg_dump -U <user> <db>` streamed
483+
// through Node's zlib gzip into outFile.
484+
//
485+
// Earlier this used `execa(... gzip ...)` with `input: sub.stdout`, but
486+
// execa v9 buffers each child's stdout into its result object by
487+
// default — so even though gzip was the real consumer, the dump was
488+
// also being held in memory and OOM-killed Node on multi-hundred-MB
489+
// dumps. Stream directly via createGzip() + pipeline() so nothing
490+
// touches the JS heap and we skip the extra `gzip` subprocess.
482491
const sub = execa(
483492
"docker",
484493
[
@@ -495,23 +504,23 @@ async function pgDumpToFile(
495504
"--no-privileges",
496505
db,
497506
],
498-
{ cwd: ctx.cwd, reject: false, stderr: "pipe" },
507+
{ cwd: ctx.cwd, reject: false, stderr: "pipe", buffer: { stdout: false } },
499508
);
500509

501-
const gzip = execa("gzip", ["-c"], {
502-
cwd: ctx.cwd,
503-
reject: false,
504-
input: sub.stdout ?? undefined,
505-
stdout: { file: outFile },
506-
});
510+
if (!sub.stdout) {
511+
throw new Error("pg_dump subprocess has no stdout stream");
512+
}
513+
514+
const out = createWriteStream(outFile);
515+
const gzip = createGzip();
507516

508-
const [pgRes, gzipRes] = await Promise.all([sub, gzip]);
517+
// Run the stream pipeline and the subprocess wait in parallel; both
518+
// must succeed before the dump is considered complete.
519+
const pipePromise = pipeline(sub.stdout, gzip, out);
520+
const [pgRes] = await Promise.all([sub, pipePromise]);
509521
if (pgRes.exitCode !== 0) {
510522
throw new Error(`pg_dump failed (exit ${pgRes.exitCode}): ${pgRes.stderr ?? ""}`);
511523
}
512-
if (gzipRes.exitCode !== 0) {
513-
throw new Error(`gzip failed (exit ${gzipRes.exitCode}): ${gzipRes.stderr ?? ""}`);
514-
}
515524
}
516525

517526
async function tarVolumeToFile(

0 commit comments

Comments
 (0)