diff --git a/CHANGELOG.md b/CHANGELOG.md index eb86e14..e77b922 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- The tar write functions (`tarDirInc`, `tarDirExc`, `tarPack`) now accept a + dictionary destination `{path: str|path, compress?: bool}`, where `compress` + overrides the extension-based gzip inference in either direction — useful + for destinations without a meaningful extension (e.g. `redo`'s `$3` temp files). + - Stream merge redirects `2>&1` (stderr to stdout's destination) and `1>&2` (stdout to stderr's destination). Each is a single token with no internal spaces, and works on command lists, pipeline stages, and quotations. diff --git a/doc/functions.inc.html b/doc/functions.inc.html index 6655dc0..7800bba 100644 --- a/doc/functions.inc.html +++ b/doc/functions.inc.html @@ -126,10 +126,10 @@

File and Directory zipExtract Extract an entire archive. Options dict is required; defaults: overwrite=false, skipExisting=false (mutually exclusive), stripComponents=0, pattern="" (glob matched before stripping), preservePermissions=true, maxBytes=0 (0 = unlimited; a cap on the total uncompressed bytes written, guarding against decompression bombs). Destination is created if missing. (path:zipPath path:destDir dict:options -- ) zipExtractEntry Extract a single entry (file or directory subtree) to a destination path. Options dict is required; defaults: overwrite=false, skipExisting=false (mutually exclusive), preservePermissions=true, mkdirs=true (create parent directories when needed), maxBytes=0 (0 = unlimited uncompressed-byte cap). (path:zipPath str:entry path:dest dict:options -- ) zipRead Read an entry’s bytes directly into the stack without writing to disk. Returns none when the entry does not exist. (path:zipPath str:entry -- Maybe[binary]) - Tar functions mirror the zip* surface (same argument order and option dicts). Compression is chosen from the destination extension when writing (.tar.gz / .tgz → gzip, .tar → uncompressed) and auto-detected from the gzip magic bytes when reading. Symlinks are preserved: packed as symlink entries and recreated on extraction (with an escape guard); hard links and device nodes are rejected. - tarDirInc Create/overwrite a .tar / .tar.gz from a directory; the archive root contains the directory’s contents (no parent folder). (path:sourceDir path:tarPath -- ) - tarDirExc Create/overwrite a .tar / .tar.gz that includes the source directory itself at the archive root (entries are prefixed with the directory name). (path:sourceDir path:tarPath -- ) - tarPack Create/overwrite a .tar / .tar.gz by packing a list of entries. Each entry is either a bare string/path (the file or directory to add, keeping its base name and mode) or a dictionary requiring path; in the dictionary form archivePath (override the in-archive name) and mode are optional. mode is a Go os.FileMode; write it with an octal literal, e.g. 0o644 (rw-r--r--), 0o755 (rwxr-xr-x), 0o600. If mode is omitted, the entry keeps the source file’s own mode. ([str|path|dict] path:tarPath -- ) + Tar functions mirror the zip* surface (same argument order and option dicts). Compression is chosen from the destination extension when writing (.tar.gz / .tgz → gzip, .tar → uncompressed) and auto-detected from the gzip magic bytes when reading. The write destination may also be a dictionary {path: str|path, compress?: bool}, where compress overrides the extension inference in either direction — useful for destinations without a meaningful extension (e.g. redo’s $3 temp files). Symlinks are preserved: packed as symlink entries and recreated on extraction (with an escape guard); hard links and device nodes are rejected. + tarDirInc Create/overwrite a .tar / .tar.gz from a directory; the archive root contains the directory’s contents (no parent folder). The destination may be a dictionary {path, compress?} to force compression on or off. (path:sourceDir path|dict:dest -- ) + tarDirExc Create/overwrite a .tar / .tar.gz that includes the source directory itself at the archive root (entries are prefixed with the directory name). The destination may be a dictionary {path, compress?} to force compression on or off. (path:sourceDir path|dict:dest -- ) + tarPack Create/overwrite a .tar / .tar.gz by packing a list of entries. Each entry is either a bare string/path (the file or directory to add, keeping its base name and mode) or a dictionary requiring path; in the dictionary form archivePath (override the in-archive name) and mode are optional. mode is a Go os.FileMode; write it with an octal literal, e.g. 0o644 (rw-r--r--), 0o755 (rwxr-xr-x), 0o600. If mode is omitted, the entry keeps the source file’s own mode. The destination may be a dictionary {path, compress?} to force compression on or off. ([str|path|dict] path|dict:dest -- ) tarList List archive entries as dictionaries with keys: name (string, forward-slash paths, directories end with /), compressedSize and uncompressedSize (int bytes; equal, since tar has no per-entry compressed size), isDir (bool), perm (int POSIX permission bits), executable (bool), modified (datetime), type ("file"/"dir"/"symlink"), and linkTarget (symlink target, empty otherwise). (path -- [dict]) tarExtract Extract an entire archive. Options dict is required; defaults: overwrite=false, skipExisting=false (mutually exclusive), stripComponents=0, pattern="" (glob matched before stripping), preservePermissions=true, maxBytes=0 (0 = unlimited; a cap on the total uncompressed bytes written, guarding against decompression bombs). Destination is created if missing. (path:tarPath path:destDir dict:options -- ) tarExtractEntry Extract a single entry (file or directory subtree) to a destination path. Options dict is required; defaults: overwrite=false, skipExisting=false (mutually exclusive), preservePermissions=true, mkdirs=true (create parent directories when needed), maxBytes=0 (0 = unlimited uncompressed-byte cap). (path:tarPath str:entry path:dest dict:options -- ) diff --git a/doc/mshell.md b/doc/mshell.md index 3e87699..c5fd11a 100644 --- a/doc/mshell.md +++ b/doc/mshell.md @@ -1484,6 +1484,11 @@ Two things differ, both driven by the tar format: `.tar.gz` or `.tgz` produce a gzip-compressed tarball, `.tar` is uncompressed. When reading, the gzip magic bytes are auto-detected, so a gzipped tarball is read transparently regardless of its filename. + The write destination (`tarDirInc`/`tarDirExc`/`tarPack`) may also be a dictionary + `{path: str|path, compress?: bool}`, where `compress` overrides the extension + inference in either direction. + This is useful for destinations without a meaningful extension, + e.g. `redo`'s `$3` temp files: `` `src` { "path": @dest, "compress": true } tarDirExc ``. - Symlinks are preserved: `tarPack`/`tarDir*` store symlinks as symlink entries, and `tarExtract`/`tarExtractEntry` recreate them (rejecting any whose target would escape the destination directory). Extraction also refuses to write @@ -1495,8 +1500,8 @@ bytes; `0` = unlimited) to guard against decompression bombs, and packing never follows a source symlink, so a symlink loop or a link to `/dev/zero` cannot hang or inflate the archive. -- `tarDirInc`: Create/overwrite a `.tar`/`.tar.gz` from a directory; the archive root contains the directory's contents (no parent folder). `(path:sourceDir path:tarPath -- )` -- `tarDirExc`: Create/overwrite a `.tar`/`.tar.gz` that includes the source directory itself at the archive root (entries are prefixed with the directory name). `(path:sourceDir path:tarPath -- )` +- `tarDirInc`: Create/overwrite a `.tar`/`.tar.gz` from a directory; the archive root contains the directory's contents (no parent folder). `(str | path str | path | {path: str | path, compress?: bool} -- )` +- `tarDirExc`: Create/overwrite a `.tar`/`.tar.gz` that includes the source directory itself at the archive root (entries are prefixed with the directory name). `(str | path str | path | {path: str | path, compress?: bool} -- )` - `tarPack`: Create/overwrite a `.tar`/`.tar.gz` by packing a list of entries. Each entry is either a bare string/path (the file or directory to add, keeping its base name and mode) or a dictionary. @@ -1505,7 +1510,7 @@ or inflate the archive. `mode` is a Go `os.FileMode`; write it with an octal literal, e.g. `0o644` (`rw-r--r--`), `0o755` (`rwxr-xr-x`), `0o600`. If `mode` is omitted, the entry keeps the source file's own mode. - Type: `([str | path | {path: str | path, archivePath?: str | path, mode?: int}] str | path -- )` + Type: `([str | path | {path: str | path, archivePath?: str | path, mode?: int}] str | path | {path: str | path, compress?: bool} -- )` - `tarList`: List archive entries as dictionaries with keys: `name` (string, forward-slash paths, directories end with `/`), `compressedSize` and `uncompressedSize` (int bytes; equal, since tar has no per-entry compressed size), `isDir` (bool), `perm` (int POSIX permission bits), `executable` (bool), `modified` (datetime from the archive entry), `type` (`"file"`/`"dir"`/`"symlink"`), and `linkTarget` (symlink target, empty otherwise). `(path -- [dict])` - `tarExtract`: Extract an entire archive. Options dict is required; defaults: `overwrite=false`, `skipExisting=false` (mutually exclusive), `stripComponents=0`, `pattern=""` (glob matched before stripping), `preservePermissions=true`, `maxBytes=0` (0 = unlimited; caps the total uncompressed bytes written to guard against decompression bombs). Destination is created if missing. `(path:tarPath path:destDir dict:options -- )` - `tarExtractEntry`: Extract a single entry (file or directory subtree) to a destination path. Options dict is required; defaults: `overwrite=false`, `skipExisting=false` (mutually exclusive), `preservePermissions=true`, `mkdirs=true`, `maxBytes=0` (0 = unlimited uncompressed-byte cap). `(path:tarPath str:entry path:dest dict:options -- )` diff --git a/mshell/Evaluator.go b/mshell/Evaluator.go index dd151db..cdde5b3 100644 --- a/mshell/Evaluator.go +++ b/mshell/Evaluator.go @@ -5279,6 +5279,36 @@ func parseZipExtractEntryOptions(dict *MShellDict) (zipExtractEntryOptions, erro return options, nil } +// parseTarDestination interprets the destination argument of the tar write +// builtins. A plain string/path infers gzip compression from the extension +// (isGzipTarget); a dict form {path, compress?} makes the choice explicit, +// overriding the extension in either direction. +func parseTarDestination(obj MShellObject) (string, bool, error) { + if dict, ok := obj.(*MShellDict); ok { + pathObj, ok := dict.Items["path"] + if !ok { + return "", false, fmt.Errorf("destination dict is missing required 'path'") + } + tarPath, err := pathObj.CastString() + if err != nil { + return "", false, fmt.Errorf("destination 'path' must be a string or path, found %s", pathObj.TypeName()) + } + compress := isGzipTarget(tarPath) + if val, ok, err := boolOption(dict, "compress"); err != nil { + return "", false, err + } else if ok { + compress = val + } + return tarPath, compress, nil + } + + tarPath, err := obj.CastString() + if err != nil { + return "", false, fmt.Errorf("Cannot tar into a %s", obj.TypeName()) + } + return tarPath, isGzipTarget(tarPath), nil +} + func boolOption(dict *MShellDict, key string) (bool, bool, error) { item, ok := dict.Items[key] if !ok { @@ -7853,9 +7883,9 @@ func (state *EvalState) evaluateToken(t Token, stack *MShellStack, context Execu return state.FailWithMessage(err.Error()) } - tarPath, err := obj1.CastString() + tarPath, compress, err := parseTarDestination(obj1) if err != nil { - return state.FailWithMessage(fmt.Sprintf("%d:%d: Cannot tar into a %s.\n", t.Line, t.Column, obj1.TypeName())) + return state.FailWithMessage(fmt.Sprintf("%d:%d: %s: %s\n", t.Line, t.Column, t.Lexeme, err.Error())) } sourceDir, err := obj2.CastString() @@ -7864,7 +7894,7 @@ func (state *EvalState) evaluateToken(t Token, stack *MShellStack, context Execu } preserveRoot := t.Lexeme == "tarDirInc" - if err := tarDirectory(sourceDir, tarPath, preserveRoot); err != nil { + if err := tarDirectory(sourceDir, tarPath, preserveRoot, compress); err != nil { return state.FailWithMessage(fmt.Sprintf("%d:%d: %s\n", t.Line, t.Column, err.Error())) } } else if t.Lexeme == "tarPack" { @@ -7873,9 +7903,9 @@ func (state *EvalState) evaluateToken(t Token, stack *MShellStack, context Execu return state.FailWithMessage(err.Error()) } - tarPath, err := obj1.CastString() + tarPath, compress, err := parseTarDestination(obj1) if err != nil { - return state.FailWithMessage(fmt.Sprintf("%d:%d: Cannot tar into a %s.\n", t.Line, t.Column, obj1.TypeName())) + return state.FailWithMessage(fmt.Sprintf("%d:%d: tarPack: %s\n", t.Line, t.Column, err.Error())) } list, ok := obj2.(*MShellList) @@ -7928,7 +7958,7 @@ func (state *EvalState) evaluateToken(t Token, stack *MShellStack, context Execu entries = append(entries, packItem) } - if err := buildTarFromEntries(entries, tarPath); err != nil { + if err := buildTarFromEntries(entries, tarPath, compress); err != nil { return state.FailWithMessage(fmt.Sprintf("%d:%d: %s\n", t.Line, t.Column, err.Error())) } } else if t.Lexeme == "tarList" { diff --git a/mshell/Tar.go b/mshell/Tar.go index d3a9567..057d566 100644 --- a/mshell/Tar.go +++ b/mshell/Tar.go @@ -73,10 +73,10 @@ func openTarReader(tarPath string) (*tar.Reader, io.Closer, error) { return tar.NewReader(br), file, nil } -// createTarWriter creates a tar archive for writing, gzip-compressing when the -// destination extension calls for it. The returned finish function must be -// called (not deferred with a discarded error) to flush and close every layer. -func createTarWriter(tarPath string) (*tar.Writer, func() error, error) { +// createTarWriter creates a tar archive for writing, gzip-compressing when +// compress is true. The returned finish function must be called (not deferred +// with a discarded error) to flush and close every layer. +func createTarWriter(tarPath string, compress bool) (*tar.Writer, func() error, error) { if err := os.MkdirAll(filepath.Dir(tarPath), 0755); err != nil { return nil, nil, fmt.Errorf("Error creating parent directory for %s: %w", tarPath, err) } @@ -86,7 +86,7 @@ func createTarWriter(tarPath string) (*tar.Writer, func() error, error) { return nil, nil, fmt.Errorf("Error creating %s: %w", tarPath, err) } - if isGzipTarget(tarPath) { + if compress { gz := gzip.NewWriter(output) tw := tar.NewWriter(gz) finish := func() error { @@ -117,7 +117,7 @@ func createTarWriter(tarPath string) (*tar.Writer, func() error, error) { // tarDirectory packs a single directory into a tarball, mirroring zipDirectory. // preserveRoot controls whether the directory itself appears at the archive // root (tarDirExc) or only its contents (tarDirInc). -func tarDirectory(sourceDir, tarPath string, preserveRoot bool) error { +func tarDirectory(sourceDir, tarPath string, preserveRoot bool, compress bool) error { info, err := os.Stat(sourceDir) if err != nil { return fmt.Errorf("Error stating %s: %w", sourceDir, err) @@ -138,7 +138,7 @@ func tarDirectory(sourceDir, tarPath string, preserveRoot bool) error { SourcePath: sourceDir, PreserveRoot: preserveRoot, } - return buildTarFromEntries([]zipPackItem{packItem}, tarPath) + return buildTarFromEntries([]zipPackItem{packItem}, tarPath, compress) } func ensureTarTargetNotInsideSource(sourceAbs string, tarPath string) error { @@ -155,7 +155,7 @@ func ensureTarTargetNotInsideSource(sourceAbs string, tarPath string) error { // buildTarFromEntries mirrors buildZipFromEntries, reusing the zipPackItem // model so the tarPack dispatch parsing is identical to zipPack. -func buildTarFromEntries(items []zipPackItem, tarPath string) error { +func buildTarFromEntries(items []zipPackItem, tarPath string, compress bool) error { if len(items) == 0 { return fmt.Errorf("tarPack requires at least one entry") } @@ -165,7 +165,7 @@ func buildTarFromEntries(items []zipPackItem, tarPath string) error { return fmt.Errorf("Error resolving %s: %w", tarPath, err) } - tw, finish, err := createTarWriter(tarPath) + tw, finish, err := createTarWriter(tarPath, compress) if err != nil { return err } diff --git a/mshell/Tar_test.go b/mshell/Tar_test.go index 83878a8..1a94841 100644 --- a/mshell/Tar_test.go +++ b/mshell/Tar_test.go @@ -78,7 +78,7 @@ func TestTarRoundTripAndGzipAutodetect(t *testing.T) { for _, name := range []string{"out.tar", "out.tar.gz"} { archive := filepath.Join(dir, name) - if err := buildTarFromEntries([]zipPackItem{{SourcePath: src, PreserveRoot: true}}, archive); err != nil { + if err := buildTarFromEntries([]zipPackItem{{SourcePath: src, PreserveRoot: true}}, archive, isGzipTarget(archive)); err != nil { t.Fatalf("%s pack: %v", name, err) } entries, err := collectTarMetadata(archive) diff --git a/mshell/Tar_unix_test.go b/mshell/Tar_unix_test.go index a0ee107..2346eb3 100644 --- a/mshell/Tar_unix_test.go +++ b/mshell/Tar_unix_test.go @@ -29,7 +29,7 @@ func TestTarPackRejectsFifoWithoutHanging(t *testing.T) { archive := filepath.Join(dir, "out.tar") done := make(chan error, 1) go func() { - done <- buildTarFromEntries([]zipPackItem{{SourcePath: src, PreserveRoot: true}}, archive) + done <- buildTarFromEntries([]zipPackItem{{SourcePath: src, PreserveRoot: true}}, archive, isGzipTarget(archive)) }() select { diff --git a/mshell/TypeBuiltins.go b/mshell/TypeBuiltins.go index 1523fbd..bdbff58 100644 --- a/mshell/TypeBuiltins.go +++ b/mshell/TypeBuiltins.go @@ -611,11 +611,15 @@ func builtinSigsByName(arena *TypeArena, names *NameTable) map[NameId][]QuoteSig // Tar ops mirror the zip surface exactly (same argument order and option // dicts). Compression is chosen from the destination extension on write // (.tar.gz / .tgz -> gzip) and sniffed from the gzip magic bytes on read. + // The write destination also accepts a dict form {path, compress?} that + // overrides the extension inference (for extensionless targets like redo's + // $3 temp files). + tarDest := "str | path | {path: str | path, compress?: bool}" r.reg("tarRead", "(str | path str | path -- Maybe[bytes])") for _, name := range []string{"tarDirInc", "tarDirExc"} { - r.reg(name, "(str | path str | path -- )") + r.reg(name, "(str | path "+tarDest+" -- )") } - r.reg("tarPack", "([str | path | {path: str | path, archivePath?: str | path, mode?: int}] str | path -- )") + r.reg("tarPack", "([str | path | {path: str | path, archivePath?: str | path, mode?: int}] "+tarDest+" -- )") r.reg("tarExtract", "(str | path str | path "+zipExtractOpts+" -- )") r.reg("tarExtractEntry", "(str str str "+zipEntryOpts+" -- )", diff --git a/tests/success/tar_compress_option.msh b/tests/success/tar_compress_option.msh new file mode 100644 index 0000000..882b0ce --- /dev/null +++ b/tests/success/tar_compress_option.msh @@ -0,0 +1,39 @@ +# Destination dict form {path, compress?}: explicit compress overrides +# extension inference (for extensionless targets like redo's $3 temp files). + +# Extensionless temp file, forced gzip. +tempFile gzTar! +`zip/base` { "path": @gzTar, "compress": true } tarDirExc + +"forced gzip magic" wl +@gzTar readFileBytes str "1f8b" startsWith str wl + +"forced gzip alpha" wl +@gzTar "alpha.txt" tarRead ? utf8Str wl + +# .tar.gz-named file, forced uncompressed. +".tar.gz" tempFileExt plainTar! +`zip/base` { "path": @plainTar, "compress": false } tarDirExc + +"forced plain magic" wl +@plainTar readFileBytes str "1f8b" startsWith str wl + +# Dict without compress falls back to extension inference (extensionless -> plain). +tempFile defTar! +[`zip/base/alpha.txt`] { "path": @defTar } tarPack + +"default plain magic" wl +@defTar readFileBytes str "1f8b" startsWith str wl + +"tarPack alpha" wl +@defTar "alpha.txt" tarRead ? utf8Str wl + +# tarPack with forced gzip on an extensionless path. +tempFile gzPack! +[`zip/base/alpha.txt`] { "path": @gzPack, "compress": true } tarPack + +"tarPack gzip magic" wl +@gzPack readFileBytes str "1f8b" startsWith str wl + +"tarPack gzip alpha" wl +@gzPack "alpha.txt" tarRead ? utf8Str wl diff --git a/tests/success/tar_compress_option.msh.stdout b/tests/success/tar_compress_option.msh.stdout new file mode 100644 index 0000000..3f54d4d --- /dev/null +++ b/tests/success/tar_compress_option.msh.stdout @@ -0,0 +1,17 @@ +forced gzip magic +true +forced gzip alpha +ALPHA-DATA + +forced plain magic +false +default plain magic +false +tarPack alpha +ALPHA-DATA + +tarPack gzip magic +true +tarPack gzip alpha +ALPHA-DATA + diff --git a/tests/typecheck_fail/tar_dest_missing_path.msh b/tests/typecheck_fail/tar_dest_missing_path.msh new file mode 100644 index 0000000..aa90918 --- /dev/null +++ b/tests/typecheck_fail/tar_dest_missing_path.msh @@ -0,0 +1,3 @@ +# The dict destination form requires `path`; a dict with only the optional +# `compress` must be rejected. +`zip/base` { 'compress': true } tarDirExc