-
Notifications
You must be signed in to change notification settings - Fork 276
feat: add image volume archive function for use as mount #5235
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
a1994sc
wants to merge
21
commits into
main
Choose a base branch
from
feature/image-volume-create
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
21 commits
Select commit
Hold shift + click to select a range
b6f0cb8
feat: start working on adding file to image for ivm
a1994sc 381a725
feat: working on image volume mount
a1994sc 3e2fda6
feat: add compression options
a1994sc 7ede3dd
feat: update logic for creating image
a1994sc 01aa27a
feat: wrap in internal command
a1994sc a849b0b
feat: expose imv as function
a1994sc 06b270e
Merge remote-tracking branch 'origin/main' into feature/image-volume-…
a1994sc 7026357
feat: logging
a1994sc 63f6b69
feat: update logic for tar creation
a1994sc 7d80725
fix: flatten temp tar filename by "/" not OS separator
a1994sc e1d8716
feat: add logic around output of file
a1994sc e4e37ca
ci: expand code coverage
a1994sc d86e258
Merge branch 'main' into feature/image-volume-create
a1994sc 72ac453
feat: add ticker for adding files
a1994sc cfa8c44
fix: windows tar balls
a1994sc bb6b5cb
feat: add max layers config
a1994sc 28a75eb
feat: use logging instead of fmt
a1994sc f2a1282
feat: add tab completion for iva
a1994sc 020b6af
feat: add more auto-complete
a1994sc 4328df7
feat: updates to gomod
a1994sc b54ac04
feat: add example for using archive
a1994sc File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
44 changes: 44 additions & 0 deletions
44
site/src/content/docs/commands/zarf_dev_image-volume-archive.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| --- | ||
| title: zarf dev image-volume-archive | ||
| description: Zarf CLI command reference for <code>zarf dev image-volume-archive</code>. | ||
| tableOfContents: false | ||
| --- | ||
|
|
||
| <!-- Page generated by Zarf; DO NOT EDIT --> | ||
|
|
||
| ## zarf dev image-volume-archive | ||
|
|
||
| Creates an image archive from a given directory with a given image reference | ||
|
|
||
| ``` | ||
| zarf dev image-volume-archive [ DIRECTORY ] [ IMAGE-REFERENCE ] [flags] | ||
| ``` | ||
|
|
||
| ### Options | ||
|
|
||
| ``` | ||
| -h, --help help for image-volume-archive | ||
| -c, --layer-compression string Compression algorthm used on the individual layers of the image volume (default "gzip") | ||
| -m, --max-layers uint8 Cap on the number of layers in the resulting image; files are batched across fewer layers as needed to stay under it. 0 disables the cap. (default 127) | ||
| -O, --output string Path to write the resulting tar archive to (default derived from IMAGE-REFERENCE) | ||
| -o, --platform-os string Operating system of the image volume (default "linux") | ||
| ``` | ||
|
|
||
| ### Options inherited from parent commands | ||
|
|
||
| ``` | ||
| -a, --architecture string Architecture for OCI images and Zarf packages | ||
| --cache string Specify the location of the Zarf cache directory (default "~/.zarf-cache") | ||
| --features stringToString Provide a comma-separated list of feature names to bools to enable or disable. Ex. --features "foo=true,bar=false,baz=true" (default []) | ||
| --insecure-skip-tls-verify Skip checking server's certificate for validity. This flag should only be used if you have a specific reason and accept the reduced security posture. | ||
| --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console") | ||
| -l, --log-level string Log level when running Zarf. Valid options are: warn, info, debug, trace (default "info") | ||
| --no-color Disable terminal color codes in logging and stdout prints. | ||
| --plain-http Allow OCI registry connections over HTTP instead of HTTPS. This flag should only be used if you have a specific reason and accept the reduced security posture. | ||
| --tmpdir string Specify the temporary directory to use for intermediate files | ||
| ``` | ||
|
|
||
| ### SEE ALSO | ||
|
|
||
| * [zarf dev](/commands/zarf_dev/) - Commands useful for developing packages | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
| // SPDX-FileCopyrightText: 2021-Present The Zarf Authors | ||
|
|
||
| package cobra | ||
|
|
||
| import ( | ||
| "fmt" | ||
|
|
||
| "github.com/zarf-dev/zarf/src/pkg/zoci/image" | ||
| ) | ||
|
|
||
| // gzipCobraDesc, zstdCobraDesc, and uncompressedCobraDesc are the shell | ||
| // completion descriptions shown alongside each VolumeCompression value. | ||
| const ( | ||
| gzipCobraDesc = "gzip-compressed layers" | ||
| zstdCobraDesc = "zstd-compressed layers" | ||
| uncompressedCobraDesc = "uncompressed layers" | ||
| ) | ||
|
|
||
| // GetDevIVACobraCompression returns the valid VolumeCompression values as | ||
| // "value\tdescription" pairs, ready to return from a cobra | ||
| // RegisterFlagCompletionFunc callback for shell tab completion. | ||
| func GetDevIVACobraCompression() []string { | ||
| return []string{ | ||
| fmt.Sprintf("%s\t%s", string(image.VolumeCompressionGzip), gzipCobraDesc), | ||
| fmt.Sprintf("%s\t%s", string(image.VolumeCompressionZstd), zstdCobraDesc), | ||
| fmt.Sprintf("%s\t%s", string(image.VolumeCompressionUncompressed), uncompressedCobraDesc), | ||
| } | ||
| } | ||
|
|
||
| // unlimitedMaxLayersDesc and defaultMaxLayersDesc are the shell completion | ||
| // descriptions shown alongside the suggested MaxLayers values. | ||
| const ( | ||
| unlimitedMaxLayersDesc = "unlimited (disables the cap)" | ||
| defaultMaxLayersDesc = "default cap" | ||
| ) | ||
|
|
||
| // GetDevIVACobraMaxLayers returns a short list of sensible MaxLayers values as | ||
| // "value\tdescription" pairs, ready to return from a cobra | ||
| // RegisterFlagCompletionFunc callback for shell tab completion. Unlike | ||
| // GetCobraCompression/GetCobraPlatformOS this isn't an exhaustive set of | ||
| // valid values - MaxLayers accepts any uint8 - just the ones worth | ||
| // suggesting. | ||
| func GetDevIVACobraMaxLayers() []string { | ||
| return []string{ | ||
| fmt.Sprintf("%d\t%s", image.UnlimiteLayers, unlimitedMaxLayersDesc), | ||
| fmt.Sprintf("%d\t%s", image.DefaultMaxLayers, defaultMaxLayersDesc), | ||
| } | ||
| } | ||
|
|
||
| // osLinuxCobraDesc and osWindowCobraDesc are the shell completion | ||
| // descriptions shown alongside each PlatformOS value. | ||
| const ( | ||
| osLinuxCobraDesc = "linux image volume" | ||
| osWindowCobraDesc = "windows image volume" | ||
| ) | ||
|
|
||
| // GetDevIVACobraPlatformOS returns the valid PlatformOS values as | ||
| // "value\tdescription" pairs, ready to return from a cobra | ||
| // RegisterFlagCompletionFunc callback for shell tab completion. | ||
| func GetDevIVACobraPlatformOS() []string { | ||
| return []string{ | ||
| fmt.Sprintf("%s\t%s", string(image.PlatformOSLinux), osLinuxCobraDesc), | ||
| fmt.Sprintf("%s\t%s", string(image.PlatformOSWindows), osWindowCobraDesc), | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
| // SPDX-FileCopyrightText: 2021-Present The Zarf Authors | ||
|
|
||
| // Package cobra holds shell tab-completion helpers shared by Zarf's cobra | ||
| // commands, returning "value\tdescription" pairs for use in | ||
| // cobra.RegisterFlagCompletionFunc callbacks. | ||
| package cobra | ||
|
|
||
| import ( | ||
| "fmt" | ||
|
|
||
| "github.com/zarf-dev/zarf/src/config" | ||
| ) | ||
|
|
||
| // osArchAMD64Desc and osArchARM64Desc are the shell completion descriptions | ||
| // shown alongside each supported --architecture value. | ||
| const ( | ||
| osArchAMD64Desc = "the x86-64, 64-bit AMD, architecture" | ||
| osArchARM64Desc = "the 64-bit ARM architecture" | ||
| ) | ||
|
|
||
| // GetRootArchitectureCobraCompression returns the valid --architecture | ||
| // values as "value\tdescription" pairs, ready to return from a cobra | ||
| // RegisterFlagCompletionFunc callback for shell tab completion. | ||
| func GetRootArchitectureCobraCompression() []string { | ||
| return []string{ | ||
| fmt.Sprintf("%s\t%s", string(config.OSArchAMD64), osArchAMD64Desc), | ||
| fmt.Sprintf("%s\t%s", string(config.OSArchARM64), osArchARM64Desc), | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,145 @@ | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
| // SPDX-FileCopyrightText: 2021-Present The Zarf Authors | ||
|
|
||
| // Package dev contains dev Zarf CLI commands. | ||
| package dev | ||
|
|
||
| import ( | ||
| "errors" | ||
| "os" | ||
|
|
||
| "github.com/spf13/cobra" | ||
| zcobra "github.com/zarf-dev/zarf/src/cmd/cobra" | ||
| "github.com/zarf-dev/zarf/src/config" | ||
| "github.com/zarf-dev/zarf/src/config/lang" | ||
| "github.com/zarf-dev/zarf/src/pkg/logger" | ||
| "github.com/zarf-dev/zarf/src/pkg/utils" | ||
| "github.com/zarf-dev/zarf/src/pkg/zoci/archive" | ||
| "github.com/zarf-dev/zarf/src/pkg/zoci/image" | ||
| ) | ||
|
|
||
| const ( | ||
| flagLayerCompression = "layer-compression" | ||
| flagPlatformOS = "platform-os" | ||
| flagMaxLayer = "max-layers" | ||
| ) | ||
|
|
||
| type imageVolumeOptions struct { | ||
| compression image.VolumeCompression | ||
| os image.PlatformOS | ||
| output string | ||
| maxLayers uint8 | ||
| } | ||
|
|
||
| // NewImageVolumeCommand creates the command to build an image volume | ||
| // archive from a directory of files. | ||
| func NewImageVolumeCommand() *cobra.Command { | ||
| o := &imageVolumeOptions{} | ||
|
|
||
| cmd := &cobra.Command{ | ||
| Use: lang.CmdDevImageVolumeArchiveUsage, | ||
| Aliases: []string{"iva"}, | ||
| Short: lang.CmdDevImageVolumeArchiveShort, | ||
| Args: cobra.ExactArgs(2), | ||
| PreRunE: o.prerun, | ||
| RunE: o.run, | ||
| } | ||
|
|
||
| cmd.Flags().StringVarP((*string)(&o.compression), flagLayerCompression, "c", string(image.VolumeCompressionGzip), lang.CmdDevImageVolumeArchiveFlagCompression) | ||
| cmd.Flags().StringVarP((*string)(&o.os), flagPlatformOS, "o", string(image.PlatformOSLinux), lang.CmdDevImageVolumeArchiveFlagPlatformOS) | ||
| cmd.Flags().StringVarP(&o.output, "output", "O", "", lang.CmdDevImageVolumeArchiveFlagOutput) | ||
| cmd.Flags().Uint8VarP(&o.maxLayers, flagMaxLayer, "m", image.DefaultMaxLayers, lang.CmdDevImageVolumeArchiveFlagMaxLayer) | ||
|
|
||
| if err := cmd.RegisterFlagCompletionFunc(flagLayerCompression, func(_ *cobra.Command, _ []string, _ string) ([]string, cobra.ShellCompDirective) { | ||
| return zcobra.GetDevIVACobraCompression(), cobra.ShellCompDirectiveNoFileComp | ||
| }); err != nil { | ||
| logger.From(cmd.Context()).Warn("failed to register out-complete", "error", err) | ||
| panic(err) | ||
| } | ||
|
|
||
| if err := cmd.RegisterFlagCompletionFunc(flagPlatformOS, func(_ *cobra.Command, _ []string, _ string) ([]string, cobra.ShellCompDirective) { | ||
| return zcobra.GetDevIVACobraPlatformOS(), cobra.ShellCompDirectiveNoFileComp | ||
| }); err != nil { | ||
| logger.From(cmd.Context()).Warn("failed to register out-complete", "error", err) | ||
| panic(err) | ||
| } | ||
|
|
||
| if err := cmd.RegisterFlagCompletionFunc(flagMaxLayer, func(_ *cobra.Command, _ []string, _ string) ([]string, cobra.ShellCompDirective) { | ||
| return zcobra.GetDevIVACobraMaxLayers(), cobra.ShellCompDirectiveNoFileComp | ||
| }); err != nil { | ||
| logger.From(cmd.Context()).Warn("failed to register out-complete", "error", err) | ||
| panic(err) | ||
| } | ||
|
|
||
| return cmd | ||
| } | ||
|
|
||
| // prerun validates the compression format in the image volume options, | ||
| // defaulting to uncompressed if not specified. | ||
| // | ||
| // _ *cobra.Command, args []string. | ||
| // error. | ||
| func (o *imageVolumeOptions) prerun(_ *cobra.Command, args []string) error { | ||
| if o.output == "" { | ||
| o.output = archive.ImageRefToTar(args[1]) | ||
| } | ||
| return errors.Join( | ||
| archive.ValidateFileEndsWithTar(o.output), | ||
| image.ValidateCompression(o.compression), | ||
| image.ValidatePlatformOS(o.os), | ||
| image.ValidatePlatformArch(image.PlatformArch(config.GetArch())), | ||
| ) | ||
| } | ||
|
|
||
| // run builds an image volume from args[0] (the source directory) tagged as | ||
| // args[1] (the image reference), then writes it to o.output as a | ||
| // Docker/OCI-compatible tar archive. | ||
| // | ||
| // cmd *cobra.Command, args []string. | ||
| // error. | ||
| func (o *imageVolumeOptions) run(cmd *cobra.Command, args []string) error { | ||
| dir := args[0] | ||
| ref := args[1] | ||
|
|
||
| tmpDir, err := utils.MakeTempDir(config.CommonOptions.TempDirectory) | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| iv, err := image.New(tmpDir, string(o.os), config.GetArch()) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| defer func() { | ||
| if err := iv.Clean(); err != nil { | ||
| logger.From(cmd.Context()).Debug("failed to clean image volume workspace", "error", err) | ||
| } | ||
| if err := os.RemoveAll(tmpDir); err != nil { | ||
| logger.From(cmd.Context()).Debug("failed to remove staging directory", "error", err) | ||
| } | ||
| }() | ||
|
|
||
| iv.Compression = o.compression | ||
| iv.MaxLayers = o.maxLayers | ||
|
|
||
| if err := iv.AddDirectory(cmd.Context(), dir, ref); err != nil { | ||
| return err | ||
| } | ||
|
|
||
| out, err := os.Create(o.output) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| defer func() { | ||
| if closeErr := out.Close(); closeErr != nil { | ||
| logger.From(cmd.Context()).Debug("failed to close image volume archive", "error", closeErr) | ||
| } | ||
| }() | ||
|
|
||
| if err := iv.WriteTar(cmd.Context(), ref, out); err != nil { | ||
| return err | ||
| } | ||
|
|
||
| logger.From(cmd.Context()).Info("wrote image volume archive", "path", o.output) | ||
| return nil | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Realized that this is a bad function name.... I will update it soon