Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 46 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -322,12 +322,13 @@ tasks:

### Includes

The `includes` key is used to import tasks from either local or remote task files. This is useful for sharing common tasks across multiple task files. When importing a task from a local task file, the path is relative to the file you are currently in. When running a task, the tasks in the task file as well as the `includes` get processed to ensure there are no infinite loop references.
The `includes` key is used to import tasks from local, remote, or OCI task files. This is useful for sharing common tasks across multiple task files. When importing a task from a local task file, the path is relative to the file you are currently in. When running a task, the tasks in the task file as well as the `includes` get processed to ensure there are no infinite loop references.

```yaml
includes:
- local: ./path/to/tasks-to-import.yaml
- remote: https://raw.githubusercontent.com/defenseunicorns/maru-runner/main/src/test/tasks/remote-import-tasks.yaml
- common: oci://ghcr.io/myorg/tasks:latest

tasks:
- name: import-local
Expand All @@ -336,6 +337,9 @@ tasks:
- name: import-remote
actions:
- task: remote:echo-var
- name: import-oci
actions:
- task: common:hello-world
```

Note that included task files can also include other task files, with the following restriction:
Expand All @@ -352,6 +356,47 @@ run import-local
run local:some-local-task
```

#### OCI Task Files

Maru supports using OCI artifacts as task files. This allows you to store your tasks in container registries and version them using tags.

**Pushing Tasks to OCI Registries**

You can push your task files to OCI registries using the built-in `push` command:

```bash
# Login to your registry first
gh auth token | maru auth login ghcr.io --token-stdin

# Push the task file
maru push hello.yaml ghcr.io/myorg/maru-tasks:v1.0.0

# Push the task file (to an insecure registry)
maru push --insecure hello.yaml ghcr.io/myorg/maru-tasks:v1.0.0
```

**Using OCI Tasks**

To use an OCI task file, use the `oci://` prefix in your includes:

```yaml
includes:
- common: oci://ghcr.io/myorg/tasks:v1.0.0

tasks:
- name: use-oci-task
actions:
- task: common:setup-env
```

Authentication to private OCI registries works the same way as for remote HTTPS task files, using the `maru auth login` command with the registry hostname.

If the registry is insecure, you can use the `--insecure` flag when pushing the task file:

```bash
maru run --insecure common:setup-env
```

#### Authenticated Includes

Some included remote task files may require authentication to access - to access these you can use the `maru auth login` command to add a personal access token (bearer auth) to your computer keychain.
Expand Down
3 changes: 3 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ require (
github.com/lithammer/fuzzysearch v1.1.8 // indirect
github.com/mailru/easyjson v0.9.0 // indirect
github.com/mattn/go-runewidth v0.0.16 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.0 // indirect
github.com/otiai10/copy v1.14.1 // indirect
github.com/otiai10/mint v1.6.3 // indirect
github.com/pelletier/go-toml/v2 v2.2.4 // indirect
Expand Down
6 changes: 6 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,12 @@ github.com/mailru/easyjson v0.9.0/go.mod h1:1+xMtQp2MRNVL/V1bOzuP3aP8VNwRW55fQUt
github.com/mattn/go-runewidth v0.0.13/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
github.com/mattn/go-runewidth v0.0.16 h1:E5ScNMtiwvlvB5paMFdw9p4kSQzbXFikJ5SQO6TULQc=
github.com/mattn/go-runewidth v0.0.16/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
github.com/mitchellh/mapstructure v1.5.0 h1:jeMsZIYE/09sWLaz43PL7Gy6RuMjD2eJVyuac5Z2hdY=
github.com/mitchellh/mapstructure v1.5.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U=
github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
github.com/opencontainers/image-spec v1.1.0 h1:8SG7/vwALn54lVB/0yZ/MMwhFrPYtpEHQb2IpWsCzug=
github.com/opencontainers/image-spec v1.1.0/go.mod h1:W4s4sFTMaBeK1BQLXbG4AdM2szdn85PY75RI83NrTrM=
github.com/otiai10/copy v1.14.1 h1:5/7E6qsUMBaH5AnQ0sSLzzTg1oTECmcCmT6lvF45Na8=
github.com/otiai10/copy v1.14.1/go.mod h1:oQwrEDDOci3IM8dJF0d8+jnbfPDllW6vUjNc3DoZm9I=
github.com/otiai10/mint v1.6.3 h1:87qsV/aw1F5as1eH1zS/yqHY85ANKVMgkDrf9rcxbQs=
Expand Down
206 changes: 206 additions & 0 deletions src/cmd/push.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,206 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: 2023-Present the Maru Authors

// Package cmd contains the CLI commands for maru.
package cmd

import (
"context"
"fmt"
"net/http"
"os"
"path/filepath"
"strings"

"github.com/defenseunicorns/maru-runner/src/config"
"github.com/defenseunicorns/maru-runner/src/message"
v1 "github.com/opencontainers/image-spec/specs-go/v1" // ocispec
"github.com/spf13/cobra"
keyring "github.com/zalando/go-keyring"
oras "oras.land/oras-go/v2"
"oras.land/oras-go/v2/content/file"
"oras.land/oras-go/v2/registry/remote"
"oras.land/oras-go/v2/registry/remote/auth"
)

// OCI artifact media types
const (
yamlMediaType = "application/yaml"
emptyConfigType = "application/vnd.oci.empty.v1+json"
defaultYamlArtifact = "application/vnd.oci.image.manifest.v1+json"
)

var pushCmd = &cobra.Command{
Use: "push TASK_FILE OCI_REFERENCE",
Short: "Push a task file to an OCI registry",
Long: `Push a Maru task file to an OCI registry.

Examples:
# Push a task file to GitHub Container Registry
maru push tasks.yaml ghcr.io/myorg/maru-tasks:latest

# Push a task file with a specific tag
maru push tasks.yaml ghcr.io/myorg/maru-tasks:v1.0.0
`,
Args: cobra.ExactArgs(2),
RunE: func(cmd *cobra.Command, args []string) error {
taskFile := args[0]
reference := args[1]

return pushTaskFile(taskFile, reference)
},
}

var pushInsecure bool

func init() {
initViper()
rootCmd.AddCommand(pushCmd)
pushCmd.Flags().BoolVar(&pushInsecure, "insecure", false, "Allow interaction with OCI registries that are not using HTTPS")
}

// Parse an OCI reference into registry, repository, and tag
func parseOCIReference(reference string) (registry string, repository string, tag string, err error) {
// Verify reference starts with oci:// prefix
if !strings.HasPrefix(reference, "oci://") {
return "", "", "", fmt.Errorf("reference must start with 'oci://', got: %s", reference)
}

// Remove oci:// prefix
reference = strings.TrimPrefix(reference, "oci://")

// Format expected: registry/repo/path:tag
parts := strings.SplitN(reference, "/", 2)
if len(parts) < 2 {
return "", "", "", fmt.Errorf("invalid reference format: %s", reference)
}

registry = parts[0] // e.g., ghcr.io
remainder := parts[1] // e.g., myorg/maru-tasks:0.0.1

// Split the remainder at the colon to get repo path and tag
repoAndTag := strings.SplitN(remainder, ":", 2)
repository = repoAndTag[0] // e.g., myorg/maru-tasks
tag = "latest"
if len(repoAndTag) > 1 {
tag = repoAndTag[1] // e.g., 0.0.1
}

return registry, repository, tag, nil
}

// Push a task file to an OCI registry
func pushTaskFile(taskFilePath, reference string) error {
ctx := context.Background()

// Verify the task file exists
if _, err := os.Stat(taskFilePath); os.IsNotExist(err) {
return fmt.Errorf("task file not found: %s", taskFilePath)
}

// Create a temporary directory for the file store
tmpDir, err := os.MkdirTemp("", "maru-push-*")
if err != nil {
return fmt.Errorf("failed to create temporary directory: %w", err)
}
defer os.RemoveAll(tmpDir)

// Create a file store
fs, err := file.New(tmpDir)
if err != nil {
return fmt.Errorf("failed to create file store: %w", err)
}
defer fs.Close()

// Parse the OCI reference
registry, repoPath, tag, err := parseOCIReference(reference)
if err != nil {
return err
}

// Full repository reference
fullRepo := fmt.Sprintf("%s/%s", registry, repoPath)
message.SLog.Info(fmt.Sprintf("Pushing %s to %s:%s", taskFilePath, fullRepo, tag))

// Get absolute path of the task file (for reading)
taskFileAbs, err := filepath.Abs(taskFilePath)
if err != nil {
return fmt.Errorf("failed to get absolute path: %w", err)
}
// Use only the base name to avoid absolute path issues in the artifact.
taskFileName := filepath.Base(taskFilePath)

// Create the empty config file (empty content to mimic /dev/null)
configFileName := "config.json"
configFileFull := filepath.Join(tmpDir, configFileName)
if err := os.WriteFile(configFileFull, []byte(""), 0644); err != nil {
return fmt.Errorf("failed to create config file: %w", err)
}

// Add the empty config to the file store using a relative name.
configDesc, err := fs.Add(ctx, configFileName, emptyConfigType, configFileFull)
if err != nil {
return fmt.Errorf("failed to add config file: %w", err)
}

// Add the task file to the file store with a relative name.
taskDesc, err := fs.Add(ctx, taskFileName, yamlMediaType, taskFileAbs)
if err != nil {
return fmt.Errorf("failed to add task file: %w", err)
}

// Pack the files into a manifest
layers := []v1.Descriptor{taskDesc}
manifestDesc, err := oras.PackManifest(ctx, fs, oras.PackManifestVersion1_1, defaultYamlArtifact, oras.PackManifestOptions{
Layers: layers,
ConfigDescriptor: &configDesc,
})
if err != nil {
return fmt.Errorf("failed to pack manifest: %w", err)
}

// Tag the manifest
err = fs.Tag(ctx, manifestDesc, tag)
if err != nil {
return fmt.Errorf("failed to tag manifest: %w", err)
}

// Create a new repository client
repo, err := remote.NewRepository(fullRepo)
if err != nil {
return fmt.Errorf("failed to create repository client: %w", err)
}

// Configure insecure mode if requested
if pushInsecure {
message.SLog.Info(fmt.Sprintf("Using insecure mode for %s", fullRepo))
repo.PlainHTTP = true
}

// Try to get token from keyring for the registry
token, err := keyring.Get(config.KeyringService, registry)
if err == nil && token != "" {
// Configure authentication
authClient := &auth.Client{
Client: http.DefaultClient,
Cache: auth.NewCache(),
Credential: auth.StaticCredential(registry, auth.Credential{
Username: "token",
Password: token,
}),
}
repo.Client = authClient
} else {
message.SLog.Debug(fmt.Sprintf("No authentication token found for %s", registry))
message.SLog.Info(fmt.Sprintf("You may need to authenticate using 'maru auth login %s --token <YOUR_TOKEN>'", registry))
}

// Copy from the file store to the remote repository
_, err = oras.Copy(ctx, fs, tag, repo, tag, oras.DefaultCopyOptions)
if err != nil {
return fmt.Errorf("failed to push OCI artifact: %w", err)
}

message.SLog.Info(fmt.Sprintf("Successfully pushed %s to %s", taskFilePath, reference))
return nil
}
9 changes: 9 additions & 0 deletions src/cmd/run.go
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,9 @@ func (i *listFlag) Set(value string) error {
// dryRun is a flag to only load / validate tasks without running commands
var dryRun bool

// insecure is a flag to allow interaction with OCI registries that are not using HTTPS
var insecure bool

// setRunnerVariables provides a map of set variables from the command line
var setRunnerVariables map[string]string

Expand Down Expand Up @@ -96,6 +99,11 @@ var runCmd = &cobra.Command{

auth := v.GetStringMapString(V_AUTH)

// Pass insecure flag to the auth map for OCI operations
if insecure {
auth["insecure"] = "true"
}

listFormat := listTasks
if listAllTasks != listOff {
listFormat = listAllTasks
Expand Down Expand Up @@ -202,6 +210,7 @@ func init() {
runFlags := runCmd.Flags()
runFlags.StringVarP(&config.TaskFileLocation, "file", "f", config.TasksYAML, lang.CmdRunFlag)
runFlags.BoolVar(&dryRun, "dry-run", false, lang.CmdRunDryRun)
runFlags.BoolVar(&insecure, "insecure", false, "Allow interaction with OCI registries that are not using HTTPS")

// Setup the --list flag
flag.Var(&listTasks, "list", lang.CmdRunList)
Expand Down
16 changes: 15 additions & 1 deletion src/pkg/runner/runner.go
Original file line number Diff line number Diff line change
Expand Up @@ -239,10 +239,24 @@ func includeTaskAbsLocation(currentFileLocation, includeFileLocation string) (st
return absIncludeFileLocation, nil
}

// LoadIncludeTask loads an included task file either from a remote or local file
// isOCIReference checks if a string is an OCI reference
func isOCIReference(reference string) bool {
return strings.HasPrefix(reference, "oci://")
}

// LoadIncludeTask loads an included task file from a remote, OCI, or local file
func LoadIncludeTask(currentFileLocation, includeFileLocation string, auth map[string]string) (string, types.TasksFile, error) {
var includedTasksFile types.TasksFile

// Check if this is an OCI reference
if isOCIReference(includeFileLocation) {
// For OCI references, we use the full reference as the location identifier
ociReference := strings.TrimPrefix(includeFileLocation, "oci://")
err := utils.ReadOCIYaml(ociReference, &includedTasksFile, auth)
return includeFileLocation, includedTasksFile, err
}

// Handle normal file or URL references
absIncludeFileLocation, err := includeTaskAbsLocation(currentFileLocation, includeFileLocation)
if err != nil {
return absIncludeFileLocation, includedTasksFile, err
Expand Down
Loading
Loading