diff --git a/.github/skills/regshape-dev/SKILL.md b/.github/skills/regshape-dev/SKILL.md index 3ce99be..cf92067 100644 --- a/.github/skills/regshape-dev/SKILL.md +++ b/.github/skills/regshape-dev/SKILL.md @@ -1,6 +1,6 @@ --- +description: 'Design, implement, and document the RegShape Python library and CLI for OCI (Open Containers Initiative) registry interactions. Use when: (1) Designing API specs or data models for registry operations, (2) Implementing Python library modules for manifests, blobs, tags, auth, or referrers, (3) Designing CLI commands using Click, (4) Writing user documentation for the library or CLI, (5) Planning break/test mode features for testing registry implementations, (6) Working on any RegShape library or CLI code.' name: regshape-dev -description: "Design, implement, and document the RegShape Python library and CLI for OCI (Open Containers Initiative) registry interactions. Use when: (1) Designing API specs or data models for registry operations, (2) Implementing Python library modules for manifests, blobs, tags, auth, or referrers, (3) Designing CLI commands using Click, (4) Writing user documentation for the library or CLI, (5) Planning break/test mode features for testing registry implementations, (6) Working on any RegShape library or CLI code." --- # RegShape Development @@ -13,11 +13,13 @@ The recommended workflow is **spec → implement → test → document**, but an ### Phase 1: Design Spec -1. Read [references/oci-distribution-spec.md](references/oci-distribution-spec.md) for the relevant OCI API details +1. Read the relevant OCI spec references for the domain: + - [references/oci-distribution-spec.md](references/oci-distribution-spec.md) — registry API endpoints, uploads, auth, error codes + - [references/oci-image-spec.md](references/oci-image-spec.md) — image manifests, image index, image config, filesystem layers, descriptors 2. Read [references/spec-templates.md](references/spec-templates.md) for the spec template to use 3. Read [references/project-conventions.md](references/project-conventions.md) for naming and structure patterns -4. Create the spec as a Markdown file in `/specs//.md` -5. Present the spec to the user for review before proceeding + +4. Create the spec as a Markdown file in `/specs//.md`5. Present the spec to the user for review before proceeding ### Phase 2: Implement @@ -27,6 +29,7 @@ The recommended workflow is **spec → implement → test → document**, but an - Module file with functions/classes - Update `__init__.py` exports - Add error types to `libs/errors.py` if needed + 4. Write tests in `src/regshape/tests/test_.py` 5. Run `pytest` to verify @@ -45,6 +48,7 @@ The recommended workflow is **spec → implement → test → document**, but an - `docs/library/` — Library API reference - `docs/cli/` — CLI command reference - `docs/guides/` — How-to guides + 3. Keep docs aligned with the specs and implementation ## Domain Areas @@ -53,13 +57,15 @@ The recommended workflow is **spec → implement → test → document**, but an |--------|-------------|-----------|----------| | Authentication | `libs/auth/` | `specs/auth/` | `docs/library/auth.md` | | Manifests | `libs/manifests/` | `specs/registry/manifest-operations.md` | `docs/library/manifests.md` | +| Image Manifests & Index | `libs/manifests/` | `specs/registry/image-manifests.md` | `docs/library/manifests.md` | +| Images | `libs/images/` | `specs/registry/image-operations.md` | `docs/library/images.md` | | Blobs | `libs/blobs/` | `specs/registry/blob-operations.md` | `docs/library/blobs.md` | | Tags | `libs/tags/` | `specs/registry/tag-operations.md` | `docs/library/tags.md` | | Referrers | `libs/referrers/` | `specs/registry/referrers.md` | `docs/library/referrers.md` | | Catalog | `libs/catalog/` | `specs/registry/catalog.md` | `docs/library/catalog.md` | | Break Mode | `libs/breakmode/` | `specs/breakmode/` | `docs/guides/break-mode.md` | -| CLI | `cli/` | `specs/cli/` | `docs/cli/` | -| Models | `libs/models/` | `specs/models/` | `docs/library/models.md` | + +| CLI | `cli/` | `specs/cli/` | `docs/cli/` || Models | `libs/models/` | `specs/models/` | `docs/library/models.md` | ## GitHub Issue Management @@ -80,9 +86,9 @@ gh issue create \ ### Common Issue Labels -- **Priority**: `priority-high`, `priority-medium`, `priority-low` +- **Priority**: `priority-high`, `priority-medium`, `priority-low` - **Type**: `enhancement`, `bug`, `documentation`, `architecture` -- **Domain**: `break-mode`, `cli`, `oci-spec`, `transport`, `models` +- **Domain**: `break-mode`, `cli`, `oci-spec`, `image-spec`, `images`, `transport`, `models` - **Status**: `blocked`, `help-wanted`, `good-first-issue` ### Issue Templates @@ -90,17 +96,20 @@ gh issue create \ Use these templates for consistent issue creation: **Feature Implementation Issues:** + - Title: `Implement [Feature Name] - [Brief Description]` - Include acceptance criteria checklist - List files to create/modify - Add appropriate priority and domain labels **Bug Report Issues:** -- Title: `Bug: [Brief Description]` + +- Title: `Bug: [Brief Description]` - Include steps to reproduce, expected vs actual behavior - Add `bug` label and relevant domain labels **Architecture Issues:** + - Title: `Architecture: [Component/Enhancement]` - Include design decisions, trade-offs, implementation notes - Add `architecture` label @@ -108,7 +117,7 @@ Use these templates for consistent issue creation: ### Workflow Integration 1. **When planning work**: Create issues for each major component or feature -2. **Before coding**: Reference issue numbers in commit messages (`fixes #123`) +2. **Before coding**: Reference issue numbers in commit messages (`fixes #123`) 3. **During implementation**: Update issue status and add progress notes 4. **After completion**: Close issues automatically via commit messages or PR merges @@ -117,6 +126,7 @@ Use these templates for consistent issue creation: RegShape's distinguishing feature: deliberately send malformed or non-conformant requests to test registry implementations. When designing break mode features: + - Provide options to modify any part of a request (headers, body, method, path, digest values) - Allow sending requests with invalid content types, wrong digests, oversized payloads - Support skipping authentication steps or using expired tokens @@ -133,17 +143,17 @@ automatically from the Docker credential store and **do not** accept per-command `--username` / `--password` flags. - The `auth login` command stores credentials using the Docker credential store mechanics - (`credHelpers` in `~/.docker/config.json` or base64-encoded `auths` entries). + (`credHelpers` in `~/.docker/config.json` or base64-encoded `auths` entries). - Non-auth commands call `resolve_credentials(registry, None, None)` to look up stored - credentials for the embedded registry. + credentials for the embedded registry. - If no stored credentials exist and the registry requires authentication, the registry - returns HTTP 401, which surfaces as exit code 1. + returns HTTP 401, which surfaces as exit code 1. ### Registry Must Be Embedded in Image Reference The `--image-ref` / `-i` flag always requires the registry to be embedded: -``` +```sh registry/repository:tag # e.g., acr.io/myrepo/myimage:latest registry/repository@sha256:... # e.g., acr.io/myrepo/myimage@sha256:abc... ``` @@ -160,6 +170,7 @@ There are no global auth options. ## Key References - **OCI Distribution Spec**: [references/oci-distribution-spec.md](references/oci-distribution-spec.md) — API endpoints, schemas, error codes +- **OCI Image Spec**: [references/oci-image-spec.md](references/oci-image-spec.md) — image manifests, image index, image config, filesystem layers, descriptors (full upstream spec mirrored in [references/oci-image-spec-source/](references/oci-image-spec-source/)) - **Project Conventions**: [references/project-conventions.md](references/project-conventions.md) — Code style, module structure, naming - **Spec Templates**: [references/spec-templates.md](references/spec-templates.md) — Templates for `/specs` documents - **Doc Templates**: [references/doc-templates.md](references/doc-templates.md) — Templates for `/docs` documents diff --git a/.github/skills/regshape-dev/references/oci-image-spec-source/annotations.md b/.github/skills/regshape-dev/references/oci-image-spec-source/annotations.md new file mode 100644 index 0000000..96f5abf --- /dev/null +++ b/.github/skills/regshape-dev/references/oci-image-spec-source/annotations.md @@ -0,0 +1,83 @@ +# Annotations + +Several components of the specification, like [Image Manifests](manifest.md) and [Descriptors](descriptor.md), feature an optional annotations property, whose format is common and defined in this section. + +This property contains arbitrary metadata. + +## Rules + +- Annotations MUST be a key-value map where both the key and value MUST be strings. +- While the value MUST be present, it MAY be an empty string. +- Keys MUST be unique within this map, and best practice is to namespace the keys. +- Keys SHOULD be named using a reverse domain notation - e.g. `com.example.myKey`. +- The prefix `org.opencontainers` is reserved for keys defined in Open Container Initiative (OCI) specifications and MUST NOT be used by other specifications and extensions. +- Keys using the `org.opencontainers.image` namespace are reserved for use in the OCI Image Specification and MUST NOT be used by other specifications and extensions, including other OCI specifications. +- If there are no annotations then this property MUST either be absent or be an empty map. +- Consumers MUST NOT generate an error if they encounter an unknown annotation key. + +## Pre-Defined Annotation Keys + +This specification defines the following annotation keys, intended for but not limited to [image index](image-index.md), image [manifest](manifest.md), and [descriptor](descriptor.md) authors. + +- **org.opencontainers.image.created** date and time on which the image was built, conforming to [RFC 3339][rfc3339]. +- **org.opencontainers.image.authors** contact details of the people or organization responsible for the image (freeform string) +- **org.opencontainers.image.url** URL to find more information on the image (string) +- **org.opencontainers.image.documentation** URL to get documentation on the image (string) +- **org.opencontainers.image.source** URL to get source code for building the image (string) +- **org.opencontainers.image.version** version of the packaged software + - The version MAY match a label or tag in the source code repository + - version MAY be [Semantic versioning-compatible](https://semver.org/) +- **org.opencontainers.image.revision** Source control revision identifier for the packaged software. +- **org.opencontainers.image.vendor** Name of the distributing entity, organization or individual. +- **org.opencontainers.image.licenses** License(s) under which contained software is distributed as an [SPDX License Expression][spdx-license-expression]. +- **org.opencontainers.image.ref.name** Name of the reference for a target (string). + - SHOULD only be considered valid when on descriptors on `index.json` within [image layout](image-layout.md). + - Character set of the value SHOULD conform to alphanum of `A-Za-z0-9` and separator set of `-._:@/+` + - A valid reference matches the following [grammar](considerations.md#ebnf): + + ```ebnf + ref ::= component ("/" component)* + component ::= alphanum (separator alphanum)* + alphanum ::= [A-Za-z0-9]+ + separator ::= [-._:@+] | "--" + ``` + +- **org.opencontainers.image.title** Human-readable title of the image (string) +- **org.opencontainers.image.description** Human-readable description of the software packaged in the image (string) +- **org.opencontainers.image.base.digest** [Digest](descriptor.md#digests) of the image this image is based on (string) + - This SHOULD be the immediate image sharing zero-indexed layers with the image, such as from a Dockerfile `FROM` statement. + - This SHOULD NOT reference any other images used to generate the contents of the image (e.g., multi-stage Dockerfile builds). +- **org.opencontainers.image.base.name** Image reference of the image this image is based on (string) + - This SHOULD be image references in the format defined by [distribution/distribution][distribution-reference]. + - This SHOULD be a fully qualified reference name, without any assumed default registry. (e.g., `registry.example.com/my-org/my-image:tag` instead of `my-org/my-image:tag`). + - This SHOULD be the immediate image sharing zero-indexed layers with the image, such as from a Dockerfile `FROM` statement. + - This SHOULD NOT reference any other images used to generate the contents of the image (e.g., multi-stage Dockerfile builds). + - If the `image.base.name` annotation is specified, the `image.base.digest` annotation SHOULD be the digest of the manifest referenced by the `image.ref.name` annotation. + +## Back-compatibility with Label Schema + +[Label Schema][label-schema] defined a number of conventional labels for container images, and these are now superseded by annotations with keys starting **org.opencontainers.image**. + +While users are encouraged to use the **org.opencontainers.image** keys, tools MAY choose to support compatible annotations using the **org.label-schema** prefix as follows. + +| `org.opencontainers.image` prefix | `org.label-schema` prefix | Compatibility notes | +|---------------------------|-------------------------|---------------------| +| `created` | `build-date` | Compatible | +| `url` | `url` | Compatible | +| `source` | `vcs-url` | Compatible | +| `version` | `version` | Compatible | +| `revision` | `vcs-ref` | Compatible | +| `vendor` | `vendor` | Compatible | +| `title` | `name` | Compatible | +| `description` | `description` | Compatible | +| `documentation` | `usage` | Value is compatible if the documentation is located by a URL | +| `authors` | | No equivalent in Label Schema | +| `licenses` | | No equivalent in Label Schema | +| `ref.name` | | No equivalent in Label Schema | +| | `schema-version`| No equivalent in the OCI Image Spec | +| | `docker.*`, `rkt.*` | No equivalent in the OCI Image Spec | + +[distribution-reference]: https://github.com/distribution/distribution/blob/d0deff9cd6c2b8c82c6f3d1c713af51df099d07b/reference/reference.go +[label-schema]: https://github.com/label-schema/label-schema.org/blob/gh-pages/rc1.md +[rfc3339]: https://tools.ietf.org/html/rfc3339#section-5.6 +[spdx-license-expression]: https://spdx.github.io/spdx-spec/v2.3/SPDX-license-expressions/ diff --git a/.github/skills/regshape-dev/references/oci-image-spec-source/artifacts-guidance.md b/.github/skills/regshape-dev/references/oci-image-spec-source/artifacts-guidance.md new file mode 100644 index 0000000..49b8c63 --- /dev/null +++ b/.github/skills/regshape-dev/references/oci-image-spec-source/artifacts-guidance.md @@ -0,0 +1,6 @@ +# Guidance for Artifacts Authors + +Content other than OCI container images MAY be packaged using the image manifest. +When this is done, the `config.mediaType` value should not be a known OCI image config [media type](media-types.md). +Historically, due to registry limitations, some tools have created non-OCI conformant artifacts using the `application/vnd.oci.image.config.v1+json` value for `config.mediaType` and values specific to the artifact in `layer[*].mediaType`. +Implementation details and examples are provided in the [image manifest specification](manifest.md#guidelines-for-artifact-usage). diff --git a/.github/skills/regshape-dev/references/oci-image-spec-source/config.md b/.github/skills/regshape-dev/references/oci-image-spec-source/config.md new file mode 100644 index 0000000..904d749 --- /dev/null +++ b/.github/skills/regshape-dev/references/oci-image-spec-source/config.md @@ -0,0 +1,320 @@ +# OCI Image Configuration + +An OCI _Image_ is an ordered collection of root filesystem changes and the corresponding execution parameters for use within a container runtime. +This specification outlines the JSON format describing images for use with a container runtime and execution tool and its relationship to filesystem changesets, described in [Layers](layer.md). + +This section defines the `application/vnd.oci.image.config.v1+json` [media type](media-types.md). + +## Terminology + +This specification uses the following terms: + +### [Layer](layer.md) + +- Image filesystems are composed of _layers_. +- Each layer represents a set of filesystem changes in a tar-based [layer format](layer.md), recording files to be added, changed, or deleted relative to its parent layer. +- Layers do not have configuration metadata such as environment variables or default arguments - these are properties of the image as a whole rather than any particular layer. +- Using a layer-based or union filesystem such as AUFS, or by computing the diff from filesystem snapshots, the filesystem changeset can be used to present a series of image layers as if they were one cohesive filesystem. + +### Image JSON + +- Each image has an associated JSON structure which describes some basic information about the image such as date created, author, as well as execution/runtime configuration like its entrypoint, default arguments, networking, and volumes. +- The JSON structure also references a cryptographic hash of each layer used by the image, and provides history information for those layers. +- This JSON is considered to be immutable, because changing it would change the computed [ImageID](#imageid). +- Changing it means creating a new derived image, instead of changing the existing image. + +### Layer DiffID + +A layer DiffID is the digest over the layer's uncompressed tar archive and serialized in the descriptor digest format, e.g., `sha256:a9561eb1b190625c9adb5a9513e72c4dedafc1cb2d4c5236c9a6957ec7dfd5a9`. +Layers SHOULD be packed and unpacked reproducibly to avoid changing the layer DiffID, for example by using [tar-split][] to save the tar headers. + +NOTE: Do not confuse DiffIDs with [layer digests](manifest.md#image-manifest-property-descriptions), often referenced in the manifest, which are digests over compressed or uncompressed content. + +### Layer ChainID + +For convenience, it is sometimes useful to refer to a stack of layers with a single identifier. +While a layer's `DiffID` identifies a single changeset, the `ChainID` identifies the subsequent application of those changesets. +This ensures that we have handles referring to both the layer itself, as well as the result of the application of a series of changesets. +Use in combination with `rootfs.diff_ids` while applying layers to a root filesystem to uniquely and safely identify the result. + +#### Definition + +The `ChainID` of an applied set of layers is defined with the following recursion: + +```text +ChainID(L₀) = DiffID(L₀) +ChainID(L₀|...|Lₙ₋₁|Lₙ) = Digest(ChainID(L₀|...|Lₙ₋₁) + " " + DiffID(Lₙ)) +``` + +For this, we define the binary `|` operation to be the result of applying the right operand to the left operand. +For example, given base layer `A` and a changeset `B`, we refer to the result of applying `B` to `A` as `A|B`. + +Above, we define the `ChainID` for a single layer (`L₀`) as equivalent to the `DiffID` for that layer. +Otherwise, the `ChainID` for a set of applied layers (`L₀|...|Lₙ₋₁|Lₙ`) is defined as the recursion `Digest(ChainID(L₀|...|Lₙ₋₁) + " " + DiffID(Lₙ))`. + +#### Explanation + +Let's say we have layers A, B, C, ordered from bottom to top, where A is the base and C is the top. +Defining `|` as a binary application operator, the root filesystem may be `A|B|C`. +While it is implied that `C` is only useful when applied to `A|B`, the identifier `C` is insufficient to identify this result, as we'd have the equality `C = A|B|C`, which isn't true. + +The main issue is when we have two definitions of `C`, `C = C` and `C = A|B|C`. +If this is true (with some handwaving), `C = x|C` where `x = any application`. +This means that if an attacker can define `x`, relying on `C` provides no guarantee that the layers were applied in any order. + +The `ChainID` addresses this problem by being defined as a compound hash. +**We differentiate the changeset `C`, from the order-dependent application `A|B|C` by saying that the resulting rootfs is identified by ChainID(A|B|C), which can be calculated by `ImageConfig.rootfs`.** + +Let's expand the definition of `ChainID(A|B|C)` to explore its internal structure: + +```text +ChainID(A) = DiffID(A) +ChainID(A|B) = Digest(ChainID(A) + " " + DiffID(B)) +ChainID(A|B|C) = Digest(ChainID(A|B) + " " + DiffID(C)) +``` + +We can replace each definition and reduce to a single equality: + +```text +ChainID(A|B|C) = Digest(Digest(DiffID(A) + " " + DiffID(B)) + " " + DiffID(C)) +``` + +Hopefully, the above is illustrative of the _actual_ contents of the `ChainID`. +Most importantly, we can easily see that `ChainID(C) != ChainID(A|B|C)`, otherwise, `ChainID(C) = DiffID(C)`, which is the base case, could not be true. + +### ImageID + +Each image's ID is given by the SHA256 hash of its [configuration JSON](#image-json). +It is represented as a hexadecimal encoding of 256 bits, e.g., `sha256:a9561eb1b190625c9adb5a9513e72c4dedafc1cb2d4c5236c9a6957ec7dfd5a9`. +Since the [configuration JSON](#image-json) that gets hashed references hashes of each layer in the image, this formulation of the ImageID makes images content-addressable. + +## Properties + +Note: Any OPTIONAL field MAY also be set to null, which is equivalent to being absent. + +- **created** _string_, OPTIONAL + + An combined date and time at which the image was created, formatted as defined by [RFC 3339, section 5.6][rfc3339-s5.6]. + +- **author** _string_, OPTIONAL + + Gives the name and/or email address of the person or entity which created and is responsible for maintaining the image. + +- **architecture** _string_, REQUIRED + + The CPU architecture which the binaries in this image are built to run on. + Configurations SHOULD use, and implementations SHOULD understand, values listed in the Go Language document for [`GOARCH`][go-environment]. + +- **os** _string_, REQUIRED + + The name of the operating system which the image is built to run on. + Configurations SHOULD use, and implementations SHOULD understand, values listed in the Go Language document for [`GOOS`][go-environment]. + +- **os.version** _string_, OPTIONAL + + This OPTIONAL property specifies the version of the operating system targeted by the referenced blob. + Implementations MAY refuse to use manifests where `os.version` is not known to work with the host OS version. + Valid values are implementation-defined. e.g. `10.0.14393.1066` on `windows`. + +- **os.features** _array of strings_, OPTIONAL + + This OPTIONAL property specifies an array of strings, each specifying a mandatory OS feature. + When `os` is `windows`, image indexes SHOULD use, and implementations SHOULD understand the following values: + + - `win32k`: image requires `win32k.sys` on the host (Note: `win32k.sys` is missing on Nano Server) + +- **variant** _string_, OPTIONAL + + The variant of the specified CPU architecture. + Configurations SHOULD use, and implementations SHOULD understand, `variant` values listed in the [Platform Variants](image-index.md#platform-variants) table. + +- **config** _object_, OPTIONAL + + The execution parameters which SHOULD be used as a base when running a container using the image. + If the `config` object is omitted, any execution parameters should be specified at creation of the container. + + - **User** _string_, OPTIONAL + + The username or UID which is a platform-specific structure that allows specific control over which user the process run as. + This acts as a default value to use when the value is not specified when creating a container. + For Linux based systems, all of the following are valid: `user`, `uid`, `user:group`, `uid:gid`, `uid:group`, `user:gid`. + If `group`/`gid` is not specified, the default group and supplementary groups of the given `user`/`uid` in `/etc/passwd` and `/etc/group` from the container are applied. + If `group`/`gid` is specified, supplementary groups from the container are ignored. + + - **ExposedPorts** _object_, OPTIONAL + + A set of ports to expose from a container running this image. + Its keys can be in the format of: +`port/tcp`, `port/udp`, `port` with the default protocol being `tcp` if not specified. + These values act as defaults and are merged with any specified when creating a container. + **NOTE:** This JSON structure value is unusual because it is a direct JSON serialization of the Go type `map[string]struct{}` and is represented in JSON as an object mapping its keys to an empty object. + + - **Env** _array of strings_, OPTIONAL + + Entries are in the format of `VARNAME=VARVALUE`. + These values act as defaults and are merged with any specified when creating a container. + + - **Entrypoint** _array of strings_, OPTIONAL + + A list of arguments to use as the command to execute when the container starts. + These values act as defaults and may be replaced by an entrypoint specified when creating a container. + + - **Cmd** _array of strings_, OPTIONAL + + Default arguments to the entrypoint of the container. + These values act as defaults and may be replaced by any specified when creating a container. + If an `Entrypoint` value is not specified, then the first entry of the `Cmd` array SHOULD be interpreted as the executable to run. + + - **Volumes** _object_, OPTIONAL + + A set of directories describing where the process is likely to write data specific to a container instance. + **NOTE:** This JSON structure value is unusual because it is a direct JSON serialization of the Go type `map[string]struct{}` and is represented in JSON as an object mapping its keys to an empty object. + + - **WorkingDir** _string_, OPTIONAL + + Sets the current working directory of the entrypoint process in the container. + This value acts as a default and may be replaced by a working directory specified when creating a container. + + - **Labels** _object_, OPTIONAL + + This field contains arbitrary metadata for the container. + This property MUST use the [annotation rules](annotations.md#rules). + + - **StopSignal** _string_, OPTIONAL + + This field contains the system call signal that will be sent to the container to exit. The signal can be a signal name in the format `SIGNAME`, for instance `SIGKILL` or `SIGRTMIN+3`. + + - **ArgsEscaped** _boolean_, OPTIONAL + + `[Deprecated]` - This field is present only for legacy compatibility with Docker and should not be used by new image builders. + It is used by Docker for Windows images to indicate that the `Entrypoint` or `Cmd` or both, contains only a single element array, that is a pre-escaped, and combined into a single string `CommandLine`. + If `true` the value in `Entrypoint` or `Cmd` should be used as-is to avoid double escaping. + Note, the exact behavior of `ArgsEscaped` is complex and subject to implementation details in Moby project. + + - **Memory** _integer_, OPTIONAL + + This property is _reserved_ for use, to [maintain compatibility](media-types.md#compatibility-matrix). + + - **MemorySwap** _integer_, OPTIONAL + + This property is _reserved_ for use, to [maintain compatibility](media-types.md#compatibility-matrix). + + - **CpuShares** _integer_, OPTIONAL + + This property is _reserved_ for use, to [maintain compatibility](media-types.md#compatibility-matrix). + + - **Healthcheck** _object_, OPTIONAL + + This property is _reserved_ for use, to [maintain compatibility](media-types.md#compatibility-matrix). + +- **rootfs** _object_, REQUIRED + + The rootfs key references the layer content addresses used by the image. + This makes the image config hash depend on the filesystem hash. + + - **type** _string_, REQUIRED + + MUST be set to `layers`. + Implementations MUST generate an error if they encounter a unknown value while verifying or unpacking an image. + + - **diff_ids** _array of strings_, REQUIRED + + An array of layer content hashes (`DiffIDs`), in order from first to last. + +- **history** _array of objects_, OPTIONAL + + Describes the history of each layer. + The array is ordered from first to last. + The object has the following fields: + + - **created** _string_, OPTIONAL + + A combined date and time at which the layer was created, formatted as defined by [RFC 3339, section 5.6][rfc3339-s5.6]. + + - **author** _string_, OPTIONAL + + The author of the build point. + + - **created_by** _string_, OPTIONAL + + The command which created the layer. + + - **comment** _string_, OPTIONAL + + A custom message set when creating the layer. + + - **empty_layer** _boolean_, OPTIONAL + + This field is used to mark if the history item created a filesystem diff. + It is set to true if this history item doesn't correspond to an actual layer in the rootfs section (for example, Dockerfile's [ENV](https://docs.docker.com/engine/reference/builder/#/env) command results in no change to the filesystem). + +Any extra fields in the Image JSON struct are considered implementation specific and MUST NOT generate an error by any implementations which are unable to interpret them. + +Whitespace is OPTIONAL and implementations MAY have compact JSON with no whitespace. + +## Example + +Here is an example image configuration JSON document: + +```json,title=Image%20JSON&mediatype=application/vnd.oci.image.config.v1%2Bjson +{ + "created": "2015-10-31T22:22:56.015925234Z", + "author": "Alyssa P. Hacker ", + "architecture": "amd64", + "os": "linux", + "config": { + "User": "alice", + "ExposedPorts": { + "8080/tcp": {} + }, + "Env": [ + "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", + "FOO=oci_is_a", + "BAR=well_written_spec" + ], + "Entrypoint": [ + "/bin/my-app-binary" + ], + "Cmd": [ + "--foreground", + "--config", + "/etc/my-app.d/default.cfg" + ], + "Volumes": { + "/var/job-result-data": {}, + "/var/log/my-app-logs": {} + }, + "WorkingDir": "/home/alice", + "Labels": { + "com.example.project.git.url": "https://example.com/project.git", + "com.example.project.git.commit": "45a939b2999782a3f005621a8d0f29aa387e1d6b" + } + }, + "rootfs": { + "diff_ids": [ + "sha256:c6f988f4874bb0add23a778f753c65efe992244e148a1d2ec2a8b664fb66bbd1", + "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" + ], + "type": "layers" + }, + "history": [ + { + "created": "2015-10-31T22:22:54.690851953Z", + "created_by": "/bin/sh -c #(nop) ADD file:a3bc1e842b69636f9df5256c49c5374fb4eef1e281fe3f282c65fb853ee171c5 in /" + }, + { + "created": "2015-10-31T22:22:55.613815829Z", + "created_by": "/bin/sh -c #(nop) CMD [\"sh\"]", + "empty_layer": true + }, + { + "created": "2015-10-31T22:22:56.329850019Z", + "created_by": "/bin/sh -c apk add curl" + } + ] +} +``` + +[rfc3339-s5.6]: https://tools.ietf.org/html/rfc3339#section-5.6 +[go-environment]: https://golang.org/doc/install/source#environment +[tar-split]: https://github.com/vbatts/tar-split diff --git a/.github/skills/regshape-dev/references/oci-image-spec-source/considerations.md b/.github/skills/regshape-dev/references/oci-image-spec-source/considerations.md new file mode 100644 index 0000000..1e069ba --- /dev/null +++ b/.github/skills/regshape-dev/references/oci-image-spec-source/considerations.md @@ -0,0 +1,153 @@ +# Considerations + +## Extensibility + +Implementations storing or copying content MUST NOT modify or alter the content in a way that would change the digest of the content. +Examples of these implementations include: + +- A [registry implementing the distribution specification][distribution-spec], including local registries, caching proxies +- An application which copies content to disk or between registries + +Implementations processing content SHOULD NOT generate an error if they encounter an unknown property in a known media type. +Examples of these implementations include: + +- A [runtime implementing the runtime specification][runtime-spec] +- An implementation using OCI to retrieve and utilize artifacts, e.g.: a WASM runtime + +## Canonicalization + +- OCI Images are [content-addressable](https://en.wikipedia.org/wiki/Content-addressable_storage). See [descriptors](descriptor.md) for more. +- One benefit of content-addressable storage is easy deduplication. +- Many images might depend on a particular [layer](layer.md), but there will only be one blob in the [store](image-layout.md). +- With a different serialization, that same semantic layer would have a different hash, and if both versions of the layer are referenced there will be two blobs with the same semantic content. +- To allow efficient storage, implementations serializing content for blobs SHOULD use a canonical serialization. +- This increases the chance that different implementations can push the same semantic content to the store without creating redundant blobs. + +## JSON + +Of the [OCI Image Format Specification media types](media-types.md), all the types ending in `+json` contain JSON content. +All [JSON][JSON] content MUST follow the I-JSON limitations in [RFC 7493][rfc7493]: + +- The encoding MUST be [UTF-8][rfc3629] +- Numbers SHOULD be limited to IEEE 754 double precision +- Objects MUST NOT contain duplicate names + +The order of entries in JSON objects is not significant. +Implementations MAY implement Canonical JSON documented in [RFC 8785][rfc8785]. + +## EBNF + +For field formats described in this specification, we use a limited subset of [Extended Backus-Naur Form][ebnf], similar to that used by the [XML specification][xmlebnf]. +Grammars present in the OCI specification are regular and can be converted to a single regular expressions. +However, regular expressions are avoided to limit ambiguity between regular expression syntax. +By defining a subset of EBNF used here, the possibility of variation, misunderstanding or ambiguities from linking to a larger specification can be avoided. + +Grammars are made up of rules in the following form: + +```ebnf +symbol ::= expression +``` + +We can say we have the production identified by symbol if the input is matched by the expression. +Whitespace is completely ignored in rule definitions. + +### Expressions + +The simplest expression is the literal, surrounded by quotes: + +```ebnf +literal ::= "matchthis" +``` + +The above expression defines a symbol, "literal", that matches the exact input of "matchthis". +Character classes are delineated by brackets (`[]`), describing either a set, range or multiple range of characters: + +```ebnf +set := [abc] +range := [A-Z] +``` + +The above symbol "set" would match one character of either "a", "b" or "c". +The symbol "range" would match any character, "A" to "Z", inclusive. +Currently, only matching for 7-bit ascii literals and character classes is defined, as that is all that is required by this specification. +Multiple character ranges and explicit characters can be specified in a single character classes, as follows: + +```ebnf +multipleranges := [a-zA-Z=-] +``` + +The above matches the characters in the range `A` to `Z`, `a` to `z` and the individual characters `-` and `=`. + +Expressions can be made up of one or more expressions, such that one must be followed by the other. +This is known as an implicit concatenation operator. +For example, to satisfy the following rule, both `A` and `B` must be matched to satisfy the rule: + +```ebnf +symbol ::= A B +``` + +Each expression must be matched once and only once, `A` followed by `B`. +To support the description of repetition and optional match criteria, the postfix operators `*` and `+` are defined. +`*` indicates that the preceding expression can be matched zero or more times. +`+` indicates that the preceding expression must be matched one or more times. +These appear in the following form: + +```ebnf +zeroormore ::= expression* +oneormore ::= expression+ +``` + +Parentheses are used to group expressions into a larger expression: + +```ebnf +group ::= (A B) +``` + +Like simpler expressions above, operators can be applied to groups, as well. +To allow for alternates, we also define the infix operator `|`. + +```ebnf +oneof ::= A | B +``` + +The above indicates that the expression should match one of the expressions, `A` or `B`. + +### Precedence + +The operator precedence is in the following order: + +- Terminals (literals and character classes) +- Grouping `()` +- Unary operators `+*` +- Concatenation +- Alternates `|` + +The precedence can be better described using grouping to show equivalents. +Concatenation has higher precedence than alternates, such `A B | C D` is equivalent to `(A B) | (C D)`. +Unary operators have higher precedence than alternates and concatenation, such that `A+ | B+` is equivalent to `(A+) | (B+)`. + +### Examples + +The following combines the previous definitions to match a simple, relative path name, describing the individual components: + +```ebnf +path ::= component ("/" component)* +component ::= [a-z]+ +``` + +The production "component" is one or more lowercase letters. +A "path" is then at least one component, possibly followed by zero or more slash-component pairs. +The above can be converted into the following regular expression: + +```regex +[a-z]+(?:/[a-z]+)* +``` + +[distribution-spec]: https://github.com/opencontainers/distribution-spec/blob/main/spec.md +[ebnf]: https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form +[JSON]: https://json.org/ +[rfc3629]: https://datatracker.ietf.org/doc/html/rfc3629 +[rfc7493]: https://datatracker.ietf.org/doc/html/rfc7493 +[rfc8785]: https://datatracker.ietf.org/doc/html/rfc8785 +[runtime-spec]: https://github.com/opencontainers/runtime-spec/blob/main/spec.md +[xmlebnf]: https://www.w3.org/TR/REC-xml/#sec-notation diff --git a/.github/skills/regshape-dev/references/oci-image-spec-source/conversion.md b/.github/skills/regshape-dev/references/oci-image-spec-source/conversion.md new file mode 100644 index 0000000..3c707c8 --- /dev/null +++ b/.github/skills/regshape-dev/references/oci-image-spec-source/conversion.md @@ -0,0 +1,130 @@ +# Conversion to OCI Runtime Configuration + +When extracting an OCI Image into an [OCI Runtime bundle][oci-runtime-bundle], two orthogonal components of the extraction are relevant: + +1. Extraction of the root filesystem from the set of [filesystem layers](layer.md). +2. Conversion of the [image configuration blob](config.md) to an [OCI Runtime configuration blob][oci-runtime-config]. + +This section defines how to convert an `application/vnd.oci.image.config.v1+json` blob to an [OCI runtime configuration blob][oci-runtime-config] (the latter component of extraction). +The former component of extraction is defined [elsewhere](layer.md) and is orthogonal to configuration of a runtime bundle. +The values of runtime configuration properties not specified by this document are implementation-defined. + +A converter MUST rely on the OCI image configuration to build the OCI runtime configuration as described by this document; this will create the "default generated runtime configuration". + +The "default generated runtime configuration" MAY be overridden or combined with externally provided inputs from the caller. +In addition, a converter MAY have its own implementation-defined defaults and extensions which MAY be combined with the "default generated runtime configuration". +The restrictions in this document refer only to combining implementation-defined defaults with the "default generated runtime configuration". +Externally provided inputs are considered to be a modification of the `application/vnd.oci.image.config.v1+json` used as a source, and such modifications have no restrictions. + +For example, externally provided inputs MAY cause an environment variable to be added, removed or changed. +However an implementation-defined default SHOULD NOT result in an environment variable being removed or changed. + +## Verbatim Fields + +Certain image configuration fields have an identical counterpart in the runtime configuration. +Some of these are purely annotation-based fields, and have been extracted into a [separate subsection](#annotation-fields). +A compliant configuration converter MUST extract the following fields verbatim to the corresponding field in the generated runtime configuration: + +| Image Field | Runtime Field | Notes | +| ------------------- | --------------- | ----- | +| `Config.WorkingDir` | `process.cwd` | | +| `Config.Env` | `process.env` | 1 | +| `Config.Entrypoint` | `process.args` | 2 | +| `Config.Cmd` | `process.args` | 2 | + +1. The converter MAY add additional entries to `process.env` but it SHOULD NOT add entries that have variable names present in `Config.Env`. +2. If both `Config.Entrypoint` and `Config.Cmd` are specified, the converter MUST append the value of `Config.Cmd` to the value of `Config.Entrypoint` and set `process.args` to that combined value. + +### Annotation Fields + +These fields all affect the `annotations` of the runtime configuration, and are thus subject to [precedence](#annotations). + +| Image Field | Runtime Field | Notes | +| ------------------- | --------------- | ----- | +| `os` | `annotations` | 1,2 | +| `architecture` | `annotations` | 1,3 | +| `variant` | `annotations` | 1,4 | +| `os.version` | `annotations` | 1,5 | +| `os.features` | `annotations` | 1,6 | +| `author` | `annotations` | 1,7 | +| `created` | `annotations` | 1,8 | +| `Config.Labels` | `annotations` | | +| `Config.StopSignal` | `annotations` | 1,9 | + +1. If a user has explicitly specified this annotation with `Config.Labels`, then the value specified in this field takes lower [precedence](#annotations) and the converter MUST instead use the value from `Config.Labels`. +2. The value of this field MUST be set as the value of `org.opencontainers.image.os` in `annotations`. +3. The value of this field MUST be set as the value of `org.opencontainers.image.architecture` in `annotations`. +4. The value of this field MUST be set as the value of `org.opencontainers.image.variant` in `annotations`. +5. The value of this field MUST be set as the value of `org.opencontainers.image.os.version` in `annotations`. +6. The value of this field MUST be set as the value of `org.opencontainers.image.os.features` in `annotations`. +7. The value of this field MUST be set as the value of `org.opencontainers.image.author` in `annotations`. +8. The value of this field MUST be set as the value of `org.opencontainers.image.created` in `annotations`. +9. The value of this field MUST be set as the value of `org.opencontainers.image.stopSignal` in `annotations`. + +## Parsed Fields + +Certain image configuration fields have a counterpart that must first be translated. +A compliant configuration converter SHOULD parse all of these fields and set the corresponding fields in the generated runtime configuration: + +| Image Field | Runtime Field | +| ------------------- | --------------- | +| `Config.User` | `process.user.*` | + +The method of parsing the above image fields are described in the following sections. + +### `Config.User` + +If the values of [`user` or `group`](config.md#properties) in `Config.User` are numeric (`uid` or `gid`) then the values MUST be copied verbatim to `process.user.uid` and `process.user.gid` respectively. +If the values of [`user` or `group`](config.md#properties) in `Config.User` are not numeric (`user` or `group`) then a converter SHOULD resolve the user information using a method appropriate for the container's context. +For Unix-like systems, this MAY involve resolution through NSS or parsing `/etc/passwd` from the extracted container's root filesystem to determine the values of `process.user.uid` and `process.user.gid`. + +In addition, a converter SHOULD set the value of `process.user.additionalGids` to a value corresponding to the user in the container's context described by `Config.User`. +For Unix-like systems, this MAY involve resolution through NSS or parsing `/etc/group` and determining the group memberships of the user specified in `process.user.uid`. +The converter SHOULD NOT modify `process.user.additionalGids` if the value of [`user`](config.md#properties) in `Config.User` is numeric or if `Config.User` specifies a group. + +If `Config.User` is not defined, the converted `process.user` value is implementation-defined. +If `Config.User` does not correspond to a user in the container's context, the converter MUST return an error. + +## Optional Fields + +Certain image configuration fields are not applicable to all conversion use cases, and thus are optional for configuration converters to implement. +A compliant configuration converter SHOULD provide a way for users to extract these fields into the generated runtime configuration: + +| Image Field | Runtime Field | Notes | +| --------------------- | ------------------ | ----- | +| `Config.ExposedPorts` | `annotations` | 1 | +| `Config.Volumes` | `mounts` | 2 | + +1. The runtime configuration does not have a corresponding field for this image field. + However, converters SHOULD set the [`org.opencontainers.image.exposedPorts` annotation](#configexposedports). +2. Implementations SHOULD provide mounts for these locations such that application data is not written to the container's root filesystem. + If a converter implements conversion for this field using mountpoints, it SHOULD set the `destination` of the mountpoint to the value specified in `Config.Volumes`. + An implementation MAY seed the contents of the mount with data in the image at the same location. + If a _new_ image is created from a container based on the image described by this configuration, data in these paths SHOULD NOT be included in the _new_ image. + The other `mounts` fields are platform and context dependent, and thus are implementation-defined. + Note that the implementation of `Config.Volumes` need not use mountpoints, as it is effectively a mask of the filesystem. + +### `Config.ExposedPorts` + +The OCI runtime configuration does not provide a way of expressing the concept of "container exposed ports". +However, converters SHOULD set the **org.opencontainers.image.exposedPorts** annotation, unless doing so will [cause a conflict](#annotations). + +**org.opencontainers.image.exposedPorts** is the list of values that correspond to the [keys defined for `Config.ExposedPorts`](config.md) (string, comma-separated values). + +## Annotations + +There are three ways of annotating an OCI image in this specification: + +1. `Config.Labels` in the [configuration](config.md) of the image. +2. `annotations` in the [manifest](manifest.md) of the image. +3. `annotations` in the [image index](image-index.md) of the image. + +In addition, there are also implicit annotations that are defined by this section which are determined from the values of the image configuration. +A converter SHOULD NOT attempt to extract annotations from [manifests](manifest.md) or [image indices](image-index.md). +If there is a conflict (same key but different value) between an implicit annotation (or annotation in [manifests](manifest.md) or [image indices](image-index.md)) and an explicitly specified annotation in `Config.Labels`, the value specified in `Config.Labels` MUST take precedence. + +A converter MAY add annotations which have keys not specified in the image. +A converter MUST NOT modify the values of annotations specified in the image. + +[oci-runtime-bundle]: https://github.com/opencontainers/runtime-spec/blob/main/bundle.md +[oci-runtime-config]: https://github.com/opencontainers/runtime-spec/blob/main/config.md diff --git a/.github/skills/regshape-dev/references/oci-image-spec-source/descriptor.md b/.github/skills/regshape-dev/references/oci-image-spec-source/descriptor.md new file mode 100644 index 0000000..56c76d6 --- /dev/null +++ b/.github/skills/regshape-dev/references/oci-image-spec-source/descriptor.md @@ -0,0 +1,236 @@ +# OCI Content Descriptors + +- An OCI image consists of several different components, arranged in a [Merkle Directed Acyclic Graph (DAG)](https://en.wikipedia.org/wiki/Merkle_tree). +- References between components in the graph are expressed through _Content Descriptors_. +- A Content Descriptor (or simply _Descriptor_) describes the disposition of the targeted content. +- A Content Descriptor includes the type of the content, a content identifier (_digest_), and the byte-size of the raw content. + Optionally, it includes the type of artifact it is describing. +- Descriptors SHOULD be embedded in other formats to securely reference external content. +- Other formats SHOULD use descriptors to securely reference external content. +- When other formats contain multiple descriptors, unless otherwise specified, those descriptors are independent of each other, allowing fields like the `mediaType` and the algorithm for the `digest` to vary within that external content. + +This section defines the `application/vnd.oci.descriptor.v1+json` [media type](media-types.md). + +## Properties + +A descriptor consists of a set of properties encapsulated in key-value fields. + +The following fields contain the primary properties that constitute a Descriptor: + +- **`mediaType`** *string* + + This REQUIRED property contains the media type of the referenced content. + Values MUST comply with [RFC 6838][rfc6838], including the [naming requirements in its section 4.2][rfc6838-s4.2]. + + The OCI image specification defines [several of its own MIME types](media-types.md) for resources defined in the specification. + +- **`digest`** *string* + + This REQUIRED property is the _digest_ of the targeted content, conforming to the requirements outlined in [Digests](#digests). + Retrieved content SHOULD be verified against this digest when consumed via untrusted sources. + +- **`size`** *int64* + + This REQUIRED property specifies the size, in bytes, of the raw content. + This property exists so that a client will have an expected size for the content before processing. + If the length of the retrieved content does not match the specified length, the content SHOULD NOT be trusted. + The size MUST NOT be negative. + +- **`urls`** *array of strings* + + This OPTIONAL property specifies a list of URIs from which this object MAY be downloaded. + Each entry MUST conform to [RFC 3986][rfc3986]. + Entries SHOULD use the `http` and `https` schemes, as defined in [RFC 7230][rfc7230-s2.7]. + +- **`annotations`** *string-string map* + + This OPTIONAL property contains arbitrary metadata for this descriptor. + This OPTIONAL property MUST use the [annotation rules](annotations.md#rules). + +- **`data`** *string* + + This OPTIONAL property contains an embedded representation of the referenced content. + Values MUST conform to the Base 64 encoding, as defined in [RFC 4648][rfc4648-s4]. + The decoded data MUST be identical to the referenced content and SHOULD be verified against the [`digest`](#digests) and `size` fields by content consumers. + See [Embedded Content](#embedded-content) for when this is appropriate. + +- **`artifactType`** *string* + + This OPTIONAL property contains the type of an artifact when the descriptor points to an artifact. + This is the value of the config descriptor `mediaType` when the descriptor references an [image manifest](manifest.md). + If defined, the value MUST comply with [RFC 6838][rfc6838], including the [naming requirements in its section 4.2][rfc6838-s4.2], and MAY be registered with [IANA][iana]. + +Descriptors pointing to [`application/vnd.oci.image.manifest.v1+json`](manifest.md) SHOULD include the extended field `platform`, see [Image Index Property Descriptions](image-index.md#image-index-property-descriptions) for details. + +### Reserved + +Extended _Descriptor_ field additions proposed in other OCI specifications SHOULD first be considered for addition into this specification. + +## Digests + +The _digest_ property of a Descriptor acts as a content identifier, enabling [content addressability](https://en.wikipedia.org/wiki/Content-addressable_storage). +It uniquely identifies content by taking a [collision-resistant hash](https://en.wikipedia.org/wiki/Cryptographic_hash_function) of the bytes. +If the _digest_ can be communicated in a secure manner, one can verify content from an insecure source by recalculating the digest independently, ensuring the content has not been modified. + +The value of the `digest` property is a string consisting of an _algorithm_ portion and an _encoded_ portion. +The _algorithm_ specifies the cryptographic hash function and encoding used for the digest; the _encoded_ portion contains the encoded result of the hash function. + +A digest string MUST match the following [grammar](considerations.md#ebnf): + +```ebnf +digest ::= algorithm ":" encoded +algorithm ::= algorithm-component (algorithm-separator algorithm-component)* +algorithm-component ::= [a-z0-9]+ +algorithm-separator ::= [+._-] +encoded ::= [a-zA-Z0-9=_-]+ +``` + +Note that _algorithm_ MAY impose algorithm-specific restriction on the grammar of the _encoded_ portion. +See also [Registered Algorithms](#registered-algorithms). + +Some example digest strings include the following: + +| digest | algorithm | Registered | +|---------------------------------------------------------------------------|-----------------------------|------------| +| `sha256:6c3c624b58dbbcd3c0dd82b4c53f04194d1247c6eebdaab7c610cf7d66709b3b` | [SHA-256](#sha-256) | Yes | +| `sha512:401b09eab3c013d4ca54922bb802bec8fd5318192b0a75f201d8b372742...` | [SHA-512](#sha-512) | Yes | +| `multihash+base58:QmRZxt2b1FVZPNqd8hsiykDL3TdBDeTSPX9Kv46HmX4Gx8` | Multihash | No | +| `sha256+b64u:LCa0a2j_xo_5m0U8HTBBNBNCLXBkg7-g-YpeiGJm564` | SHA-256 with urlsafe base64 | No | +| `blake3:6c3c624b58dbbcd3c0dd82b4c53f04194d1247c6eebdaab7c610cf7d66709b3b` | [BLAKE3](#blake3) | Yes | + +Please see [Registered Algorithms](#registered-algorithms) for a list of registered algorithms. + +Implementations SHOULD allow digests with unrecognized algorithms to pass validation if they comply with the above grammar. +While `sha256` will only use hex encoded digests, separators in _algorithm_ and alphanumerics in _encoded_ are included to allow for extensions. +As an example, we can parameterize the encoding and algorithm as `multihash+base58:QmRZxt2b1FVZPNqd8hsiykDL3TdBDeTSPX9Kv46HmX4Gx8`, which would be considered valid but unregistered by this specification. + +### Verification + +Before consuming content targeted by a descriptor from untrusted sources, the byte content SHOULD be verified against the digest string. +Before calculating the digest, the size of the content SHOULD be verified to reduce hash collision space. +Heavy processing before calculating a hash SHOULD be avoided. +Implementations MAY employ [canonicalization](considerations.md#canonicalization) of the underlying content to ensure stable content identifiers. + +### Digest calculations + +A _digest_ is calculated by the following pseudo-code, where `H` is the selected hash algorithm, identified by string ``: + +```text +let ID(C) = Descriptor.digest +let C = +let D = ':' + Encode(H(C)) +let verified = ID(C) == D +``` + +Above, we define the content identifier as `ID(C)`, extracted from the `Descriptor.digest` field. +Content `C` is a string of bytes. +Function `H` returns the hash of `C` in bytes and is passed to function `Encode` and prefixed with the algorithm to obtain the digest. +The result `verified` is true if `ID(C)` is equal to `D`, confirming that `C` is the content identified by `D`. +After verification, the following is true: + +```text +D == ID(C) == ':' + Encode(H(C)) +``` + +The _digest_ is confirmed as the content identifier by independently calculating the _digest_. + +### Registered algorithms + +While the _algorithm_ component of the digest string allows the use of a variety of cryptographic algorithms, compliant implementations SHOULD use [SHA-256](#sha-256). + +The following algorithm identifiers are currently defined by this specification: + +| algorithm identifier | algorithm | +|----------------------|---------------------| +| `sha256` | [SHA-256](#sha-256) | +| `sha512` | [SHA-512](#sha-512) | +| `blake3` | [BLAKE3](#blake3) | + +If a useful algorithm is not included in the above table, it SHOULD be submitted to this specification for registration. + +#### SHA-256 + +[SHA-256][rfc4634-s4.1] is a collision-resistant hash function, chosen for ubiquity, reasonable size and secure characteristics. +Implementations MUST implement SHA-256 digest verification for use in descriptors. + +When the _algorithm identifier_ is `sha256`, the _encoded_ portion MUST match `/[a-f0-9]{64}/`. +Note that `[A-F]` MUST NOT be used here. + +#### SHA-512 + +[SHA-512][rfc4634-s4.2] is a collision-resistant hash function which [may be more performant][sha256-vs-sha512] than [SHA-256](#sha-256) on some CPUs. +Implementations MAY implement SHA-512 digest verification for use in descriptors. + +When the _algorithm identifier_ is `sha512`, the _encoded_ portion MUST match `/[a-f0-9]{128}/`. +Note that `[A-F]` MUST NOT be used here. + +#### BLAKE3 + +[BLAKE3][blake3] is a high performance, highly parallelizable, collision-resistant hash function which [is more performant][blake3-vs-sha2] than [SHA-256][rfc4634-s4.1]. +The hash output length MUST be 256 bits. +Implementations MAY implement BLAKE3 digest verification for use in descriptors. + +When the _algorithm identifier_ is `blake3`, the _encoded_ portion MUST match `/[a-f0-9]{64}/`. +Note that `[A-F]` MUST NOT be used here. + +## Embedded Content + +In many contexts, such as when downloading content over a network, resolving a descriptor to its content has a measurable fixed "roundtrip" latency cost. +For large blobs, the fixed cost is usually inconsequential, as the majority of time will be spent actually fetching the content. +For very small blobs, the fixed cost can be quite significant. + +Implementations MAY choose to embed small pieces of content directly within a descriptor to avoid roundtrips. + +Implementations MUST NOT populate the `data` field in situations where doing so would modify existing content identifiers. +For example, a registry MUST NOT arbitrarily populate `data` fields within uploaded manifests, as that would modify the content identifier of those manifests. +In contrast, a client MAY populate the `data` field before uploading a manifest, because the manifest would not yet have a content identifier in the registry. + +Implementations SHOULD consider portability when deciding whether to embed data, as some providers are known to refuse to accept or parse manifests that exceed a certain size. + +## Examples + +The following example describes a [_Manifest_](manifest.md#image-manifest) with a content identifier of "sha256:5b0bcabd1ed22e9fb1310cf6c2dec7cdef19f0ad69efa1f392e94a4333501270" and a size of 7682 bytes: + +```json,title=Content%20Descriptor&mediatype=application/vnd.oci.descriptor.v1%2Bjson +{ + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "size": 7682, + "digest": "sha256:5b0bcabd1ed22e9fb1310cf6c2dec7cdef19f0ad69efa1f392e94a4333501270" +} +``` + +In the following example, the descriptor indicates that the referenced manifest is retrievable from a particular URL: + +```json,title=Content%20Descriptor&mediatype=application/vnd.oci.descriptor.v1%2Bjson +{ + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "size": 7682, + "digest": "sha256:5b0bcabd1ed22e9fb1310cf6c2dec7cdef19f0ad69efa1f392e94a4333501270", + "urls": [ + "https://example.com/example-manifest" + ] +} +``` + +In the following example, the descriptor indicates the type of artifact it is referencing: + +```json,title=Content%20Descriptor&mediatype=application/vnd.oci.descriptor.v1%2Bjson +{ + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "size": 123, + "digest": "sha256:87923725d74f4bfb94c9e86d64170f7521aad8221a5de834851470ca142da630", + "artifactType": "application/vnd.example.sbom.v1" +} +``` + +[rfc3986]: https://tools.ietf.org/html/rfc3986 +[rfc4634-s4.1]: https://tools.ietf.org/html/rfc4634#section-4.1 +[rfc4634-s4.2]: https://tools.ietf.org/html/rfc4634#section-4.2 +[rfc4648-s4]: https://tools.ietf.org/html/rfc4648#section-4 +[rfc6838]: https://tools.ietf.org/html/rfc6838 +[rfc6838-s4.2]: https://tools.ietf.org/html/rfc6838#section-4.2 +[rfc7230-s2.7]: https://tools.ietf.org/html/rfc7230#section-2.7 +[sha256-vs-sha512]: https://groups.google.com/a/opencontainers.org/forum/#!topic/dev/hsMw7cAwrZE +[iana]: https://www.iana.org/assignments/media-types/media-types.xhtml +[blake3]: https://github.com/C2SP/C2SP/blob/BLAKE3/v1.0.0/BLAKE3.md +[blake3-vs-sha2]: https://github.com/BLAKE3-team/BLAKE3-specs/blob/master/blake3.pdf diff --git a/.github/skills/regshape-dev/references/oci-image-spec-source/image-index.md b/.github/skills/regshape-dev/references/oci-image-spec-source/image-index.md new file mode 100644 index 0000000..7cd44ea --- /dev/null +++ b/.github/skills/regshape-dev/references/oci-image-spec-source/image-index.md @@ -0,0 +1,190 @@ +# OCI Image Index Specification + +The image index is a higher-level manifest which points to specific [image manifests](manifest.md), ideal for one or more platforms. +While the use of an image index is OPTIONAL for image providers, image consumers SHOULD be prepared to process them. + +This section defines the `application/vnd.oci.image.index.v1+json` [media type](media-types.md). + +For the media type(s) that this document is compatible with, see the [matrix][matrix]. + +## _Image Index_ Property Descriptions + +- **`schemaVersion`** *int* + + This REQUIRED property specifies the image manifest schema version. + For this version of the specification, this MUST be `2` to ensure backward compatibility with older versions of Docker. + The value of this field will not change. + This field MAY be removed in a future version of the specification. + +- **`mediaType`** *string* + + This property SHOULD be used and [remain compatible][matrix] with earlier versions of this specification and with other similar external formats. + When used, this field MUST contain the media type `application/vnd.oci.image.index.v1+json`. + This field usage differs from the [descriptor](descriptor.md#properties) use of `mediaType`. + +- **`artifactType`** *string* + + This OPTIONAL property contains the type of an artifact when the manifest is used for an artifact. + If defined, the value MUST comply with [RFC 6838][rfc6838], including the [naming requirements in its section 4.2][rfc6838-s4.2], and MAY be registered with [IANA][iana]. + +- **`manifests`** *array of objects* + + This REQUIRED property contains a list of [manifests](manifest.md) for specific platforms. + While this property MUST be present, the size of the array MAY be zero. + + Each object in `manifests` includes a set of [descriptor properties](descriptor.md#properties) with the following additional properties and restrictions: + + - **`mediaType`** *string* + + This [descriptor property](descriptor.md#properties) has additional restrictions for `manifests`. + Implementations MUST support at least the following media types: + + - [`application/vnd.oci.image.manifest.v1+json`](manifest.md) + + Also, implementations SHOULD support the following media types: + + - `application/vnd.oci.image.index.v1+json` (nested index) + + Image indexes concerned with portability SHOULD use one of the above media types. + Future versions of the spec MAY use a different mediatype (i.e. a new versioned format). + An encountered `mediaType` that is unknown to the implementation MUST NOT generate an error. + + - **`platform`** *object* + + This OPTIONAL property describes the minimum runtime requirements of the image. + This property SHOULD be present if its target is platform-specific. + + - **`architecture`** *string* + + This REQUIRED property specifies the CPU architecture. + Image indexes SHOULD use, and implementations SHOULD understand, values listed in the Go Language document for [`GOARCH`][go-environment2]. + + - **`os`** *string* + + This REQUIRED property specifies the operating system. + Image indexes SHOULD use, and implementations SHOULD understand, values listed in the Go Language document for [`GOOS`][go-environment2]. + + - **`os.version`** *string* + + This OPTIONAL property specifies the version of the operating system targeted by the referenced blob. + Implementations MAY refuse to use manifests where `os.version` is not known to work with the host OS version. + Valid values are implementation-defined. e.g. `10.0.14393.1066` on `windows`. + + - **`os.features`** *array of strings* + + This OPTIONAL property specifies an array of strings, each specifying a mandatory OS feature. + When `os` is `windows`, image indexes SHOULD use, and implementations SHOULD understand the following values: + + - `win32k`: image requires `win32k.sys` on the host (Note: `win32k.sys` is missing on Nano Server) + + When `os` is not `windows`, values are implementation-defined and SHOULD be submitted to this specification for standardization. + + - **`variant`** *string* + + This OPTIONAL property specifies the variant of the CPU. + Image indexes SHOULD use, and implementations SHOULD understand, `variant` values listed in the [Platform Variants](#platform-variants) table. + + - **`features`** *array of strings* + + This property is RESERVED for future versions of the specification. + + If multiple manifests match a client or runtime's requirements, the first matching entry SHOULD be used. + +- **`subject`** *[descriptor](descriptor.md)* + + This OPTIONAL property specifies a [descriptor](descriptor.md) of another manifest. + This value defines a weak association to a separate [Merkle Directed Acyclic Graph (DAG)][dag] structure, and is used by the [`referrers` API][referrers-api] to include this manifest in the list of responses for the subject digest. + +- **`annotations`** *string-string map* + + This OPTIONAL property contains arbitrary metadata for the image index. + This OPTIONAL property MUST use the [annotation rules](annotations.md#rules). + + See [Pre-Defined Annotation Keys](annotations.md#pre-defined-annotation-keys). + +## Platform Variants + +When the variant of the CPU is not listed in the table, values are implementation-defined and SHOULD be submitted to this specification for standardization. +These values SHOULD match (or be similar to) their analog listed in [the Go Language document][go-environment2]. + +| ISA/ABI | `architecture` | `variant` | Go analog | +|------------|----------------|-----------------------|-------------| +| ARM 32-bit | `arm` | `v6`, `v7`, `v8` | `GOARM` | +| ARM 64-bit | `arm64` | `v8`, `v8.1`, … | `GOARM64` | +| POWER8+ | `ppc64le` | `power8`, `power9`, … | `GOPPC64` | +| RISC-V | `riscv64` | `rva20u64`, … | `GORISCV64` | +| x86-64 | `amd64` | `v1`, `v2`, `v3`, … | `GOAMD64` | + +## Example Image Index + +*Example showing a simple image index pointing to image manifests for two platforms:* + +```json,title=Image%20Index&mediatype=application/vnd.oci.image.index.v1%2Bjson +{ + "schemaVersion": 2, + "mediaType": "application/vnd.oci.image.index.v1+json", + "manifests": [ + { + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "size": 7143, + "digest": "sha256:e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f", + "platform": { + "architecture": "ppc64le", + "os": "linux" + } + }, + { + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "size": 7682, + "digest": "sha256:5b0bcabd1ed22e9fb1310cf6c2dec7cdef19f0ad69efa1f392e94a4333501270", + "platform": { + "architecture": "amd64", + "os": "linux" + } + } + ], + "annotations": { + "com.example.key1": "value1", + "com.example.key2": "value2" + } +} +``` + +## Example Image Index with multiple media types + +*Example showing an image index pointing to manifests with multiple media types:* + +```json,title=Image%20Index&mediatype=application/vnd.oci.image.index.v1%2Bjson +{ + "schemaVersion": 2, + "mediaType": "application/vnd.oci.image.index.v1+json", + "manifests": [ + { + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "size": 7143, + "digest": "sha256:e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f", + "platform": { + "architecture": "ppc64le", + "os": "linux" + } + }, + { + "mediaType": "application/vnd.oci.image.index.v1+json", + "size": 7682, + "digest": "sha256:601570aaff1b68a61eb9c85b8beca1644e698003e0cdb5bce960f193d265a8b7" + } + ], + "annotations": { + "com.example.key1": "value1", + "com.example.key2": "value2" + } +} +``` + +[dag]: https://en.wikipedia.org/wiki/Merkle_tree +[go-environment2]: https://golang.org/doc/install/source#environment +[iana]: https://www.iana.org/assignments/media-types/media-types.xhtml +[matrix]: media-types.md#compatibility-matrix +[referrers-api]: https://github.com/opencontainers/distribution-spec/blob/main/spec.md#listing-referrers +[rfc6838]: https://tools.ietf.org/html/rfc6838 +[rfc6838-s4.2]: https://tools.ietf.org/html/rfc6838#section-4.2 diff --git a/.github/skills/regshape-dev/references/oci-image-spec-source/image-layout.md b/.github/skills/regshape-dev/references/oci-image-spec-source/image-layout.md new file mode 100644 index 0000000..026e71e --- /dev/null +++ b/.github/skills/regshape-dev/references/oci-image-spec-source/image-layout.md @@ -0,0 +1,213 @@ +# OCI Image Layout Specification + +- The OCI Image Layout is the directory structure for OCI content-addressable blobs and [location-addressable](https://en.wikipedia.org/wiki/Content-addressable_storage#Content-addressed_vs._location-addressed) references (refs). +- This layout MAY be used in a variety of different transport mechanisms: archive formats (e.g. tar, zip), shared filesystem environments (e.g. nfs), or networked file fetching (e.g. http, ftp, rsync). + +Given an image layout and a ref, a tool can create an [OCI Runtime Specification bundle](https://github.com/opencontainers/runtime-spec/blob/main/bundle.md) by: + +- Following the ref to find a [manifest](manifest.md#image-manifest), possibly via an [image index](image-index.md) +- [Applying the filesystem layers](layer.md#applying-changesets) in the specified order +- Converting the [image configuration](config.md) into an [OCI Runtime Specification `config.json`](https://github.com/opencontainers/runtime-spec/blob/main/config.md) + +## Content + +The image layout is as follows: + +- `blobs` directory + - Contains content-addressable blobs + - A blob has no schema and SHOULD be considered opaque + - Directory MUST exist and MAY be empty + - See [blobs](#blobs) section +- `oci-layout` file + - It MUST exist + - It MUST be a JSON object + - It MUST contain an `imageLayoutVersion` field + - See [oci-layout file](#oci-layout-file) section + - It MAY include additional fields +- `index.json` file + - It MUST exist + - It MUST be an [image index](image-index.md) JSON object. + - See [index.json](#indexjson-file) section + +**Implementor's Note:** +For extensibility and future expansion, additional files may be included in the directory. +Implementations should not error when encountering unknown files. +A common usage includes the `manifest.json` file associated with a backwards compatible `docker save` format. + +## Example Layout + +This is an example image layout: + +```shell +$ cd example.com/app/ +$ find . -type f +./index.json +./oci-layout +./blobs/sha256/3588d02542238316759cbf24502f4344ffcc8a60c803870022f335d1390c13b4 +./blobs/sha256/4b0bc1c4050b03c95ef2a8e36e25feac42fd31283e8c30b3ee5df6b043155d3c +./blobs/sha256/7968321274dc6b6171697c33df7815310468e694ac5be0ec03ff053bb135e768 +``` + +Blobs are named by their contents: + +```shell +$ shasum -a 256 ./blobs/sha256/afff3924849e458c5ef237db5f89539274d5e609db5db935ed3959c90f1f2d51 +afff3924849e458c5ef237db5f89539274d5e609db5db935ed3959c90f1f2d51 ./blobs/sha256/afff3924849e458c5ef237db5f89539274d5e609db5db935ed3959c90f1f2d51 +``` + +## Blobs + +- Object names in the `blobs` subdirectories are composed of a directory for each hash algorithm, the children of which will contain the actual content. +- The content of `blobs//` MUST match the digest `:` (referenced per [descriptor](descriptor.md#digests)). For example, the content of `blobs/sha256/da39a3ee5e6b4b0d3255bfef95601890afd80709` MUST match the digest `sha256:da39a3ee5e6b4b0d3255bfef95601890afd80709`. +- The character set of the entry name for `` and `` MUST match the respective grammar elements described in [descriptor](descriptor.md#digests). +- The blobs directory MAY contain blobs which are not referenced by any of the [refs](#indexjson-file). +- The blobs directory MAY be missing referenced blobs, in which case the missing blobs SHOULD be fulfilled by an external blob store. + +### Example Blobs + +```shell +$ cat ./blobs/sha256/9b97579de92b1c195b85bb42a11011378ee549b02d7fe9c17bf2a6b35d5cb079 | jq +{ + "schemaVersion": 2, + "manifests": [ + { + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "size": 7143, + "digest": "sha256:afff3924849e458c5ef237db5f89539274d5e609db5db935ed3959c90f1f2d51", + "platform": { + "architecture": "ppc64le", + "os": "linux" + } + }, +... +``` + +```shell +$ cat ./blobs/sha256/afff3924849e458c5ef237db5f89539274d5e609db5db935ed3959c90f1f2d51 | jq +{ + "schemaVersion": 2, + "config": { + "mediaType": "application/vnd.oci.image.config.v1+json", + "size": 7023, + "digest": "sha256:5b0bcabd1ed22e9fb1310cf6c2dec7cdef19f0ad69efa1f392e94a4333501270" + }, + "layers": [ + { + "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", + "size": 32654, + "digest": "sha256:9834876dcfb05cb167a5c24953eba58c4ac89b1adf57f28f2f9d09af107ee8f0" + }, +... +``` + +```shell +$ cat ./blobs/sha256/5b0bcabd1ed22e9fb1310cf6c2dec7cdef19f0ad69efa1f392e94a4333501270 | jq +{ + "architecture": "amd64", + "author": "Alyssa P. Hacker ", + "config": { + "Hostname": "8dfe43d80430", + "Domainname": "", + "User": "", + "AttachStdin": false, + "AttachStdout": false, + "AttachStderr": false, + "Tty": false, + "OpenStdin": false, + "StdinOnce": false, + "Env": [ + "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" + ], + "Cmd": null, + "Image": "sha256:6986ae504bbf843512d680cc959484452034965db15f75ee8bdd1b107f61500b", +... +``` + +```shell +$ cat ./blobs/sha256/9834876dcfb05cb167a5c24953eba58c4ac89b1adf57f28f2f9d09af107ee8f0 +[gzipped tar stream] +``` + +## oci-layout file + +This JSON object serves as a marker for the base of an Open Container Image Layout and to provide the version of the image-layout in use. +The `imageLayoutVersion` value will align with the OCI Image Specification version at the time changes to the layout are made, and will pin a given version until changes to the image layout are required. +This section defines the `application/vnd.oci.layout.header.v1+json` [media type](media-types.md). + +### oci-layout Example + +```json,title=OCI%20Layout&mediatype=application/vnd.oci.layout.header.v1%2Bjson +{ + "imageLayoutVersion": "1.0.0" +} +``` + +## index.json file + +This REQUIRED file is the entry point for references and descriptors of the image-layout. +The [image index](image-index.md) is a multi-descriptor entry point. + +This index provides an established path (`/index.json`) to have an entry point for an image-layout and to discover auxiliary descriptors. + +- No semantic restriction is given for the "org.opencontainers.image.ref.name" annotation of descriptors. +- In general the `mediaType` of each [descriptor][descriptors] object in the `manifests` field will be either `application/vnd.oci.image.index.v1+json` or `application/vnd.oci.image.manifest.v1+json`. +- Future versions of the spec MAY use a different mediatype (i.e. a new versioned format). +- An encountered `mediaType` that is unknown MUST NOT generate an error. + +**Implementor's Note:** +A common use case of descriptors with a "org.opencontainers.image.ref.name" annotation is representing a "tag" for a container image. +For example, an image may have a tag for different versions or builds of the software. +In the wild you often see "tags" like "v1.0.0-vendor.0", "2.0.0-debug", etc. +Those tags will often be represented in an image-layout repository with matching "org.opencontainers.image.ref.name" annotations like "v1.0.0-vendor.0", "2.0.0-debug", etc. + +### Index Example + +```json,title=Image%20Index&mediatype=application/vnd.oci.image.index.v1%2Bjson +{ + "schemaVersion": 2, + "mediaType": "application/vnd.oci.image.index.v1+json", + "manifests": [ + { + "mediaType": "application/vnd.oci.image.index.v1+json", + "size": 7143, + "digest": "sha256:0228f90e926ba6b96e4f39cf294b2586d38fbb5a1e385c05cd1ee40ea54fe7fd", + "annotations": { + "org.opencontainers.image.ref.name": "stable-release" + } + }, + { + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "size": 7143, + "digest": "sha256:e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f", + "platform": { + "architecture": "ppc64le", + "os": "linux" + }, + "annotations": { + "org.opencontainers.image.ref.name": "v1.0" + } + }, + { + "mediaType": "application/xml", + "size": 7143, + "digest": "sha256:b3d63d132d21c3ff4c35a061adf23cf43da8ae054247e32faa95494d904a007e", + "annotations": { + "org.freedesktop.specifications.metainfo.version": "1.0", + "org.freedesktop.specifications.metainfo.type": "AppStream" + } + } + ], + "annotations": { + "com.example.index.revision": "r124356" + } +} +``` + +This illustrates an index that provides two named references and an auxiliary mediatype for this image layout. + +The first named reference (`stable-release`) points to another index that might contain multiple references with distinct platforms and annotations. +Note that the [`org.opencontainers.image.ref.name` annotation](annotations.md) SHOULD only be considered valid when on descriptors on `index.json`. + +The second named reference (`v1.0`) points to a manifest that is specific to the linux/ppc64le platform. + +[descriptors]: ./descriptor.md diff --git a/.github/skills/regshape-dev/references/oci-image-spec-source/layer.md b/.github/skills/regshape-dev/references/oci-image-spec-source/layer.md new file mode 100644 index 0000000..ce7bd56 --- /dev/null +++ b/.github/skills/regshape-dev/references/oci-image-spec-source/layer.md @@ -0,0 +1,367 @@ +# Image Layer Filesystem Changeset + +This document describes how to serialize a filesystem and filesystem changes like removed files into a blob called a layer. +One or more layers are applied on top of each other to create a complete filesystem. +This document will use a concrete example to illustrate how to create and consume these filesystem layers. + +This section defines the `application/vnd.oci.image.layer.v1.tar`, `application/vnd.oci.image.layer.v1.tar+gzip`, `application/vnd.oci.image.layer.v1.tar+zstd`, `application/vnd.oci.image.layer.nondistributable.v1.tar`, `application/vnd.oci.image.layer.nondistributable.v1.tar+gzip`, and `application/vnd.oci.image.layer.nondistributable.v1.tar+zstd` [media types](media-types.md). + +## `+gzip` Media Types + +- The media type `application/vnd.oci.image.layer.v1.tar+gzip` represents an `application/vnd.oci.image.layer.v1.tar` payload which has been compressed with [gzip][rfc1952_2]. +- The media type `application/vnd.oci.image.layer.nondistributable.v1.tar+gzip` represents an `application/vnd.oci.image.layer.nondistributable.v1.tar` payload ([deprecation notice](#non-distributable-layers)) which has been compressed with [gzip][rfc1952_2]. + +## `+zstd` Media Types + +- The media type `application/vnd.oci.image.layer.v1.tar+zstd` represents an `application/vnd.oci.image.layer.v1.tar` payload which has been compressed with [zstd][rfc8478]. +- The media type `application/vnd.oci.image.layer.nondistributable.v1.tar+zstd` represents an `application/vnd.oci.image.layer.nondistributable.v1.tar` payload ([deprecation notice](#non-distributable-layers)) which has been compressed with [zstd][rfc8478]. + +## Distributable Format + +- Layer Changesets for the [media type](media-types.md) `application/vnd.oci.image.layer.v1.tar` MUST be packaged in [tar archive][tar-archive]. +- Layer Changesets for the [media type](media-types.md) `application/vnd.oci.image.layer.v1.tar` MUST NOT include duplicate entries for file paths in the resulting [tar archive][tar-archive]. + +## Change Types + +Types of changes that can occur in a changeset are: + +- Additions +- Modifications +- Removals + +Additions and Modifications are represented the same in the changeset tar archive. + +Removals are represented using "[whiteout](#whiteouts)" file entries (See [Representing Changes](#representing-changes)). + +### File Types + +Throughout this document section, the use of word "files" or "entries" includes the following, where supported: + +- regular files +- directories +- sockets +- symbolic links +- block devices +- character devices +- FIFOs + +### File Attributes + +Where supported, MUST include file attributes for Additions and Modifications include: + +- Modification Time (`mtime`) +- User ID (`uid`) + - User Name (`uname`) should be ignored on platforms that support User ID (`uid`) +- Group ID (`gid`) + - Group Name (`gname`) should be ignored on platforms that support Group ID (`gid`) +- Mode (`mode`) +- Extended Attributes (`xattrs`) +- Symlink reference (`linkname` + symbolic link type) +- [Hardlink](#hardlinks) reference (`linkname`) + +[Sparse files](https://en.wikipedia.org/wiki/Sparse_file) SHOULD NOT be used because they lack consistent support across tar implementations. + +#### Hardlinks + +- Hardlinks are a [POSIX concept](https://pubs.opengroup.org/onlinepubs/9699919799/functions/link.html) for having one or more directory entries for the same file on the same device. +- Not all filesystems support hardlinks (e.g. [FAT](https://en.wikipedia.org/wiki/File_Allocation_Table)). +- Hardlinks are possible with all [file types](#file-types) except `directories`. +- Non-directory files are considered "hardlinked" when their link count is greater than 1. +- Hardlinked files are on a same device (i.e. comparing Major:Minor pair) and have the same inode. +- The corresponding files that share the link with the > 1 linkcount may be outside the directory that the changeset is being produced from, in which case the `linkname` is not recorded in the changeset. +- Union filesystem implementations may have limited or no support for hardlinks, particularly when a change is made to a hardlinked file or a hardlink is created to a file in a lower filesystem. (See the [overlay specification](https://docs.kernel.org/filesystems/overlayfs.html#index) for more details.) +- Extracting a layer with hardlink references to files outside of the layer may fail. +- Hardlinks are stored in a tar archive with type of a `1` char, per the [GNU Basic Tar Format][gnu-tar-standard] and [libarchive tar(5)][libarchive-tar]. +- While approaches to deriving new or changed hardlinks may vary, a possible approach is: + +```text +SET LinkMap to map[< Major:Minor String >]map[< inode integer >]< path string > +SET LinkNames to map[< src path string >]< dest path string > +FOR each path in root path + IF path type is directory + CONTINUE + ENDIF + SET filestat to stat(path) + IF filestat num of links == 1 + CONTINUE + ENDIF + IF LinkMap[filestat device][filestat inode] is not empty + SET LinkNames[path] to LinkMap[filestat device][filestat inode] + ELSE + SET LinkMap[filestat device][filestat inode] to path + ENDIF +END FOR +``` + +With this approach, the link map and links names of a directory could be compared against that of another directory to derive additions and changes to hardlinks. + +#### Platform-specific attributes + +Implementations on Windows MUST support these additional attributes, encoded in [PAX vendor extensions](https://github.com/libarchive/libarchive/wiki/ManPageTar5#pax-interchange-format) as follows: + +- [Windows file attributes](https://msdn.microsoft.com/en-us/library/windows/desktop/gg258117(v=vs.85).aspx) (`MSWINDOWS.fileattr`) +- [Security descriptor](https://msdn.microsoft.com/en-us/library/cc230366.aspx) (`MSWINDOWS.rawsd`): base64-encoded self-relative binary security descriptor +- Mount points (`MSWINDOWS.mountpoint`): if present on a directory symbolic link, then the link should be created as a [directory junction](https://en.wikipedia.org/wiki/NTFS_junction_point) +- Creation time (`LIBARCHIVE.creationtime`) + +## Creating + +### Initial Root Filesystem + +The initial root filesystem is the base or parent layer. + +For this example, an image root filesystem has an initial state as an empty directory. +The name of the directory is not relevant to the layer itself, only for the purpose of producing comparisons. + +Here is an initial empty directory structure for a changeset, with a unique directory name `rootfs-c9d-v1`. + +```text +rootfs-c9d-v1/ +``` + +### Populate Initial Filesystem + +Files and directories are then created: + +```text +rootfs-c9d-v1/ + etc/ + my-app-config + bin/ + my-app-binary + my-app-tools +``` + +The `rootfs-c9d-v1` directory is then created as a plain [tar archive][tar-archive] with relative path to `rootfs-c9d-v1`. +Entries for the following files: + +```text +./ +./etc/ +./etc/my-app-config +./bin/ +./bin/my-app-binary +./bin/my-app-tools +``` + +### Populate a Comparison Filesystem + +Create a new directory and initialize it with a copy or snapshot of the prior root filesystem. +Example commands that can preserve [file attributes](#file-attributes) to make this copy are: + +- [cp(1)](https://linux.die.net/man/1/cp): `cp -a rootfs-c9d-v1/ rootfs-c9d-v1.s1/` +- [rsync(1)](https://linux.die.net/man/1/rsync): `rsync -aHAX rootfs-c9d-v1/ rootfs-c9d-v1.s1/` +- [tar(1)](https://linux.die.net/man/1/tar): `mkdir rootfs-c9d-v1.s1 && tar --acls --xattrs -C rootfs-c9d-v1/ -c . | tar -C rootfs-c9d-v1.s1/ --acls --xattrs -x` (including `--selinux` where supported) + +Any [changes](#change-types) to the snapshot MUST NOT change or affect the directory it was copied from. + +For example `rootfs-c9d-v1.s1` is an identical snapshot of `rootfs-c9d-v1`. +In this way `rootfs-c9d-v1.s1` is prepared for updates and alterations. + +**Implementor's Note**: _a copy-on-write or union filesystem can efficiently make directory snapshots_ + +Initial layout of the snapshot: + +```text +rootfs-c9d-v1.s1/ + etc/ + my-app-config + bin/ + my-app-binary + my-app-tools +``` + +See [Change Types](#change-types) for more details on changes. + +For example, add a directory at `/etc/my-app.d` containing a default config file, removing the existing config file. +Also a change (in attribute or file content) to `./bin/my-app-tools` binary to handle the config layout change. + +Following these changes, the representation of the `rootfs-c9d-v1.s1` directory: + +```text +rootfs-c9d-v1.s1/ + etc/ + my-app.d/ + default.cfg + bin/ + my-app-binary + my-app-tools +``` + +### Determining Changes + +When two directories are compared, the relative root is the top-level directory. +The directories are compared, looking for files that have been [added, modified, or removed](#change-types). + +For this example, `rootfs-c9d-v1/` and `rootfs-c9d-v1.s1/` are recursively compared, each as relative root path. + +The following changeset is found: + +```text +Added: /etc/my-app.d/ +Added: /etc/my-app.d/default.cfg +Modified: /bin/my-app-tools +Deleted: /etc/my-app-config +``` + +This reflects the removal of `/etc/my-app-config` and creation of a file and directory at `/etc/my-app.d/default.cfg`. +`/bin/my-app-tools` has also been replaced with an updated version. + +### Representing Changes + +A [tar archive][tar-archive] is then created which contains _only_ this changeset: + +- Added and modified files and directories in their entirety +- Deleted files or directories marked with a [whiteout file](#whiteouts) + +The resulting tar archive for `rootfs-c9d-v1.s1` has the following entries: + +```text +./etc/my-app.d/ +./etc/my-app.d/default.cfg +./bin/my-app-tools +./etc/.wh.my-app-config +``` + +To signify that the resource `./etc/my-app-config` MUST be removed when the changeset is applied, the basename of the entry is prefixed with `.wh.`. + +## Applying Changesets + +- Layer Changesets of [media type](media-types.md) `application/vnd.oci.image.layer.v1.tar` are _applied_, rather than simply extracted as tar archives. +- Applying a layer changeset requires special consideration for the [whiteout](#whiteouts) files. +- In the absence of any [whiteout](#whiteouts) files in a layer changeset, the archive is extracted like a regular tar archive. + +### Changeset over existing files + +This section specifies applying an entry from a layer changeset if the target path already exists. + +If the entry and the existing path are both directories, then the existing path's attributes MUST be replaced by those of the entry in the changeset. +In all other cases, the implementation MUST do the semantic equivalent of the following: + +- removing the file path (e.g. [`unlink(2)`](https://linux.die.net/man/2/unlink) on Linux systems) +- recreating the file path, based on the contents and attributes of the changeset entry + +## Whiteouts + +- A whiteout file is an empty file with a special filename that signifies a path should be deleted. +- A whiteout filename consists of the prefix `.wh.` plus the basename of the path to be deleted. +- A `.wh.` file, without a basename to delete, is invalid and implementations SHOULD return an error when encountering such an entry. +- As files prefixed with `.wh.` are special whiteout markers, it is not possible to create a filesystem which has a file or directory with a name beginning with `.wh.`. +- Once a whiteout is applied, the whiteout itself MUST also be hidden. +- Whiteout files MUST only apply to resources in lower/parent layers. +- Files that are present in the same layer as a whiteout file can only be hidden by whiteout files in subsequent layers. + +The following is a base layer with several resources: + +```text +file1 +a/file2 +b/ +c/file3 +``` + +If we then delete `file1`, `file2`, and `b/`, while leaving `file3` and adding `file4`, the next layer looks like: + +```text +.wh.file1 +a/.wh.file2 +.wh.b +file4 +``` + +Note that regardless of the path being deleted, the whiteout file is a regular file in the archive. + +Implementations SHOULD generate layers such that the whiteout files appear before sibling directory entries. + +### Opaque Whiteout + +- In addition to expressing that a single entry should be removed from a lower layer, layers may remove all of the children using an opaque whiteout entry. +- An opaque whiteout entry is a file with the name `.wh..wh..opq` indicating that all siblings are hidden in the lower layer. + +Let's take the following base layer as an example: + +```text +etc/ + my-app-config +bin/ + my-app-binary + my-app-tools + tools/ + my-app-tool-one +``` + +If all children of `bin/` are removed, the next layer would have the following: + +```text +bin/ + .wh..wh..opq +``` + +This is called _opaque whiteout_ format. +An _opaque whiteout_ file hides _all_ children of the `bin/` including sub-directories and all descendants. +Using _explicit whiteout_ files, this would be equivalent to the following: + +```text +bin/ + .wh.my-app-binary + .wh.my-app-tools + .wh.tools +``` + +In this case, a unique whiteout file is generated for each entry. +If there were more children of `bin/` in the base layer, there would be an entry for each. +Note that this opaque file will apply to _all_ children, including sub-directories, other resources and all descendants. + +Implementations SHOULD generate layers using _explicit whiteout_ files, but MUST accept both. + +As another example, consider the following base layer: + +```text +a/ +a/b/ +a/b/c/ +a/b/c/bar +``` + +When the next layer is created, the original `a/b` directory is deleted and recreated with `a/b/c/foo`: + +```text +a/ +a/.wh..wh..opq +a/b/ +a/b/c/ +a/b/c/foo +``` + +When processing the second layer, `a/.wh..wh..opq` is applied first, before creating the new version of `a/b`, regardless of the ordering in which the whiteout file was encountered. +For example, the following layer is equivalent to the layer above: + +```text +a/ +a/b/ +a/b/c/ +a/b/c/foo +a/.wh..wh..opq +``` + +Any given image is likely to be composed of several of these Image Filesystem Changeset tar archives. + +## Non-Distributable Layers + +> **NOTE**: Non-distributable layers are deprecated, and not recommended for future use. +> Implementations SHOULD NOT produce new non-distributable layers. +> Implementations are expected to support preexisting images with non-distributable layers. + +Due to legal requirements, certain layers may not be regularly distributable. +Such "non-distributable" layers are typically downloaded directly from a distributor but never uploaded. + +Non-distributable layers SHOULD be tagged with an alternative mediatype of `application/vnd.oci.image.layer.nondistributable.v1.tar`, `application/vnd.oci.image.layer.nondistributable.v1.tar+gzip`, or `application/vnd.oci.image.layer.nondistributable.v1.tar+zstd`. +Implementations SHOULD NOT upload layers tagged with this media type; however, such a media type SHOULD NOT affect whether an implementation downloads the layer. + +[Descriptors](descriptor.md) referencing non-distributable layers MAY include `urls` for downloading these layers directly; however, the presence of the `urls` field SHOULD NOT be used to determine whether or not a layer is non-distributable. + +[libarchive-tar]: https://github.com/libarchive/libarchive/wiki/ManPageTar5#POSIX_ustar_Archives +[gnu-tar-standard]: https://www.gnu.org/software/tar/manual/html_node/Standard.html +[rfc1952_2]: https://tools.ietf.org/html/rfc1952 +[tar-archive]: https://en.wikipedia.org/wiki/Tar_(computing) +[rfc8478]: https://tools.ietf.org/html/rfc8478 diff --git a/.github/skills/regshape-dev/references/oci-image-spec-source/manifest.md b/.github/skills/regshape-dev/references/oci-image-spec-source/manifest.md new file mode 100644 index 0000000..7a0ef73 --- /dev/null +++ b/.github/skills/regshape-dev/references/oci-image-spec-source/manifest.md @@ -0,0 +1,270 @@ +# OCI Image Manifest Specification + +There are three main goals of the Image Manifest Specification. +The first goal is content-addressable images, by supporting an image model where the image's configuration can be hashed to generate a unique ID for the image and its components. +The second goal is to allow multi-architecture images, through a "fat manifest" which references image manifests for platform-specific versions of an image. +In OCI, this is codified in an [image index](image-index.md). +The third goal is to be [translatable](conversion.md) to the [OCI Runtime Specification](https://github.com/opencontainers/runtime-spec). + +This section defines the `application/vnd.oci.image.manifest.v1+json` [media type](media-types.md). +For the media type(s) that this is compatible with see the [matrix](media-types.md#compatibility-matrix). + +## Image Manifest + +Unlike the [image index](image-index.md), which contains information about a set of images that can span a variety of architectures and operating systems, an image manifest provides a configuration and set of layers for a single container image for a specific architecture and operating system. + +## _Image Manifest_ Property Descriptions + +- **`schemaVersion`** *int* + + This REQUIRED property specifies the image manifest schema version. + For this version of the specification, this MUST be `2` to ensure backward compatibility with older versions of Docker. The value of this field will not change. This field MAY be removed in a future version of the specification. + +- **`mediaType`** *string* + + This property SHOULD be used and [remain compatible](media-types.md#compatibility-matrix) with earlier versions of this specification and with other similar external formats. + When used, this field MUST contain the media type `application/vnd.oci.image.manifest.v1+json`. + This field usage differs from the [descriptor](descriptor.md#properties) use of `mediaType`. + +- **`artifactType`** *string* + + This OPTIONAL property contains the type of an artifact when the manifest is used for an artifact. + This MUST be set when `config.mediaType` is set to the [empty value](#guidance-for-an-empty-descriptor). + If defined, the value MUST comply with [RFC 6838][rfc6838], including the [naming requirements in its section 4.2][rfc6838-s4.2], and MAY be registered with [IANA][iana]. + Implementations storing or copying image manifests MUST NOT error on encountering an `artifactType` that is unknown to the implementation. + +- **`config`** *[descriptor](descriptor.md)* + + This REQUIRED property references a configuration object for a container, by digest. + Beyond the [descriptor requirements](descriptor.md#properties), the value has the following additional restrictions: + + - **`mediaType`** *string* + + This [descriptor property](descriptor.md#properties) has additional restrictions for `config`. + + Implementations MUST NOT attempt to parse the referenced content if this media type is unknown and instead consider the referenced content as arbitrary binary data (e.g.: as `application/octet-stream`). + + Implementations storing or copying image manifests MUST NOT error on encountering a value that is unknown to the implementation. + + Implementations MUST support at least the following media types: + + - [`application/vnd.oci.image.config.v1+json`](config.md) + + Manifests for container images concerned with portability SHOULD use one of the above media types. + Manifests for artifacts concerned with portability SHOULD use `config.mediaType` as described in [Guidelines for Artifact Usage](#guidelines-for-artifact-usage). + + If the manifest uses a different media type than the above, it MUST comply with [RFC 6838][rfc6838], including the [naming requirements in its section 4.2][rfc6838-s4.2], and MAY be registered with [IANA][iana]. + + To set an effectively null or empty config and maintain portability see the [guidance for an empty descriptor](#guidance-for-an-empty-descriptor) below, and `DescriptorEmptyJSON` of the reference code. + + If this image manifest will be "runnable" by a runtime of some kind, it is strongly recommended to ensure it includes enough data to be unique (such as the `rootfs` and `diff_ids` included in `application/vnd.oci.image.config.v1+json`) so that it has a unique [`ImageID`](config.md#imageid). + +- **`layers`** *array of objects* + + Each item in the array MUST be a [descriptor](descriptor.md). + For portability, `layers` SHOULD have at least one entry. + See the [guidance for an empty descriptor](#guidance-for-an-empty-descriptor) below, and `DescriptorEmptyJSON` of the reference code. + + When the `config.mediaType` is set to `application/vnd.oci.image.config.v1+json`, the following additional restrictions apply: + + - The array MUST have the base layer at index 0. + - Subsequent layers MUST then follow in stack order (i.e. from `layers[0]` to `layers[len(layers)-1]`). + - The final filesystem layout MUST match the result of [applying](layer.md#applying-changesets) the layers to an empty directory. + - The [ownership, mode, and other attributes](layer.md#file-attributes) of the initial empty directory are unspecified. + + Beyond the [descriptor requirements](descriptor.md#properties), the value has the following additional restrictions: + + - **`mediaType`** *string* + + This [descriptor property](descriptor.md#properties) has additional restrictions for `layers[]`. + Implementations MUST support at least the following media types: + + - [`application/vnd.oci.image.layer.v1.tar`](layer.md) + - [`application/vnd.oci.image.layer.v1.tar+gzip`](layer.md#gzip-media-types) + - [`application/vnd.oci.image.layer.nondistributable.v1.tar`](layer.md#non-distributable-layers) ([deprecation notice](./layer.md#non-distributable-layers)) + - [`application/vnd.oci.image.layer.nondistributable.v1.tar+gzip`](layer.md#gzip-media-types) ([deprecation notice](./layer.md#non-distributable-layers)) + + Manifests concerned with portability SHOULD use one of the above media types. + Entries in this field will frequently use the `+gzip` types. + + Implementations SHOULD also support the following media types: + + - [`application/vnd.oci.image.layer.v1.tar+zstd`](layer.md#zstd-media-types) + + Implementations storing or copying image manifests MUST NOT error on encountering a `mediaType` that is unknown to the implementation. + + If the manifest uses a different media type than the above, it MUST comply with [RFC 6838][rfc6838], including the [naming requirements in its section 4.2][rfc6838-s4.2], and MAY be registered with [IANA][iana]. + + See [Guidelines for Artifact Usage](#guidelines-for-artifact-usage) for other uses of the `layers`. + +- **`subject`** *[descriptor](descriptor.md)* + + This OPTIONAL property specifies a [descriptor](descriptor.md) of another manifest. + This value defines a weak association to a separate [Merkle Directed Acyclic Graph (DAG)][dag] structure, and is used by the [`referrers` API][referrers-api] to include this manifest in the list of responses for the subject digest. + +- **`annotations`** *string-string map* + + This OPTIONAL property contains arbitrary metadata for the image manifest. + This OPTIONAL property MUST use the [annotation rules](annotations.md#rules). + + See [Pre-Defined Annotation Keys](annotations.md#pre-defined-annotation-keys). + +## Example Image Manifest + +*Example showing an image manifest:* + +```json,title=Manifest&mediatype=application/vnd.oci.image.manifest.v1%2Bjson +{ + "schemaVersion": 2, + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "config": { + "mediaType": "application/vnd.oci.image.config.v1+json", + "digest": "sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b86451d9bce1f432a537bc7", + "size": 7023 + }, + "layers": [ + { + "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", + "digest": "sha256:9834876dcfb05cb167a5c24953eba58c4ac89b1adf57f28f2f9d09af107ee8f0", + "size": 32654 + }, + { + "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", + "digest": "sha256:3c3a4604a545cdc127456d94e421cd355bca5b528f4a9c1905b15da2eb4a4c6b", + "size": 16724 + }, + { + "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", + "digest": "sha256:ec4b8955958665577945c89419d1af06b5f7636b4ac3da7f12184802ad867736", + "size": 73109 + } + ], + "subject": { + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "digest": "sha256:5b0bcabd1ed22e9fb1310cf6c2dec7cdef19f0ad69efa1f392e94a4333501270", + "size": 7682 + }, + "annotations": { + "com.example.key1": "value1", + "com.example.key2": "value2" + } +} +``` + +## Guidance for an Empty Descriptor + +*Implementers note*: The following is considered GUIDANCE for portability. + +Parts of the spec necessitate including a descriptor to a blob where some implementations of artifacts do not have associated content. +While an empty blob (`size` of 0) may be preferable, practice has shown that not to be ubiquitously supported. +The media type `application/vnd.oci.empty.v1+json` (`MediaTypeEmptyJSON`) has been specified for a descriptor that has no content for the implementation. +The blob payload is the most minimal content that is still a valid JSON object: `{}` (`size` of 2). +The blob digest of `{}` is `sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a`. +The data field is optional, and if included is the base64 encoding of `{}`: `e30=`. + +The resulting descriptor shown here is also defined in reference code as `DescriptorEmptyJSON`: + +```json,title=empty%20config&mediatype=application/vnd.oci.descriptor.v1%2Bjson +{ + "mediaType": "application/vnd.oci.empty.v1+json", + "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "size": 2, + "data": "e30=" +} +``` + +## Guidelines for Artifact Usage + +Content other than OCI container images MAY be packaged using the image manifest. +When this is done, the `config.mediaType` value MUST be set to a value specific to the artifact type or the [empty value](#guidance-for-an-empty-descriptor). +If the `config.mediaType` is set to the empty value, the `artifactType` MUST be defined. +If the artifact does not need layers, a single layer SHOULD be included with a non-zero size. +The suggested content for an unused `layers` array is the [empty descriptor](#guidance-for-an-empty-descriptor). + +The design of the artifact depends on what content is being packaged with the artifact. +The decision tree below and the associated examples MAY be used to design new artifacts: + +1. Does the artifact consist of at least one file or blob? + If yes, continue to 2. + If no, specify the `artifactType`, and set the `config` and a single `layers` element to the empty descriptor value. + Here is an example of this with annotations included: + + ```json,title=Minimal%20artifact&mediatype=application/vnd.oci.image.manifest.v1%2Bjson + { + "schemaVersion": 2, + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "artifactType": "application/vnd.example+type", + "config": { + "mediaType": "application/vnd.oci.empty.v1+json", + "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "size": 2 + }, + "layers": [ + { + "mediaType": "application/vnd.oci.empty.v1+json", + "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "size": 2 + } + ], + "annotations": { + "oci.opencontainers.image.created": "2023-01-02T03:04:05Z", + "com.example.data": "payload" + } + } + ``` + +2. Does the artifact have additional JSON formatted metadata as configuration? + If yes, continue to 3. + If no, specify the `artifactType`, include the artifact in the `layers`, and set `config` to the empty descriptor value. + Here is an example of this with a single layer: + + ```json,title=Artifact%20without%20config&mediatype=application/vnd.oci.image.manifest.v1%2Bjson + { + "schemaVersion": 2, + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "artifactType": "application/vnd.example+type", + "config": { + "mediaType": "application/vnd.oci.empty.v1+json", + "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "size": 2 + }, + "layers": [ + { + "mediaType": "application/vnd.example+type", + "digest": "sha256:e258d248fda94c63753607f7c4494ee0fcbe92f1a76bfdac795c9d84101eb317", + "size": 1234 + } + ] + } + ``` + +3. For artifacts with a config blob, specify the `artifactType` to a common value for your artifact tooling, specify the `config` with the metadata for this artifact, and include the artifact in the `layers`. + Here is an example of this: + + ```json,title=Artifact%20with%20config&mediatype=application/vnd.oci.image.manifest.v1%2Bjson + { + "schemaVersion": 2, + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "artifactType": "application/vnd.example+type", + "config": { + "mediaType": "application/vnd.example.config.v1+json", + "digest": "sha256:5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03", + "size": 123 + }, + "layers": [ + { + "mediaType": "application/vnd.example.data.v1.tar+gzip", + "digest": "sha256:e258d248fda94c63753607f7c4494ee0fcbe92f1a76bfdac795c9d84101eb317", + "size": 1234 + } + ] + } + ``` + +_Implementers note:_ artifacts have historically been created without an `artifactType` field, and tooling to work with artifacts should fallback to the `config.mediaType` value. + +[dag]: https://en.wikipedia.org/wiki/Merkle_tree +[iana]: https://www.iana.org/assignments/media-types/media-types.xhtml +[referrers-api]: https://github.com/opencontainers/distribution-spec/blob/main/spec.md#listing-referrers +[rfc6838]: https://tools.ietf.org/html/rfc6838 +[rfc6838-s4.2]: https://tools.ietf.org/html/rfc6838#section-4.2 diff --git a/.github/skills/regshape-dev/references/oci-image-spec-source/media-types.md b/.github/skills/regshape-dev/references/oci-image-spec-source/media-types.md new file mode 100644 index 0000000..25a26fa --- /dev/null +++ b/.github/skills/regshape-dev/references/oci-image-spec-source/media-types.md @@ -0,0 +1,91 @@ +# OCI Image Media Types + +The following media types identify the formats described here and their referenced resources: + +- `application/vnd.oci.descriptor.v1+json`: [Content Descriptor](descriptor.md) +- `application/vnd.oci.layout.header.v1+json`: [OCI Layout](image-layout.md#oci-layout-file) +- `application/vnd.oci.image.index.v1+json`: [Image Index](image-index.md) +- `application/vnd.oci.image.manifest.v1+json`: [Image manifest](manifest.md#image-manifest) +- `application/vnd.oci.image.config.v1+json`: [Image config](config.md) +- `application/vnd.oci.image.layer.v1.tar`: ["Layer", as a tar archive](layer.md) +- `application/vnd.oci.image.layer.v1.tar+gzip`: ["Layer", as a tar archive](layer.md#gzip-media-types) compressed with [gzip][rfc1952] +- `application/vnd.oci.image.layer.v1.tar+zstd`: ["Layer", as a tar archive](layer.md#zstd-media-types) compressed with [zstd][rfc8478] +- `application/vnd.oci.empty.v1+json`: [Empty for unused descriptors](manifest.md#guidance-for-an-empty-descriptor) + +The following media types identify a ["Layer" with distribution restrictions](layer.md#non-distributable-layers), but are **deprecated** and not recommended for future use: + +- `application/vnd.oci.image.layer.nondistributable.v1.tar`: "Layer", as a tar archive +- `application/vnd.oci.image.layer.nondistributable.v1.tar+gzip`: ["Layer", as a tar archive with distribution restrictions](layer.md#gzip-media-types) compressed with [gzip][rfc1952] +- `application/vnd.oci.image.layer.nondistributable.v1.tar+zstd`: ["Layer", as a tar archive with distribution restrictions](layer.md#zstd-media-types) compressed with [zstd][rfc8478] + +## Media Type Conflicts + +[Blob](image-layout.md) retrieval methods MAY return media type metadata. +For example, a HTTP response might return a manifest with the Content-Type header set to `application/vnd.oci.image.manifest.v1+json`. +Implementations MAY also have expectations for the blob's media type and digest (e.g. from a [descriptor](descriptor.md) referencing the blob). + +- Implementations that do not have an expected media type for the blob SHOULD respect the returned media type. +- Implementations that have an expected media type which matches the returned media type SHOULD respect the matched media type. +- Implementations that have an expected media type which does not match the returned media type SHOULD: + - Respect the expected media type if the blob matches the expected digest. + Implementations MAY warn about the media type mismatch. + - Return an error if the blob does not match the expected digest (as [recommended for descriptors](descriptor.md#properties)). + - Return an error if they do not have an expected digest. + +## Compatibility Matrix + +The OCI Image Specification strives to be backwards and forwards compatible when possible. +Breaking compatibility with existing systems creates a burden on users whether they are build systems, distribution systems, container engines, etc. +This section shows where the OCI Image Specification is compatible with formats external to the OCI Image and different versions of this specification. + +### application/vnd.oci.image.index.v1+json + +Similar/related schema: + +- [application/vnd.docker.distribution.manifest.list.v2+json](https://github.com/distribution/distribution/blob/v2.8.3/docs/spec/manifest-v2-2.md#manifest-list) + - `.annotations`: only present in OCI + - `.[]manifests.annotations`: only present in OCI + - `.[]manifests.urls`: only present in OCI + +### application/vnd.oci.image.manifest.v1+json + +Similar/related schema: + +- [application/vnd.docker.distribution.manifest.v2+json](https://github.com/distribution/distribution/blob/v2.8.3/docs/spec/manifest-v2-2.md#image-manifest) + - `.annotations`: only present in OCI + - `.config.annotations`: only present in OCI + - `.config.urls`: only present in OCI + - `.[]layers.annotations`: only present in OCI + +### application/vnd.oci.image.layer.v1.tar+gzip + +Interchangeable and fully compatible mime-types: + +- [application/vnd.docker.image.rootfs.diff.tar.gzip](https://github.com/moby/moby/blob/v20.10.8/image/spec/v1.2.md#creating-an-image-filesystem-changeset) + +### application/vnd.oci.image.config.v1+json + +Similar/related schema: + +- [application/vnd.docker.container.image.v1+json](https://github.com/moby/moby/blob/v20.10.8/image/spec/v1.2.md#image-json-description) (Docker Image Spec v1.2) + - `.config.Memory`: only present in Docker, and reserved in OCI + - `.config.MemorySwap`: only present in Docker, and reserved in OCI + - `.config.CpuShares`: only present in Docker, and reserved in OCI + - `.config.Healthcheck`: only present in Docker, and reserved in OCI +- [Moby/Docker](https://github.com/moby/moby) + - `.config.ArgsEscaped`: Windows-specific Moby/Docker extension, deprecated in OCI, available for compatibility with older images. + +`.config.StopSignal` and `.config.Labels` are accidentally undocumented in Docker Image Spec v1.2, but these fields are not OCI-specific concepts. + +## Relations + +The following figure shows how the above media types reference each other: + +![media types](img/media-types.png) + +[Descriptors](descriptor.md) are used for all references. +The image-index being a "fat manifest" references a list of image manifests per target platform. +An image manifest references exactly one target configuration and possibly many layers. + +[rfc1952]: https://tools.ietf.org/html/rfc1952 +[rfc8478]: https://tools.ietf.org/html/rfc8478 diff --git a/.github/skills/regshape-dev/references/oci-image-spec-source/spec.md b/.github/skills/regshape-dev/references/oci-image-spec-source/spec.md new file mode 100644 index 0000000..162350c --- /dev/null +++ b/.github/skills/regshape-dev/references/oci-image-spec-source/spec.md @@ -0,0 +1,70 @@ +# Open Container Initiative + +## Image Format Specification + +This specification defines an OCI Image, consisting of an [image manifest](manifest.md), an [image index](image-index.md) (optional), a set of [filesystem layers](layer.md), and a [configuration](config.md). + +The goal of this specification is to enable the creation of interoperable tools for building, transporting, and preparing a container image to run. + +### Table of Contents + +- [Notational Conventions](#notational-conventions) +- [Overview](#overview) + - [Understanding the Specification](#understanding-the-specification) + - [Media Types](media-types.md) +- [Content Descriptors](descriptor.md) +- [Image Layout](image-layout.md) +- [Image Manifest](manifest.md) +- [Image Index](image-index.md) +- [Filesystem Layers](layer.md) +- [Image Configuration](config.md) +- [Annotations](annotations.md) +- [Conversion](conversion.md) +- [Considerations](considerations.md) + - [Extensibility](considerations.md#extensibility) + - [Canonicalization](considerations.md#canonicalization) + +## Notational Conventions + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" are to be interpreted as described in [RFC 2119](https://tools.ietf.org/html/rfc2119) (Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997). + +The key words "unspecified", "undefined", and "implementation-defined" are to be interpreted as described in the [rationale for the C99 standard][c99-unspecified]. + +An implementation is not compliant if it fails to satisfy one or more of the MUST, MUST NOT, REQUIRED, SHALL, or SHALL NOT requirements for the protocols it implements. +An implementation is compliant if it satisfies all the MUST, MUST NOT, REQUIRED, SHALL, and SHALL NOT requirements for the protocols it implements. + +## Overview + +At a high level the image manifest contains metadata about the contents and dependencies of the image including the content-addressable identity of one or more [filesystem layer changeset](layer.md) archives that will be unpacked to make up the final runnable filesystem. +The image configuration includes information such as application arguments, environments, etc. +The image index is a higher-level manifest which points to a list of manifests and descriptors. +Typically, these manifests may provide different implementations of the image, possibly varying by platform or other attributes. + +![build diagram](img/build-diagram.png) + +Once built the OCI Image can then be discovered by name, downloaded, verified by hash, trusted through a signature, and unpacked into an [OCI Runtime Bundle](https://github.com/opencontainers/runtime-spec/blob/main/bundle.md). + +![runtime diagram](img/run-diagram.png) + +### Understanding the Specification + +The [OCI Image Media Types](media-types.md) document is a starting point to understanding the overall structure of the specification. + +The high-level components of the spec include: + +- [Image Manifest](manifest.md) - a document describing the components that make up a container image +- [Image Index](image-index.md) - an annotated list of manifests +- [Image Layout](image-layout.md) - a filesystem layout representing the contents of an image +- [Filesystem Layer](layer.md) - a changeset that describes a container's filesystem +- [Image Configuration](config.md) - a document determining layer ordering and configuration of the image suitable for translation into a [runtime bundle][runtime-spec] +- [Conversion](conversion.md) - a document describing how this translation should occur +- [Artifacts Guidance](artifacts-guidance.md) - a document describing how to use the spec for packaging content other than OCI images +- [Descriptor](descriptor.md) - a reference that describes the type, metadata and content address of referenced content + +Future versions of this specification may include the following OPTIONAL features: + +- Signatures that are based on signing image content address +- Naming that is federated based on DNS and can be delegated + +[c99-unspecified]: https://www.open-std.org/jtc1/sc22/wg14/www/C99RationaleV5.10.pdf#page=18 +[runtime-spec]: https://github.com/opencontainers/runtime-spec diff --git a/.github/skills/regshape-dev/references/oci-image-spec.md b/.github/skills/regshape-dev/references/oci-image-spec.md new file mode 100644 index 0000000..8f27f9f --- /dev/null +++ b/.github/skills/regshape-dev/references/oci-image-spec.md @@ -0,0 +1,249 @@ +# OCI Image Spec Reference + +Quick reference for the OCI Image Format Specification v1.1.0 relevant to RegShape. + +This reference complements [oci-distribution-spec.md](oci-distribution-spec.md): the +distribution spec describes how content moves over the wire (endpoints, uploads, +auth), while the image spec describes the *content itself* — descriptors, image +manifests, image indexes, image configuration, and filesystem layers. + +This is a condensed summary. The full upstream OCI Image Format Specification is +saved locally under [oci-image-spec-source/](oci-image-spec-source/) (mirrored from +). Consult these files for authoritative, +normative detail: + +- [spec.md](oci-image-spec-source/spec.md) — top-level overview +- [descriptor.md](oci-image-spec-source/descriptor.md) — content descriptors +- [manifest.md](oci-image-spec-source/manifest.md) — image manifest +- [image-index.md](oci-image-spec-source/image-index.md) — image index +- [config.md](oci-image-spec-source/config.md) — image configuration +- [layer.md](oci-image-spec-source/layer.md) — filesystem layers +- [media-types.md](oci-image-spec-source/media-types.md) — media types +- [annotations.md](oci-image-spec-source/annotations.md) — annotation keys +- [image-layout.md](oci-image-spec-source/image-layout.md) — OCI image layout +- [conversion.md](oci-image-spec-source/conversion.md), [considerations.md](oci-image-spec-source/considerations.md), [artifacts-guidance.md](oci-image-spec-source/artifacts-guidance.md) + +## Table of Contents + +- [Image Components](#image-components) +- [Content Descriptors](#content-descriptors) +- [Image Manifest](#image-manifest) +- [Image Index](#image-index) +- [Image Configuration](#image-configuration) +- [Filesystem Layers](#filesystem-layers) +- [Digests and Verification](#digests-and-verification) +- [Media Types](#media-types) +- [Predefined Annotations](#predefined-annotations) +- [Assembling an Image](#assembling-an-image) +- [RegShape Implementation Notes](#regshape-implementation-notes) + +## Image Components + +An OCI image is a set of content-addressable objects tied together by digests: + +| Component | Media Type | Role | +|-----------|-----------|------| +| Image Index | `application/vnd.oci.image.index.v1+json` | Optional top-level, points to per-platform manifests | +| Image Manifest | `application/vnd.oci.image.manifest.v1+json` | Points to one config + ordered layers | +| Image Config | `application/vnd.oci.image.config.v1+json` | Execution parameters + rootfs + history | +| Layer | `application/vnd.oci.image.layer.v1.tar+gzip` | A filesystem changeset (tar, usually gzipped) | + +A single-platform image is `manifest -> config + layers`. A multi-platform image is +`index -> [manifest -> config + layers, ...]`. + +## Content Descriptors + +Every reference between objects is a descriptor. It is the fundamental link type. + + { + "mediaType": "application/vnd.oci.image.config.v1+json", + "digest": "sha256:...", + "size": 1234, + "urls": [], + "annotations": {}, + "data": "", + "artifactType": "", + "platform": { "architecture": "amd64", "os": "linux" } + } + +Required fields: `mediaType`, `digest`, `size`. `platform` is only used inside an +image index. `data` allows small content to be embedded inline (must match `digest`). + +## Image Manifest + + { + "schemaVersion": 2, + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "artifactType": "", + "config": { + "mediaType": "application/vnd.oci.image.config.v1+json", + "digest": "sha256:...", + "size": 1234 + }, + "layers": [ + { + "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", + "digest": "sha256:...", + "size": 5678 + } + ], + "subject": { + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "digest": "sha256:...", + "size": 1234 + }, + "annotations": {} + } + +- `layers` are ordered from base (first) to top (last); this order is applied when + building the rootfs. +- `config.mediaType` determines whether this is a runnable image or a generic + artifact. For artifacts, set `artifactType` and use `application/vnd.oci.empty.v1+json` + as the config. +- `subject` links this manifest to another manifest for the referrers API. + +## Image Index + + { + "schemaVersion": 2, + "mediaType": "application/vnd.oci.image.index.v1+json", + "artifactType": "", + "manifests": [ + { + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "digest": "sha256:...", + "size": 1234, + "platform": { "architecture": "amd64", "os": "linux" } + } + ], + "subject": {}, + "annotations": {} + } + +Each entry SHOULD carry a `platform` object so clients can select the right manifest. +Optional platform fields: `os.version`, `os.features`, `variant` (e.g. `v7` for arm). + +## Image Configuration + +The config blob describes how to run the image and how its rootfs is built. + + { + "created": "2024-01-01T00:00:00Z", + "author": "regshape", + "architecture": "amd64", + "os": "linux", + "config": { + "User": "1000:1000", + "ExposedPorts": { "8080/tcp": {} }, + "Env": ["PATH=/usr/bin"], + "Entrypoint": ["/app"], + "Cmd": ["--serve"], + "Volumes": { "/data": {} }, + "WorkingDir": "/app", + "Labels": {}, + "StopSignal": "SIGTERM" + }, + "rootfs": { + "type": "layers", + "diff_ids": [ + "sha256:", + "sha256:" + ] + }, + "history": [ + { + "created": "2024-01-01T00:00:00Z", + "created_by": "COPY app /app", + "empty_layer": false + } + ] + } + +- `architecture` and `os` are required and MUST match the index descriptor `platform`. +- `rootfs.diff_ids` are digests over the __uncompressed__ tar of each layer, in the + same order as `manifest.layers`. Entries with `empty_layer: true` in `history` have + no corresponding layer/diff_id. + +## Filesystem Layers + +A layer is a tar archive representing a changeset applied on top of the previous layer. + +- **Additions / modifications**: files present in the tar overwrite lower layers. +- **Deletions (whiteouts)**: a file named `.wh.` marks `` as removed. +- **Opaque directories**: a file named `.wh..wh..opq` clears all lower-layer contents + of its directory. + +Layers are applied in order; the union of all changesets is the final rootfs. + +Compression: layers are normally gzip (`+gzip`) or zstd (`+zstd`). The layer digest in +the manifest is taken over the __compressed__ bytes, while the `diff_id` in the config +is taken over the __uncompressed__ tar. + +## Digests and Verification + +- Digests use the form `:`, e.g. `sha256:...` (sha512 also allowed). +- A blob is valid only if its computed digest matches the descriptor `digest` AND its + byte length matches `size`. +- The manifest digest is computed over the exact serialized manifest bytes; do not + re-serialize before digesting when verifying a pulled manifest. + +## Media Types + +| Media Type | Description | +|------------|-------------| +| `application/vnd.oci.image.index.v1+json` | Image index (multi-platform) | +| `application/vnd.oci.image.manifest.v1+json` | Image manifest | +| `application/vnd.oci.image.config.v1+json` | Image configuration | +| `application/vnd.oci.image.layer.v1.tar` | Uncompressed layer changeset | +| `application/vnd.oci.image.layer.v1.tar+gzip` | Gzip-compressed layer | +| `application/vnd.oci.image.layer.v1.tar+zstd` | Zstd-compressed layer | +| `application/vnd.oci.image.layer.nondistributable.v1.tar+gzip` | Foreign/nondistributable layer (deprecated) | +| `application/vnd.oci.empty.v1+json` | Empty config for artifact manifests (`{}`) | + +## Predefined Annotations + +Common `org.opencontainers.image.*` annotation keys: + +| Key | Description | +|-----|-------------| +| `org.opencontainers.image.created` | Build date/time (RFC 3339) | +| `org.opencontainers.image.authors` | Contact for the image | +| `org.opencontainers.image.url` | URL to find more info | +| `org.opencontainers.image.source` | URL to source repository | +| `org.opencontainers.image.version` | Version of the packaged software | +| `org.opencontainers.image.revision` | Source control revision | +| `org.opencontainers.image.title` | Human-readable title | +| `org.opencontainers.image.description` | Human-readable description | +| `org.opencontainers.image.base.name` | Base image reference | +| `org.opencontainers.image.base.digest` | Digest of the base image manifest | + +## Assembling an Image + +To build and push a single-platform image from scratch: + +1. Create each layer tar, compute its compressed digest (`layer.digest`/`size`) and + its uncompressed `diff_id`. +2. Build the config JSON with `architecture`, `os`, `rootfs.diff_ids` (uncompressed, + ordered), and any `config`/`history`. +3. Push each layer blob and the config blob (see blob operations in the distribution + spec). +4. Build the manifest referencing the config descriptor and the ordered layer + descriptors. +5. Push the manifest by tag or digest. +6. (Optional) Build an image index referencing per-platform manifests and push it. + +Pulling reverses the process: fetch manifest (or select from index by platform), +fetch config, fetch and apply layers to reconstruct the rootfs. + +## RegShape Implementation Notes + +- Model image manifests, indexes, config, and descriptors in `libs/models/` as typed, + serializable structures with strict digest/size validation. +- Keep manifest/index parsing and pushing under the manifests domain (`libs/manifests/`); + keep image-level assembly (config building, layer packing/unpacking, diff_id + computation, rootfs reconstruction) under the images domain (`libs/images/`). +- Preserve exact manifest bytes on pull so digest verification is byte-accurate. +- Break mode opportunities specific to images: mismatched `diff_ids` vs layers, wrong + layer media types, config/architecture mismatches with the index platform, corrupt + whiteout entries, and manifests whose `config.size`/`layers[].size` disagree with the + pushed blobs.