Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .changeset/config.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"$schema": "https://unpkg.com/@changesets/config@3.1.1/schema.json",
"$schema": "https://unpkg.com/@changesets/config@4.0.0/schema.json",
"changelog": "@changesets/cli/changelog",
"commit": false,
"fixed": [],
Expand Down
16 changes: 16 additions & 0 deletions .changeset/delegate-native-conditionals.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---
'@nestm/storage': minor
---

Delegate conditional create, replace, exact ETag read, delete, and paired copy
to the Files SDK 2.3 operation pipeline so plugins, hooks, retries, and receipts
apply without bypassing native provider preconditions. Keep the direct NestM
fallback only for version predicates, conditional multipart/resumable
completion, and one-sided conditional copies, and fail those shapes closed when
caller Files policy is configured.

Expose paired-copy dependency flags in provider capabilities and preserve
`StorageError.applied` plus `appliedEtag` so callers can reconcile a conditional
mutation that committed before a post-operation failure. Preserve the same
bounded reconciliation signal through workspace, AI-tool, and gateway error
boundaries.
20 changes: 1 addition & 19 deletions .changeset/pre.json
Original file line number Diff line number Diff line change
@@ -1,22 +1,4 @@
{
"mode": "pre",
"tag": "alpha",
"initialVersions": {
"@nestm/storage": "0.1.0-alpha.0"
},
"changesets": [
"add-agent-workspaces",
"add-artifact-storage",
"add-workspace-last-write-wins",
"compose-workspace-tool-conflicts",
"fail-closed-conditional-files-policy",
"fix-workspace-cursor-continuity",
"framework-neutral-core",
"harden-storage-boundaries",
"initial-storage-release",
"map-foreign-files-not-found",
"restore-generic-storage-boundary",
"runtime-provider-selection",
"split-s3-conditional-capabilities"
]
"tag": "alpha"
}
5 changes: 5 additions & 0 deletions .changeset/stable-nest-twelve.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@nestm/storage': patch
---

Replace the NestJS 12 prerelease peer range and test matrix with the stable NestJS 12 line, and refresh the package toolchain and optional integration dependencies.
16 changes: 1 addition & 15 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ jobs:
- run: pnpm run test
env: { CI: 'true' }

# Non-blocking early warning for changes in Nest's prerelease API.
# Non-blocking early warning for changes in Nest's next release line.
canary:
runs-on: ubuntu-latest
continue-on-error: true
Expand All @@ -73,17 +73,3 @@ jobs:
- run: pnpm install --no-frozen-lockfile
- run: pnpm up "@nestjs/*@next"
- run: pnpm run test

npm-peer-smoke:
runs-on: ubuntu-latest
if: github.event_name != 'pull_request'
steps:
- uses: actions/setup-node@v7
with: { node-version: 24 }
- run: |
mkdir smoke && cd smoke && npm init -y > /dev/null
if npm install @nestjs/common@next @nestjs/core@next @nestjs/platform-express@next reflect-metadata rxjs --no-audit --no-fund; then
echo "::warning::Nest 12 alpha sibling peers appear fixed — review pnpm-workspace.yaml and README."
else
echo "npm ERESOLVE still present (expected while Nest 12 alphas declare stale sibling peers)."
fi
13 changes: 7 additions & 6 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,13 +41,14 @@ jobs:
- run: pnpm dlx @arethetypeswrong/cli --pack . --profile esm-only
- name: Create release PR or publish to npm
id: changesets
uses: changesets/action@v1
uses: changesets/action@8488615a623b1b9c987934bb89eae8af6a946ac1 # v2.1.1
with:
publish: pnpm run release
version: pnpm changeset version
title: 'chore: release @nestm/storage'
commit: 'chore: release @nestm/storage'
createGithubReleases: true
github-token: ${{ secrets.GITHUB_TOKEN }}
publish-script: pnpm run release
version-script: pnpm changeset version
pr-title: 'chore: release @nestm/storage'
commit-message: 'chore: release @nestm/storage'
create-github-releases: true
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Reconcile git tag with npm
Expand Down
118 changes: 69 additions & 49 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,19 +8,18 @@ The package uses [`files-sdk`](https://github.com/haydenbleasel/files-sdk) as
its provider engine, but owns the API injected into Nest applications. Provider
SDK types, errors, and `files.raw` do not leak through the root package.

> This package targets the NestJS 12 prerelease line and is itself published on
> the `alpha` dist-tag.
> This package targets stable NestJS 12 and is itself published on the `alpha`
> dist-tag.

## Requirements

- Node.js 22.12 or newer
- ESM

The framework-neutral `@nestm/storage/core` entry point does not require
NestJS. The root entry point and HTTP gateway additionally require NestJS
`12.0.0-alpha.5` or newer in the Nest 12 prerelease line, `reflect-metadata`,
and RxJS. Those framework peers are optional at installation time so core-only
consumers do not download NestJS.
NestJS. The root entry point and HTTP gateway additionally require NestJS 12,
`reflect-metadata`, and RxJS. Those framework peers are optional at installation
time so core-only consumers do not download NestJS.

## Install

Expand All @@ -33,7 +32,7 @@ pinned engine explicitly only when the application imports another adapter
such as `files-sdk/gcs`:

```sh
pnpm add files-sdk@2.2.3
pnpm add files-sdk@2.3.0
```

Pass AWS-SDK-backed S3 adapters through the package-owned `s3()` and
Expand All @@ -58,20 +57,15 @@ pnpm add @google-cloud/storage google-auth-library
pnpm add @azure/storage-blob @azure/core-auth @azure/identity
```

`@aws-sdk/client-s3` 3.919.0 or newer is required. Earlier releases omit
destination `If-Match` and `If-None-Match` from the serialized `CopyObject`
request even when those fields appear in a newer compile-time command shape;
the package peer range and packed minimum-peer smoke test enforce this floor.
`@aws-sdk/client-s3` 3.1079.0 or newer is required, matching the Files SDK 2.3
peer floor. The supported range includes destination `If-Match` and
`If-None-Match` serialization for `CopyObject`; the package peer range and
packed minimum-peer smoke test enforce this floor.

`files-sdk` currently declares its optional Nest peer for Nest 10 and 11. This
library does not import `files-sdk/nestjs`; the Nest 12 integration is entirely
owned here. A package manager may nevertheless report that temporary optional
peer mismatch while Nest 12 remains prerelease.

NestJS 12 alpha also has prerelease peer declarations that npm may reject under
its strict resolver. If npm reports an `ERESOLVE` error for Nest's own peers,
install with `npm install --legacy-peer-deps`; pnpm works with the repository's
checked-in peer-version policy.
peer mismatch until Files SDK widens its declaration to include Nest 12.

## Framework-neutral core

Expand Down Expand Up @@ -268,7 +262,7 @@ Install AI SDK 7 and Zod only in applications that use the optional adapter:
pnpm add ai@^7 zod@^4
```

`files-sdk` 2.2.x still declares an optional `ai@^6` peer for its own adapter,
`files-sdk` 2.3.x still declares an optional `ai@^6` peer for its own adapter,
so some package managers may print a peer warning when AI SDK 7 is installed.
This package does not import that adapter; `@nestm/storage/ai-sdk` targets AI
SDK 7 directly.
Expand Down Expand Up @@ -609,36 +603,37 @@ provide: NestJS 12 named stores, exact native conditional/CAS capabilities,
`StorageWorkspace` permissions and limits, bounded storage errors, and
capability-scoped AI tools.

On the alpha.8 base, ordinary `FilesSdkStorageDriver` operations already
delegate to the Files SDK pipeline. The exception is the native conditional
adapter extensions: the current Files SDK operation union does not include
them, so they cannot run through caller-configured Files plugins, hooks, or
receipts. Until Files SDK provides one interception boundary for ordinary and
conditional operations, the driver applies this interim fail-closed
compatibility rule:

| Caller Files configuration | Ordinary operations | Conditional operations |
| ------------------------------------------------- | ------------------------------------ | -------------------------------------------------------- |
| No plugins, active hooks, or receipts | Files pipeline | Advertised when the adapter supports the exact primitive |
| One or more plugins | Files pipeline, including transforms | Hidden; direct invocation returns `NOT_SUPPORTED` |
| Any active hook | Files pipeline and hook callbacks | Hidden; direct invocation returns `NOT_SUPPORTED` |
| Receipts enabled with `true` or an options object | Files pipeline and receipts | Hidden; direct invocation returns `NOT_SUPPORTED` |

An empty plugin list, an empty hooks object, and `receipts: false` do not trigger
the gate. NestM's internal physical-key guard does not trigger it either. When
available, direct conditional paths independently apply prefixing, the
physical-key budget, mutation read-only restrictions, default
retry/signal/timeout options, and bounded error mapping. `StoragePlugin` remains
a separate veto/observation boundary; it is not a substitute for Files body or
result transforms. This compatibility gate is intended to be removed once
native CAS can traverse the upstream operation and plugin pipeline rather than
becoming a second generic CRUD facade here.

`StorageWorkspace` therefore exposes both contracts without weakening either:
its existing create/replace, exact-read copy/move, and conditional-delete paths
retain native CAS and this fail-closed gate, while explicit overwrite and
unconditional-delete variants use the ordinary Files pipeline. Lower-level
conditional client and driver APIs remain available to callers that need them.
Files SDK 2.3 provides native conditional operations through the same
interception boundary as ordinary CRUD. NestM adapters expose their exact
create, replace, ETag read, delete, and paired conditional-copy primitives to
that boundary. These operations now pass through caller-configured Files
plugins, hooks, retries, and receipts; a body transform, veto, retry observer,
or audit policy therefore sees the conditional operation instead of being
bypassed.

NestM retains a narrow direct fallback only for conditional shapes Files SDK
2.3 cannot represent: immutable version predicates, conditional
multipart/resumable completion, and a copy with only its source or only its
destination conditioned. Because those fallbacks cannot traverse the Files
operation pipeline, they remain fail-closed when caller Files policy is active:

| Operation shape | Execution path | With Files plugins, active hooks, or receipts |
| ----------------------------------------------------------- | --------------------- | --------------------------------------------- |
| Ordinary operations | Files pipeline | Available |
| Create/replace/ETag read/delete/paired conditional copy | Files 2.3 pipeline | Available |
| Version, conditional multipart/resumable, or one-sided copy | NestM direct fallback | Hidden; invocation returns `NOT_SUPPORTED` |

An empty plugin list, an empty hooks object, and `receipts: false` do not count
as caller policy. When available, a direct fallback still applies prefixing,
the physical-key budget, read-only restrictions, default retry/signal/timeout
options, and bounded error mapping. `StoragePlugin` remains a separate
veto/observation boundary; it is not a substitute for Files body or result
transforms.

`StorageWorkspace` uses the Files pipeline whenever its conditional operation
has an upstream representation. Lower-level conditional client and driver APIs
remain available for applications that intentionally use the policy-free
NestM-only fallback shapes.

## Storage API

Expand Down Expand Up @@ -702,6 +697,15 @@ completion. They also declare the complete physical-key byte budget. Callers
must check the exact primitive they need; a missing field is unsupported and is
never widened from another operation.

Some adapters expose conditional copy only as a paired source-and-destination
operation. In that case,
`capabilities.conditionalCopySource.requiresDestinationPredicate` and
`capabilities.conditionalCopyDestination.requiresSourcePredicate` are `true`.
`StorageClient.promote` rejects a request missing the required counterpart
before provider I/O. A paired request must also satisfy the advertised
create/replace bit and
`capabilities.conditionalCopyDestination.atomicWithSource`.

The physical-key budget applies to the exact key sent to the adapter. It
therefore counts leading slashes for unprefixed drivers, the separator added to
a configured driver prefix, and provider prefixes derived for `list` or
Expand Down Expand Up @@ -1021,6 +1025,18 @@ when a provider adapter and this driver resolve separate copies of `files-sdk`.
`isStorageError()` likewise recognizes branded and exact legacy structural
errors produced by a duplicated `@nestm/storage` package copy.

For a conditional mutation, `error.applied === true` means the provider commit
succeeded but acknowledgement failed afterward, for example in an awaited
post-operation plugin. Conditional uploads also expose `error.appliedEtag` when
the committed generation is known. Do not blindly retry the original
predicate: reconcile the logical destination first, using an exact ETag read
when `appliedEtag` is present. This is a one-way signal: `applied === false`
does not prove that a remote mutation did not commit. A timeout, connection
loss, or exhausted retry can lose the provider's success response, so reconcile
ambiguous transport/provider failures before repeating a conditional mutation.
The sanitized workspace, AI-tool, and gateway error boundaries retain only
this bounded reconciliation metadata.

Capability flags cover range reads, native byte-level upload progress,
delimiter listing, metadata, cache control, resumable uploads, server-side
copy, conditional promotion, and signed transfers.
Expand Down Expand Up @@ -1058,7 +1074,11 @@ StorageModule.forRoot({
Bodies are written verbatim at `<root>/<key>`. A `<key>.meta.json` sidecar beside
each one carries the content type, ETag, and custom metadata a filesystem has
nowhere else to put; sidecars never surface as keys, and uploading a key ending
in `.meta.json` fails closed rather than colliding with one.
in `.meta.json` fails closed rather than colliding with one. Ordinary and
conditional mutations made through the decorated adapter share one
process-local lock domain. Conditional guarantees therefore require a
dedicated root: do not mutate it through another process, an unwrapped adapter,
or direct filesystem calls.

## License

Expand Down
37 changes: 23 additions & 14 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,11 +93,17 @@ additional workflow controls, not the authorization boundary.

Conditional mutations can still have an ambiguous outcome when a remote
provider commits and then loses or violates its response, or when a configured
post-operation plugin fails after the driver has committed. The API fails
closed in that case: inspect the logical destination and reconcile it before
retrying. Create-only and ETag preconditions prevent a blind retry from
silently overwriting a different object, but they cannot make a multi-object
move transactionally atomic.
post-operation plugin fails after the driver has committed. Known post-commit
failures report `StorageError.applied === true`; a conditional upload also
carries `appliedEtag` when its committed generation is known. This is positive
evidence, not a complete remote-commit oracle: a timeout, connection loss, or
exhausted retry can lose a provider success response and still leave
`applied === false`. Do not reissue an ambiguous predicate blindly. Reconcile
the logical destination first, using an exact ETag read when `appliedEtag` is
available and inspecting the relevant source/destination state for copy or
delete. Create-only and ETag preconditions prevent a blind retry from silently
overwriting a different object, but they cannot make a multi-object move
transactionally atomic.

This guarantee covers calls made through `StorageWorkspace`. It does not
confine arbitrary `node:fs`, shell, subprocess, or native-code access in the
Expand All @@ -106,12 +112,15 @@ inside an OS sandbox (container, VM, or equivalent) that exposes only a
materialized workspace. Setting `cwd` to a workspace directory is not
isolation.

For local filesystem storage, use a dedicated service-owned root and do not let
another untrusted process mutate its directory tree concurrently. High-level
Node filesystem checks reject symlinks and hard-linked object files in existing
workspace read and mutation paths, including metadata sidecars, but cannot
provide a race-proof boundary against an actor that can replace path components
between validation and use. For that threat model, mount only the workspace
into a separate UID/container/VM and synchronize approved results back through
storage. The local adapter also commits the body and metadata sidecar as two
files; a process crash between their atomic renames can require reconciliation.
For local filesystem storage, use a dedicated service-owned root and route all
in-process mutations through the package-decorated adapter; its ordinary and
conditional mutation paths share one lock domain. A separate process,
unwrapped adapter, or direct filesystem writer can bypass that coordination and
must not mutate the tree concurrently. High-level Node filesystem checks reject
symlinks and hard-linked object files in existing workspace read and mutation
paths, including metadata sidecars, but cannot provide a race-proof boundary
against an actor that can replace path components between validation and use.
For that threat model, mount only the workspace into a separate
UID/container/VM and synchronize approved results back through storage. The
local adapter also commits the body and metadata sidecar as two files; a process
crash between their atomic renames can require reconciliation.
6 changes: 6 additions & 0 deletions docs/provider-conformance.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,12 @@ Profiles that advertise source and destination predicates but leave
`atomicWithSource` false run the same complete cross-product as negative cases:
every combined pair must fail before it can alter the destination.

When a provider exposes only paired conditional copy, set
`conditionalCopySource.requiresDestinationPredicate` and
`conditionalCopyDestination.requiresSourcePredicate`. The harness then proves
that either predicate fails closed when exercised alone, while still running
the advertised paired combinations through the atomicity matrix.

Run only these suites with:

```sh
Expand Down
Loading