DIAL Folder As Resource (agent skills as a folder-shaped resource)

[astsiapanay]

DIAL Folder As Resource

Type: EPIC / Design

Issue tracker

Child issue	Status
#1634 — Foundation: SKILL type, v2 route, whole-resource PUT/GET	[X]
#1635 — Whole-resource DELETE (tombstone marker state)	[X]
#1636 — Single-file GET/PUT/DELETE inside a skill	[Х]
#1637 — Per-resource limits & skill validation hardening	[X]
#1638 — Metadata listing + DIAL folders (auto-vivify, structural invariants)	[X]
#1639 — GC / reconciliation sweep	[X]
#1640 — Access control integration	[X]
#1641 — Share a skill as a whole	[X]
#1642 — Publish a skill (review → public flow)	[X]

Problem

DIAL Core today models every resource as a single blob. The v1 API family
^/v1/(files|conversations|prompts|applications|toolsets)/(?<bucket>[a-zA-Z0-9]+)/(?<path>.*)$
manages each resource as one blob, and folders are purely virtual — computed on
the fly from blob listings and never stored as entities.

We want to support agent skills (Agent Skills specification)
as a DIAL resource. A skill is not a single blob — it is a folder containing a
mandatory SKILL.md plus an arbitrary hierarchy of auxiliary files. DIAL Core has
no notion of a "resource that is a folder".

Rather than building a skill-specific mechanism, we introduce a generic
folder-as-resource primitive and make skills its first concrete instance, keeping
a clean seam for future folder-shaped resource types.

Use cases

1. Get / put / delete an entire agent skill as a unit (download as an archive,
   replace the whole file set, remove the whole folder).
2. List skill metadata (enumerate skills and grouping folders, each with parsed
   name/description/version and version identifier).
3. Get / put / delete a single file inside a skill folder.
4. Organize skills into grouping folders within a bucket.
5. Share a skill as a whole and publish it through the existing review →
   public flow.

Rejected alternatives

• A. Skill = folder of FILE resources under files/{bucket}/skills/my-skill/.
  Rejected: no server-side notion of the skill as a unit; any /v1/files/... call
  can corrupt it.
• B. Skill = new SKILL type stored as a packed archive (zip/tar). Rejected:
  opaque single blob, no per-file access, awkward partial edits.
• C. Skill = JSON envelope with embedded file contents (like
  Application.Function's code bundle). Rejected: does not scale to arbitrary binary
  aux files; whole-resource read/write only.

Technical requirements

1. Introduce two generic concepts:
   • DIAL resource — a folder that is one logical resource, identified by a marker
     file .dial-resource. May contain an arbitrary file hierarchy.
   • DIAL folder — a grouping folder, identified by a marker file .dial-folder,
     used to organize DIAL resources.
2. Enforce structural invariants:
   • A DIAL resource must not be stored inside another DIAL resource.
   • DIAL resources may be grouped under DIAL folders (or live at bucket root).
   • An arbitrary file hierarchy is allowed inside a DIAL resource.
3. The marker file is the only source of truth that a folder is a DIAL
   resource/folder. No secondary index. (Listing/classification pay O(N) marker reads
   per page, bounded by per-resource limits.)
4. Whole-resource and per-file operations must keep the resource consistent under
   concurrency and partial failure (no half-written or markerless folders observable).
5. The resource as a whole must support optimistic concurrency (If-Match).
6. Skills must be validated server-side (mandatory SKILL.md, parseable frontmatter).
7. Must integrate with existing access control, sharing, and publication.
8. Must respect platform limits and add new ones (per-file size, per-resource file
   count and total size).
9. Reuse existing storage primitives (blob store, Redis cache, distributed locking)
   and the Vert.x non-blocking model (no event-loop blocking on large archives).

Solution overview

Concepts and markers

A folder becomes a DIAL resource/folder solely by the presence of a marker file:

• .dial-resource — marks a DIAL resource.
• .dial-folder — marks a grouping DIAL folder.

graph TD
    B["bucket"]
    F1["category/<br/>(.dial-folder)"]
    S1["my-skill/<br/>(.dial-resource, type=skill)"]
    S2["other-skill/<br/>(.dial-resource, type=skill)"]
    M["SKILL.md"]
    SC["scripts/run.sh"]
    AS["assets/logo.png"]
    MK[".dial-resource (marker, hidden)"]

    B --> F1
    F1 --> S1
    F1 --> S2
    S1 --> MK
    S1 --> M
    S1 --> SC
    S1 --> AS

    classDef folder fill:#e3f2fd,stroke:#1565c0;
    classDef resource fill:#e8f5e9,stroke:#2e7d32;
    classDef file fill:#fff8e1,stroke:#f9a825;
    classDef marker fill:#f3e5f5,stroke:#6a1b9a,stroke-dasharray: 4 3;
    class B,F1 folder;
    class S1,S2 resource;
    class M,SC,AS file;
    class MK marker;

Loading

The marker is a small self-describing JSON document:

// .dial-resource
{
  "type": "skill",          // concrete folder-resource type
  "schemaVersion": 1,
  "state": "ready",         // lifecycle: "ready" | "deleting"
  "currentVersion": "...",  // pointer to immutable version subtree (see atomicity)
  "etag": "<aggregate-etag>",
  "createdAt": ..., "updatedAt": ..., "author": ...,
  "deletedAt": ...,         // set when state="deleting"; drives restore/GC grace window
  "metadata": {             // type-specific cached metadata
    "name": "...",          // for skills: parsed from SKILL.md frontmatter
    "description": "...",
    "version": "..."
  }
}

A marker in state: "deleting" is a tombstone: classification, listing, and all
access paths treat it as absent. The marker is the only source of truth; there is no
Redis index of folder-resources.

Storage namespace

Skills get a new, isolated resource type SKILL with URL group skills, stored
under its own blob prefix. The v1 router does not know the skills group, so skill
blobs are invisible to the v1 files API — every mutation flows through the new
v2 handler, and the resource's invariants cannot be bypassed.

API (new v2 family)

All routes use a reluctant path component; the segment name files and v/ are
reserved structural tokens.

# Whole resource
GET    /v2/skills/{bucket}/{path}            -> stream a ZIP of the skill
PUT    /v2/skills/{bucket}/{path}            -> multipart/form-data; replace entire skill
DELETE /v2/skills/{bucket}/{path}         -> skill: remove tree; DIAL folder: only if empty

# DIAL Folder
GET    /v2/skills/{bucket}/{path}/            -> 400
PUT    /v2/skills/{bucket}/{path}/            -> create folder, 400 if folder already exists
DELETE /v2/skills/{bucket}/{path}/         -> delete only if empty

# Single file inside a resource
GET    /v2/skills/{bucket}/{path}/files/{filePath}
PUT    /v2/skills/{bucket}/{path}/files/{filePath}
DELETE /v2/skills/{bucket}/{path}/files/{filePath}

# Metadata / listing
GET    /v2/metadata/skills/{bucket}/{path}                   -> list skills + DIAL folders here
GET    /v2/metadata/skills/{bucket}/{path}/files/{filePath}  -> list files inside a skill

Archive contract (asymmetric):

• GET of a whole resource returns application/zip. The .dial-resource marker is
  stripped. The archive is produced by streaming zip entries on a worker
  thread, never buffering the whole archive in memory.
• PUT of a whole resource accepts multipart/form-data with one part per file;
  each part's filename is the file's relative path inside the resource. The marker
  is synthesized server-side so clients can never corrupt it.

GET on a DIAL folder returns 400 (use metadata listing). DELETE on a DIAL
folder succeeds only if it is empty.

Per-type behavior (generic engine + handler)

A generic engine handles all folder/marker/archive/lock mechanics. Per-type rules
plug in through a registered handler keyed by group:

• validate(files) — skills: require SKILL.md at root; YAML frontmatter must parse
  with required name and description.
• buildMarkerMetadata(files) — extract type-specific cached metadata (skills:
  name, description, version).
• validateFileMutation(relativePath, op) — guard single-file ops (e.g. reject
  deleting SKILL.md).

Skills are the first handler; future folder-resource types register their own.

Versioning and single-file consistency

The resource carries a single aggregate etag in the marker, bumped on any
mutation. This gives clients If-Match on the skill as a unit and a cheap "did this
skill change" signal. Single-file operations re-run the handler's validation and
update the marker (deleting SKILL.md is rejected; editing it re-parses frontmatter
and refreshes cached metadata; every successful mutation bumps the aggregate etag and
updatedAt).

Listing

GET /v2/metadata/skills/{bucket}/{path} lists immediate children at a grouping
level, reads each child's marker, and returns classified, enriched items:
nodeType = DIAL_RESOURCE | DIAL_FOLDER, plus cached metadata (type, name,
description, etag, updatedAt). Plain folders without a marker are ignored.
Pagination reuses the existing token mechanism.

Atomicity — versioned-prefix layout (recommended approach)

The "write-new-then-promote" promote step is not atomic on DIAL's storage layer:
BlobStorage exposes only per-object store/load/delete/copy/list (no atomic
multi-object commit, no atomic rename, no blob CAS); copyFolder/deleteFolder are
loop-based and partially failable; LockService is a Redisson lock with a 5-minute
auto-expiring lease. The only atomic unit is a single-object write, so the commit of
a multi-file change must hinge on exactly one single-object write.

Adopt the versioned-prefix layout — the marker is a pointer to an immutable
per-version subtree, and the marker write is the one atomic operation that flips the
whole resource:

my-skill/
  .dial-resource              # { state, currentVersion, etag, metadata, ... }
  v/{versionId}/SKILL.md
  v/{versionId}/scripts/run.sh
  v/{versionId}/assets/logo.png

Whole-resource PUT:

1. Take the folder/bucket lock (per structural-vs-content rules).
2. Write all parts under a fresh v/{versionId}/ prefix (pure creates, never
   touch live data). Run validate(files) + buildMarkerMetadata(files); compute the
   aggregate etag.
3. Commit = a single putResource of .dial-resource with currentVersion = versionId (and the new etag), guarded by If-Match. This is the linearization point.
4. The old version prefix is now unreferenced; an async sweep deletes it after a grace
   period.

Reads resolve marker → currentVersion → read under that prefix; an in-flight reader
of the old version keeps a consistent snapshot. Crash safety is total — no
partial-tree state is ever observable.

Single-file PUT/DELETE: default to copy-on-write of a new version (server-side
BlobStorage.copy is native — N cheap metadata copies), apply the change, commit the
marker. A cheaper in-place variant is acceptable since each file op is a single-object
write.

GC / sweep: one background job deletes v/{versionId}/ prefixes that no marker
references and that are older than a grace TTL. This unifies "staging orphaned by a
crashed PUT" and "superseded old versions" into one concept (unreferenced version
prefix). Reuses deleteFolder.

sequenceDiagram
    participant C as Client
    participant Ctl as FolderResourceController
    participant Svc as FolderResourceService
    participant H as SkillHandler
    participant Blob as Blob storage
    participant Sweep as GC sweep

    C->>Ctl: PUT /v2/skills/.../ (multipart file parts)
    Ctl->>Svc: replace(resource, parts)
    Svc->>Svc: acquire lock (folder or bucket)
    Svc->>Blob: write all parts under fresh v/{versionId}/ (invisible)
    Svc->>H: validate(files) + buildMarkerMetadata(files)
    H-->>Svc: ok + name/description/version
    Svc->>Svc: compute aggregate etag
    Svc->>Blob: putResource(.dial-resource){currentVersion, etag} If-Match  %% single atomic commit
    Svc->>Svc: release lock
    Svc-->>Ctl: aggregate etag
    Ctl-->>C: 200 OK with ETag
    Note over Sweep,Blob: later: delete old v/{prevVersionId}/ (unreferenced, past grace TTL)

Loading

Upgrade path (Option 2): content-addressed blobs (blobs/{sha256}) + manifest
marker for dedup across versions — adopt once incremental edits over large shared
asset sets become hot.

Atomicity — whole-resource DELETE (tombstone marker state)

The recursive deleteFolder is a paginated per-object loop with no rollback. Apply
the same fix: commit on a single marker write; demote tree removal to best-effort +
async reclamation.

Give the marker an explicit lifecycle field state: ready | deleting:

1. Take the bucket lock (DELETE is structural — removes the resource and may affect
   ancestor-folder emptiness) and validate If-Match against the marker's aggregate etag.
2. Clean up access-control metadata as a unit, reusing the existing delete path
   (shareService.revokeSharedResource, invitationService.cleanUpResourceLink).
3. Commit = a single putResource of .dial-resource rewriting it with state: deleting (and deletedAt), guarded by If-Match. From here all access paths treat
   the resource as gone (404); the change propagates via ResourceEvent.
4. Best-effort deleteFolder of the v/* tree, then delete the marker itself.

A reconciliation sweep finds markers left in state: deleting, idempotently
finishes deleting the tree, then deletes the marker. The tombstone is an explicit
durable record of delete intent that also enables restore (flip deleting → ready
within a grace window) and a clean audit trail. No partial-tree state is ever
observable. DELETE on a .dial-folder follows the identical pattern, succeeding only
if empty.

Cross-cutting: lock-lease expiry is safe (marker is already deleting); re-create
while deleting supersedes the tombstone with a fresh version; folder emptiness is not
auto-cascaded (mirror of PUT's auto-vivify asymmetry).

Reserved names

The path segment files is reserved (structural token) — neither a DIAL folder
nor a DIAL resource may be named files. Keeps the .../{path}/files/{filePath}
grammar unambiguous. v/ is likewise reserved by the versioned layout.

Access control, sharing, publication

• Access control: owner access by bucket-location match (no marker walks).
  In-resource file access inherits from the enclosing resource's folder URL via the
  existing recursive parent-permission lookup.
• Sharing: the whole skill is the shared unit; files beneath inherit access.
  Skills get an entry in the share limits table.
• Publication: the whole skill is the published unit. Publishing copies the
  entire tree through source → review → public, re-synthesizing a valid marker
  (correct target URL, fresh etag, fresh version) at each hop, and adds SKILL to the
  set of publishable types.

Limits and validation

• Reuse the existing 512 MB per-file cap.
• Add configurable per-resource limits: maxFiles (default ~1000) and
  maxTotalBytes (default ~1 GB) — bounds listing reads, archive size, lock-hold time.
• Skill validation: SKILL.md required at root; frontmatter must parse with
  required name + description; name/description/version extracted into the
  marker. Invalid skills are rejected with 400.

Operational changes

• New static settings: per-resource maxFiles and maxTotalBytes.
• Storage layout: new top-level blob prefix for the skills type; markers are part
  of the data (no separate index to rebuild on backup/restore).
• Transient artifacts: unreferenced v/{versionId}/ prefixes cleaned by the
  periodic GC sweep; excluded from listings.
• Distributed locking: structural ops take the bucket lock; content updates use
  finer-grained per-resource locks.
• No change to the v1 API; v2 is additive.

Agent Skill authorization matrix

A future EXECUTE permission is planned (today there are READ and WRITE):

Authentication method	Permission	What user can do
API-key/JWT	READ	Read skill content
API-key/JWT	WRITE	Write/delete skill content
API-key/JWT	EXECUTE	Application may read content. No direct access
Per request API Key	READ	Read skill content
Per request API Key	WRITE	Write/delete skill content
Per request API Key	EXECUTE	Application may read content. No direct access

For now, users with READ may read and execute skills.

References

1. Agent Skills specification: https://agentskills.io/specification
2. Existing resource model: ResourceTypes, ResourceDescriptor, ResourceService
   (copyFolder/deleteFolder/getFolderMetadata), LockService (lockResource,
   underBucketLock), RouteTemplate / ControllerSelector.