S3 Directory Bucket (One Zone Express) vs Standard Buckets

[bestickley]

I tried S3 Directory Buckets for cdk-nextjs NextjsCache with NextjsGlobalFunctions where we cannot place in same AZ. Directory buckets were not meaningfully faster for this scenario. You need to have same AZ for single digit ms responses. See attached logs and Claude's summary below.

Other factors:

• I had to create my own L2 construct for DirectoryBucket. Implemented IBucket but felt hacky since many of IBuckets methods and properties aren't supported on directory buckets
• Had to create custom directory bucket deployment construct to use copy operation since regular bucket deployment which uses aws s3 sync operation isn't supported.
• Directory buckets require you to specify AZ ID, so we have to keep track of supported regions and AZ IDs (could get out of date). This required cdk-nextjs users to specify region in stack to support this which is more first time setup which we want to avoid
• S3 Standard Buckets just have way more features and are much more widely supported so s3 directory buckets would have to have significant perf gains.

We are leaving on the table cheaper GET and PUT request costs with directory buckets (but more expensive storage), but this benefit is not worth complication without improve perf.

s3-directory-bucket.txt
standard-s3-logs.txt

Based on the logs, S3 Directory Buckets are NOT meaningfully faster for this Next.js caching workload. Here's the analysis:

Performance Comparison
S3 GET Operations
Standard S3:

Range: 37-95ms
Average: ~50-60ms
Directory Buckets:

Range: 27-258ms
Average: ~50-100ms
S3 PUT Operations
Standard S3:

Range: 34-97ms
Average: ~50-70ms
Directory Buckets:

Range: 13-207ms (with outliers up to 7,570ms!)
Average: ~20-50ms, but highly variable
Key Findings
Similar Average Performance: Both configurations show comparable GET/PUT times in the 40-70ms range for most operations

Directory Buckets Have Worse Outliers:

3,905ms PUT for /isr/2
7,570ms PUT for /isr/3

Standard S3 has no such extreme outliers
No Consistent Speed Advantage: Directory buckets occasionally show faster operations (13-20ms), but also show much slower operations

Both Have Streaming Errors: The pipeTo is not a function errors appear in both logs, indicating this is unrelated to bucket type

Conclusion
S3 Directory Buckets with One Zone Express are NOT worth it for this use case. They:

Don't provide meaningful latency improvements
Have higher variance and worse outliers
Cost more than standard S3
Add complexity with single-AZ architecture
For Next.js caching, standard S3 appears to be the better choice.

[bestickley]

For future reference:

`DirectoryBucket` L2 Construct

import {
  RemovalPolicy,
  Stack,
  Names,
  ResourceEnvironment,
  Token,
} from "aws-cdk-lib";
import { Rule } from "aws-cdk-lib/aws-events";
import {
  Grant,
  IGrantable,
  PolicyStatement,
  AddToResourcePolicyResult,
} from "aws-cdk-lib/aws-iam";
import { IKey } from "aws-cdk-lib/aws-kms";
import {
  BucketPolicy,
  IBucket,
  EventType,
  IBucketNotificationDestination,
  NotificationKeyFilter,
  OnCloudTrailBucketEventOptions,
  GrantReplicationPermissionProps,
  TransferAccelerationUrlOptions,
  VirtualHostedStyleUrlOptions,
} from "aws-cdk-lib/aws-s3";
import * as s3express from "aws-cdk-lib/aws-s3express";
import type { BucketReference } from "aws-cdk-lib/interfaces/generated/aws-s3-interfaces.generated";
import { Construct } from "constructs";

export interface DirectoryBucketProps {
  /**
   * Optional prefix for the bucket name. The required suffix `--{az-id}--x-s3` will be automatically appended.
   * If not provided, a name will be generated using the stack name, construct ID, and a unique hash.
   * Note: Use lowercase alphanumeric characters, dots (.), and hyphens (-) only.
   * @example 'my-cache' becomes 'my-cache--use1-az1--x-s3'
   */
  readonly bucketPrefix?: string;
  /**
   * The Availability Zone ID where the bucket will be created (e.g., 'use1-az4').
   * If not provided, will default to the first supported AZ of the stack's region.
   *
   * Auto-detection works for:
   * - Environment-specific stacks: validated at synth time
   * - Environment-agnostic stacks: resolved via CloudFormation Mapping at deploy time
   *
   * @default - Auto-detected from stack region for supported regions
   */
  readonly availabilityZone?: string;
  /**
   * Policy to apply when the bucket is removed from the stack
   * @default RemovalPolicy.RETAIN
   */
  readonly removalPolicy?: RemovalPolicy;
}

/**
 * S3 Express One Zone Directory Bucket wrapper that implements IBucket.
 *
 * This construct provides an L2-like interface for S3 Express One Zone directory buckets,
 * which are designed for high-performance, single-AZ storage with single-digit millisecond latency.
 *
 * @see https://aws.amazon.com/s3/storage-classes/express-one-zone/
 */
export class DirectoryBucket extends Construct implements IBucket {
  // Static members
  /**
   * Static mapping of AWS regions to their first Availability Zone ID that support directory buckets.
   * Only includes regions that support S3 Express One Zone directory buckets.
   * @see https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-express-Endpoints.html
   */
  public static readonly AZ_ID_MAP: Record<string, string> = {
    "us-east-1": "use1-az4",
    "us-east-2": "use2-az1",
    "us-west-2": "usw2-az1",
    "ap-northeast-1": "apne1-az1",
    "ap-south-1": "aps1-az1",
    "eu-north-1": "eun1-az1",
    "eu-west-1": "euw1-az1",
  };

  /**
   * Get Availability Zone ID for a given region.
   * @param region - AWS region code (e.g., 'us-east-1')
   * @returns Availability Zone ID (e.g., 'use1-az4') or undefined if not mapped
   */
  public static getAzIdForRegion(region: string): string | undefined {
    return DirectoryBucket.AZ_ID_MAP[region];
  }

  /**
   * Get list of regions that support S3 Express One Zone.
   * @returns Array of supported region codes
   */
  public static supportedRegions(): string[] {
    return Object.keys(DirectoryBucket.AZ_ID_MAP);
  }

  /**
   * Check if a region supports S3 Express One Zone directory buckets.
   * @param region - AWS region code to check
   * @returns true if region supports S3 Express One Zone
   */
  public static isRegionSupported(region: string): boolean {
    return region in DirectoryBucket.AZ_ID_MAP;
  }

  // Instance fields
  readonly bucketArn: string;
  readonly bucketName: string;
  readonly isWebsite?: boolean;
  readonly env: ResourceEnvironment;
  readonly stack: Stack;
  readonly bucketRef: BucketReference;

  private readonly directoryBucket: s3express.CfnDirectoryBucket;
  private _policy?: BucketPolicy;

  constructor(scope: Construct, id: string, props: DirectoryBucketProps) {
    super(scope, id);

    // Determine availability zone - must be resolved at synth time
    const availabilityZone = this.determineAvailabilityZone(props);

    // Generate bucket prefix if not provided
    const prefix = props.bucketPrefix ?? this.generateBucketPrefix();

    // Build bucket name with required suffix
    const suffix = `--${availabilityZone}--x-s3`;
    const bucketName = prefix.endsWith(suffix)
      ? prefix // Already has suffix
      : `${prefix}${suffix}`; // Add suffix

    this.directoryBucket = new s3express.CfnDirectoryBucket(this, "Resource", {
      bucketName,
      dataRedundancy: "SingleAvailabilityZone",
      locationName: availabilityZone,
      bucketEncryption: {
        serverSideEncryptionConfiguration: [
          {
            serverSideEncryptionByDefault: {
              sseAlgorithm: "AES256",
            },
          },
        ],
      },
    });

    if (props.removalPolicy) {
      this.directoryBucket.applyRemovalPolicy(props.removalPolicy);
    }

    this.bucketArn = this.directoryBucket.attrArn;
    this.bucketName = this.directoryBucket.ref;
    this.isWebsite = false;
    this.stack = Stack.of(this);
    this.env = this.stack.env;
    this.bucketRef = {
      bucketArn: this.bucketArn,
      bucketName: this.bucketName,
    };
  }

  /**
   * Access the underlying CfnDirectoryBucket for advanced configuration
   */
  public get cfnDirectoryBucket(): s3express.CfnDirectoryBucket {
    return this.directoryBucket;
  }

  public get bucketRegionalDomainName(): string {
    throw new Error(
      "Regional domain names are not supported for S3 Express One Zone. Use bucketName instead.",
    );
  }

  public get bucketDomainName(): string {
    throw new Error(
      "Domain names are not supported for S3 Express One Zone. Use bucketName instead.",
    );
  }

  public get bucketWebsiteUrl(): string {
    throw new Error(
      "Static website hosting is not supported for S3 Express One Zone",
    );
  }

  public get bucketWebsiteDomainName(): string {
    throw new Error(
      "Static website hosting is not supported for S3 Express One Zone",
    );
  }

  public get bucketDualStackDomainName(): string {
    throw new Error(
      "Dual-stack endpoints are not supported for S3 Express One Zone",
    );
  }

  public get encryptionKey(): IKey | undefined {
    throw new Error(
      "Custom encryption keys are not supported for S3 Express One Zone. Only S3-managed encryption (AES256) is available.",
    );
  }

  public get policy(): BucketPolicy | undefined {
    return this._policy;
  }

  public set policy(_value: BucketPolicy | undefined) {
    throw new Error(
      "Bucket policies are not yet supported for S3 Express One Zone",
    );
  }

  /**
   * Generate a bucket prefix similar to how L2 Bucket construct works.
   * Uses hyphens for S3 Express One Zone format (underscores not allowed in bucket name).
   *
   * Must match syntax: `[bucket_name]--[azid]--x-s3`
   */
  private generateBucketPrefix(): string {
    // Use CDK's built-in unique resource name generator
    const uniqueName = Names.uniqueResourceName(this, {
      maxLength: 45, // Reserve space for suffix: --{az-id}--x-s3 (roughly 18 chars)
    });

    // Sanitize to ensure it meets S3 Express requirements
    const sanitized = uniqueName
      .toLowerCase()
      .replace(/[^a-z0-9-]/g, "-") // Allow only lowercase alphanumeric and hyphens
      .replace(/-+/g, "-") // Replace multiple hyphens with single
      .replace(/^-|-$/g, ""); // Remove leading/trailing hyphens

    return sanitized;
  }

  /**
   * Determine the Availability Zone ID for the directory bucket.
   * Priority: 1) props.availabilityZone, 2) resolved region lookup, 3) error.
   * Directory buckets require the AZ to be known at synth time (no token support).
   */
  private determineAvailabilityZone(props: DirectoryBucketProps): string {
    // Priority 1: Use explicit availabilityZone from props
    if (props.availabilityZone) {
      return props.availabilityZone;
    }

    // Priority 2: Try to resolve from stack region
    const stack = Stack.of(this);
    const region = stack.region;

    // Directory buckets require resolved AZ at synth time - cannot use tokens
    if (Token.isUnresolved(region)) {
      throw new Error(
        `DirectoryBucket requires a resolved region at synth time. ` +
          `For environment-agnostic stacks, either: 1) specify 'env' in Stack props, or 2) provide 'availabilityZone' explicitly in DirectoryBucketProps. ` +
          `Supported regions: ${DirectoryBucket.supportedRegions().join(", ")}`,
      );
    }

    // Look up AZ ID from region
    const azId = DirectoryBucket.getAzIdForRegion(region);
    if (!azId) {
      throw new Error(
        `Region "${region}" does not support S3 Express One Zone directory buckets. ` +
          `Supported regions: ${DirectoryBucket.supportedRegions().join(", ")}. ` +
          `Alternatively, provide 'availabilityZone' explicitly in DirectoryBucketProps.`,
      );
    }

    return azId;
  }

  public grantRead(identity: IGrantable, _objectsKeyPattern?: any): Grant {
    return Grant.addToPrincipal({
      grantee: identity,
      actions: ["s3express:CreateSession", "s3:GetObject", "s3:ListBucket"],
      resourceArns: [this.bucketArn, `${this.bucketArn}/*`],
    });
  }

  public grantWrite(
    identity: IGrantable,
    _objectsKeyPattern?: any,
    _allowedActionPatterns?: string[],
  ): Grant {
    return Grant.addToPrincipal({
      grantee: identity,
      actions: [
        "s3express:CreateSession",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket",
      ],
      resourceArns: [this.bucketArn, `${this.bucketArn}/*`],
    });
  }

  public grantReadWrite(identity: IGrantable, _objectsKeyPattern?: any): Grant {
    return Grant.addToPrincipal({
      grantee: identity,
      actions: [
        "s3express:CreateSession",
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket",
      ],
      resourceArns: [this.bucketArn, `${this.bucketArn}/*`],
    });
  }

  public grantDelete(identity: IGrantable, _objectsKeyPattern?: any): Grant {
    return Grant.addToPrincipal({
      grantee: identity,
      actions: ["s3express:CreateSession", "s3:DeleteObject"],
      resourceArns: [this.bucketArn, `${this.bucketArn}/*`],
    });
  }

  public grantPut(identity: IGrantable, _objectsKeyPattern?: any): Grant {
    return Grant.addToPrincipal({
      grantee: identity,
      actions: ["s3express:CreateSession", "s3:PutObject"],
      resourceArns: [this.bucketArn, `${this.bucketArn}/*`],
    });
  }

  public grantPutAcl(
    _identity: IGrantable,
    _objectsKeyPattern?: string,
  ): Grant {
    throw new Error("ACLs are not supported for S3 Express One Zone");
  }

  public grantPublicAccess(
    _keyPrefix?: string,
    ..._allowedActions: string[]
  ): Grant {
    throw new Error("Public access is not supported for S3 Express One Zone");
  }

  public addToResourcePolicy(
    _statement: PolicyStatement,
  ): AddToResourcePolicyResult {
    throw new Error(
      "Resource policies are not yet supported for S3 Express One Zone",
    );
  }

  public arnForObjects(keyPattern: string): string {
    return `${this.bucketArn}/${keyPattern}`;
  }

  public s3UrlForObject(key?: string): string {
    const urlKey = key ? `/${key}` : "";
    return `https://${this.bucketName}.s3express-${this.env.region}.amazonaws.com${urlKey}`;
  }

  public urlForObject(key?: string): string {
    return this.s3UrlForObject(key);
  }

  public virtualHostedUrlForObject(
    key?: string,
    _options?: VirtualHostedStyleUrlOptions,
  ): string {
    return this.s3UrlForObject(key);
  }

  public transferAccelerationUrlForObject(
    _key?: string,
    _options?: TransferAccelerationUrlOptions,
  ): string {
    throw new Error(
      "Transfer acceleration is not supported for S3 Express One Zone",
    );
  }

  public addCorsRule(_rule: any): void {
    throw new Error("CORS is not supported for S3 Express One Zone");
  }

  public addLifecycleRule(_rule: any): void {
    console.warn(
      "Lifecycle rules should be configured via the CfnDirectoryBucket props",
    );
  }

  public addMetric(_metric: any): void {
    throw new Error("Metrics are not yet supported for S3 Express One Zone");
  }

  public addEventNotification(
    _event: EventType,
    _dest: IBucketNotificationDestination,
    ..._filters: NotificationKeyFilter[]
  ): void {
    throw new Error(
      "Event notifications are not yet supported for S3 Express One Zone",
    );
  }

  public addObjectCreatedNotification(
    _dest: IBucketNotificationDestination,
    ..._filters: NotificationKeyFilter[]
  ): void {
    throw new Error(
      "Event notifications are not yet supported for S3 Express One Zone",
    );
  }

  public addObjectRemovedNotification(
    _dest: IBucketNotificationDestination,
    ..._filters: NotificationKeyFilter[]
  ): void {
    throw new Error(
      "Event notifications are not yet supported for S3 Express One Zone",
    );
  }

  public addInventory(_inventory: any): void {
    throw new Error("Inventory is not supported for S3 Express One Zone");
  }

  public enableEventBridgeNotification(): void {
    throw new Error(
      "EventBridge notifications are not yet supported for S3 Express One Zone",
    );
  }

  public onCloudTrailEvent(
    _id: string,
    _options?: OnCloudTrailBucketEventOptions,
  ): Rule {
    throw new Error(
      "CloudTrail events are not yet supported for S3 Express One Zone",
    );
  }

  public onCloudTrailPutObject(
    _id: string,
    _options?: OnCloudTrailBucketEventOptions,
  ): Rule {
    throw new Error(
      "CloudTrail events are not yet supported for S3 Express One Zone",
    );
  }

  public onCloudTrailWriteObject(
    _id: string,
    _options?: OnCloudTrailBucketEventOptions,
  ): Rule {
    throw new Error(
      "CloudTrail events are not yet supported for S3 Express One Zone",
    );
  }

  public grantReplicationPermission(
    _identity: IGrantable,
    _props: GrantReplicationPermissionProps,
  ): Grant {
    throw new Error("Replication is not supported for S3 Express One Zone");
  }

  public addReplicationPolicy(
    _roleArn: string,
    _accessControlTransition?: boolean,
    _account?: string,
  ): void {
    throw new Error("Replication is not supported for S3 Express One Zone");
  }

  public applyRemovalPolicy(policy: RemovalPolicy): void {
    this.directoryBucket.applyRemovalPolicy(policy);
  }
}

`DirectoryBucketDeployment` (uses copy instead of unsupported sync)

import { CustomResource, Duration, RemovalPolicy } from "aws-cdk-lib";
import { PolicyStatement } from "aws-cdk-lib/aws-iam";
import { IBucket } from "aws-cdk-lib/aws-s3";
import { Asset } from "aws-cdk-lib/aws-s3-assets";
import { Construct } from "constructs";
import { DirectoryBucketDeploymentFunction } from "../lambdas/directory-bucket-deployment/directory-bucket-deployment-function";

export interface DirectoryBucketDeploymentSource {
  /**
   * Local file path or directory to deploy
   */
  readonly asset: Asset;
}

export interface DirectoryBucketDeploymentProps {
  /**
   * The sources to deploy to the directory bucket
   */
  readonly sources: DirectoryBucketDeploymentSource[];

  /**
   * The destination directory bucket
   */
  readonly destinationBucket: IBucket;

  /**
   * Key prefix in the destination bucket
   * @default - no prefix
   */
  readonly destinationKeyPrefix?: string;
}

/**
 * Custom deployment construct for S3 Express One Zone directory buckets.
 * Uses AWS SDK v3 to copy files, since 'aws s3 sync' doesn't support directory buckets.
 */
export class DirectoryBucketDeployment extends Construct {
  readonly customResource: CustomResource;

  constructor(
    scope: Construct,
    id: string,
    props: DirectoryBucketDeploymentProps,
  ) {
    super(scope, id);

    if (props.sources.length === 0) {
      throw new Error("At least one source must be provided");
    }

    if (props.sources.length > 1) {
      throw new Error(
        "Multiple sources not yet supported for DirectoryBucketDeployment",
      );
    }

    const source = props.sources[0];

    // Create Lambda function for custom resource handler
    const handlerFunction = new DirectoryBucketDeploymentFunction(
      this,
      "Handler",
      {
        timeout: Duration.minutes(15),
        memorySize: 1024,
      },
    );

    // Grant permissions to read from source bucket
    source.asset.grantRead(handlerFunction);

    // Grant permissions to write to destination directory bucket
    props.destinationBucket.grantReadWrite(handlerFunction);

    // Add s3express:CreateSession permission for directory bucket
    handlerFunction.addToRolePolicy(
      new PolicyStatement({
        actions: ["s3express:CreateSession"],
        resources: [props.destinationBucket.bucketArn],
      }),
    );

    // Create custom resource
    this.customResource = new CustomResource(this, "CustomResource", {
      serviceToken: handlerFunction.functionArn,
      removalPolicy: RemovalPolicy.DESTROY,
      properties: {
        sourceBucketName: source.asset.s3BucketName,
        sourceObjectKey: source.asset.s3ObjectKey,
        destinationBucketName: props.destinationBucket.bucketName,
        destinationPrefix: props.destinationKeyPrefix,
        // Add timestamp to force update on each deployment
        timestamp: Date.now().toString(),
      },
    });
  }
}