Extend AZ/region discovery to FSx deployment types (which AZs offer PERSISTENT_2 Lustre?)

[scttfrdmn]

Motivation

truffle's core value is 'discover which AWS regions and AZs support specific EC2 instance types' — via DescribeInstanceTypeOfferings(LocationType=AvailabilityZone). The same discovery question exists for FSx, and there's no read-only way to answer it today.

Concrete case that motivated this: trying to create an FSx for Lustre PERSISTENT_2 filesystem in us-east-1, every attempt in us-east-1a/1b/1c failed with:

BadRequest: The requested Lustre configuration: PERSISTENT_2 is not available in this availability zone.

Quota was NOT the issue (Persistent_2 file systems = 100, storage = 100,800 GB — full headroom). It's a per-AZ service-offering condition. With no lookup API exposed, the only way to find a working AZ was to blind-probe by issuing real create-file-system calls across AZs — billable, slow, and exactly the kind of trial-and-error truffle exists to eliminate for EC2.

Proposal

Add FSx deployment-type AZ/region discovery, mirroring the existing az/find model. E.g.:

truffle fsx lustre persistent_2 --regions us-east-1     # which AZs offer it
truffle az --service fsx --offering lustre:persistent_2 # AZ-first view, like EC2

Output: per-AZ availability (✓/✗) for the FSx type, same shape as instance-type-per-AZ today. This lets a caller (e.g. spawn --fsx-create, or a benchmark harness) pick a viable AZ up front instead of discovering unavailability via a failed create.

Implementation note

AWS doesn't expose a clean DescribeFileSystemTypeOfferings the way EC2 has DescribeInstanceTypeOfferings, so this likely needs either (a) the FSx/pricing/SSM offering data if available, or (b) a documented static map of FSx-type × AZ availability refreshed periodically, or (c) a safe capability probe. Worth investigating which AWS surface (pricing API / SSM /aws/service/ parameters / FSx itself) exposes per-AZ deployment-type availability without a create call.

Context

Consumer: aws-microbiome-demo, which already uses truffle for EC2 vCPU-quota + spot discovery to size a Nextflow fan-out. It needs a shared FSx Lustre (PERSISTENT_2) for a wide-fan-out reference-DB mount (see nf-spawn#67, spawn#206); picking the right AZ is currently blind. Filing as a feature request — not modifying truffle.

[scttfrdmn]

Since this is your project — some AWS-surface recon to make implementation concrete (probed on acct 942542972736, us-east-1):

Pricing API carries FSx deployment options at REGION granularity (not AZ):
aws pricing get-products --service-code AmazonFSx --filters fileSystemType=Lustre regionCode=us-east-1 returns Lustre deploymentOption=Persistent at throughput tiers 50/250/500 MB/s/TiB for us-east-1. And get-attribute-values --service-code AmazonFSx --attribute-name deploymentOption enumerates Single-AZ_2 / Multi-AZ / Persistent / etc. So 'is PERSISTENT_2 offered in region X (and at which throughput tiers)?' is answerable read-only via Pricing — that alone would have saved us, since it'd show whether to even try a region.

The gap: per-AZ. Our actual failure was 'not available in this availability zone' (us-east-1a/1b/1c all rejected PERSISTENT_2, but pricing says the region has it). I could not find an AWS API that exposes FSx deployment-type availability per-AZ without a create call — EC2 has DescribeInstanceTypeOfferings(LocationType=AvailabilityZone), FSx has no equivalent. Quota API confirms it's not a limit (Persistent_2 fs=100, storage=100,800GB).

So a pragmatic two-tier design:

1. Region tier (easy, read-only via Pricing API): truffle fsx lustre persistent_2 --regions ... → which regions offer it + throughput tiers. Mirrors the existing pricing/spot plumbing.
2. AZ tier (harder): either a safe capability probe, a maintained static map, or wait for AWS to expose an offerings API. Could degrade gracefully — 'region offers it; AZ availability unknown, try and fall back.'

The demo would consume tier 1 immediately (pick a region known to offer PERSISTENT_2 before building the FSx), and tier 2 later for AZ selection. Happy to test against this account as you build it.

[scttfrdmn]

API-surface investigation — is per-AZ FSx availability discoverable read-only?

Checked the candidate surfaces against a real account (us-east-1). Short answer: the per-AZ question that motivated this issue has no clean read-only API. The region-level question does.

What's true about FSx availability

• A Lustre PERSISTENT_2 file system is single-AZ — per the AWS docs, "data is automatically replicated within the same Availability Zone in which the file system is located." You choose the AZ via the subnet at create time.
• AWS only documents availability at the Region (and Local Zone) level — the Deployment type availability table is per-region (e.g. "US East (N. Virginia) ✓ Persistent 2").
• The PERSISTENT_2 is not available in this availability zone error is a real, undocumented, sub-region condition: a region can be ✓ overall while specific AZs don't yet offer the deployment type (hardware-generation rollouts are AZ-by-AZ). There is no published per-AZ table.

Read-only surfaces checked

Surface	Result
FSx API	❌ No describe-*-offerings operation exists (unlike EC2's DescribeInstanceTypeOfferings). FSx exposes only describe-file-systems, describe-backups, etc.
CreateFileSystem DryRun	❌ No DryRun parameter — so no free capability probe.
Pricing API (AmazonFSx)	⚠️ Has deploymentOption (Persistent, Single-AZ_2, Multi-AZ-2, …) and location/locationType — but locationType is only AWS Region / AWS Local Zone, never per-AZ. Good for region-level, useless for per-AZ.
SSM public parameters (/aws/service/...)	❌ No FSx/Lustre offering namespace.

Implication for this feature

• Region-level ("which regions offer Persistent_2 / Scratch_2 / Intelligent-Tiering") is cleanly buildable from the Pricing API (or the static doc table), mirroring truffle's EC2 find/az model at region granularity.
• Per-AZ ("which AZs in us-east-1 offer Persistent_2") — the thing that actually bit aws-microbiome-demo — is not answerable read-only. The only ways are the two this issue already flagged as bad:
  1. a billable CreateFileSystem probe per AZ (slow, real cost), or
  2. a hand-maintained static AZ map that goes stale as AWS rolls out hardware.

Recommendation

Keep #66 open. When built, scope it to region-level discovery (honest about what the APIs expose) and document the per-AZ limitation, rather than implying truffle can answer per-AZ the way it does for EC2 instance types. An opt-in, loudly-flagged create-probe (--probe-az, off by default) could be a later add for callers who accept the cost. No code yet — filing this so the constraint is captured before implementation.

[scttfrdmn]

Correction / update — the motivating case was substantially a spawn bug (spawn#208, fixed in v0.59.0)

My earlier comment framed this around "per-AZ unavailability with no read-only API." After reproducing it, most of the motivating pain was not an AWS discovery gap — it was a spawn bug:

spawn#208: spawn launch --fsx-create --az <zone> applied --az to the EC2 instance but never to the FSx — the filesystem fell back to subnets[0] of the default VPC regardless of --az. So the field report's three "different AZ" attempts all hit the same subnet, which is why the PERSISTENT_2 is not available in this availability zone error was byte-identical each time. On an account whose subnets[0] AZ doesn't offer PERSISTENT_2, every --az failed identically — a per-AZ-availability illusion. Fixed in spawn v0.59.0 (resolves --az → co-located subnet).

Probe evidence (direct CreateFileSystem, 2 runs, all 6 us-east-1 AZs, deleted + leak-checked): AWS accepts PERSISTENT_2 in 5 of 6 AZs — including the 1a/1b/1c the report "failed" on. Only use1-az3 consistently rejected it.

What remains genuinely valid for this issue

A real, narrower residual: PERSISTENT_2 availability does vary by AZ (use1-az3 truly didn't offer it), and there is no read-only AWS API to discover that per-AZ — confirmed:

• FSx has no describe-*-offerings operation.
• CreateFileSystem has no DryRun.
• Pricing API locationType is only AWS Region / AWS Local Zone, never per-AZ.

So region-level discovery is cleanly buildable (Pricing API); per-AZ is not, short of a billable create-probe or a static map that goes stale. Recommendation stands: if built, scope to region-level and document the per-AZ limitation — but note the urgency dropped now that spawn#208 (the thing that actually bit the consumer) is fixed.

[scttfrdmn]

👀 For the filer — action: update to spawn v0.59.0 and re-run

Since you're watching this issue, the practical upshot:

The PERSISTENT_2 is not available in this availability zone errors on --az us-east-1a/1b/1c were a spawn bug, not AWS. --fsx-create applied your --az to the EC2 instance but not to the filesystem, which always fell back to the default VPC's first subnet — so all three attempts created the FSx in the same (wrong) subnet, hence the identical error each time.

Verified directly (probed CreateFileSystem across all 6 us-east-1 AZs, twice): AWS accepts PERSISTENT_2 in 5 of 6 — including the 1a/1b/1c you saw fail. Only us-east-1e (use1-az3) genuinely lacks it.

Fixed in spawn v0.59.0 — --fsx-create now resolves --az to a co-located subnet, so the filesystem lands in the AZ you request, in the same AZ as the instance (FSx Lustre is single-AZ; they must match or the mount can't work).

To retest:

# update spawn to >= 0.59.0, then:
spawn launch <name> --fsx-create --fsx-s3-bucket <bucket> --az us-east-1a ...

It should now provision in us-east-1a. No more trial-and-error AZ hunting — and if you ever do hit a genuinely unsupported AZ (like 1e here), just pick another; there's no read-only AWS API to look that up ahead of time (that's the narrower thing this issue still tracks).