Skip to content

Commit c7b11aa

Browse files
Pigbibiclaude
andcommitted
refactor(cloud): complete AWS fetch_id_token, document URI schemes, update README
- AwsDeploymentContext.fetch_id_token: implement EC2/ECS identity resolution with STS fallback for local dev. Added clear docs that AWS has no native equivalent to GCP audience-based ID tokens. - env_provider: document design rationale for reusing local ObjectStore/ DocumentStore (not a gap — by design for zero-dependency CI support) - ports.py: document URI scheme → provider mapping, cross-provider constraints - README / README.zh-CN: add AWS provider to supported list Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 2faa8b9 commit c7b11aa

5 files changed

Lines changed: 132 additions & 15 deletions

File tree

README.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,8 +40,9 @@ QuantPlatformKit includes a cloud provider abstraction layer at `quant_platform_
4040
| Provider | Env var | Description |
4141
|----------|---------|-------------|
4242
| **Google Cloud** (default) | `QSL_CLOUD_PROVIDER=gcp` | Uses GCP Secret Manager, Cloud Storage, Firestore — original behavior, no config change needed. |
43+
| **AWS** | `QSL_CLOUD_PROVIDER=aws` | Uses AWS Secrets Manager, S3, DynamoDB — requires boto3 and valid AWS credentials. |
4344
| **Local filesystem** | `QSL_CLOUD_PROVIDER=local` | Reads secrets and stores data under `~/.qsl/`. No cloud credentials required — ideal for development and testing. |
44-
| **Environment variables** | `QSL_CLOUD_PROVIDER=env` | Reads secrets from environment variables; otherwise uses local filesystem. Suitable for CI. |
45+
| **Environment variables** | `QSL_CLOUD_PROVIDER=env` | Reads secrets from environment variables; uses local filesystem for object/document storage. Suitable for CI. |
4546

4647
**Usage:**
4748

README.zh-CN.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ python -m pytest -q
4040
| Provider | 环境变量 | 说明 |
4141
|----------|---------|------|
4242
| **Google Cloud**(默认) | `QSL_CLOUD_PROVIDER=gcp` | 使用 GCP Secret Manager、Cloud Storage、Firestore—— 保持原有行为,无需修改配置。 |
43+
| **AWS** | `QSL_CLOUD_PROVIDER=aws` | 使用 AWS Secrets Manager、S3、DynamoDB—— 需要 boto3 和有效的 AWS 凭证。 |
4344
| **本地文件系统** | `QSL_CLOUD_PROVIDER=local` | 密钥和数据库存储在 `~/.qsl/` 目录下。无需任何云凭证——适合开发、测试和离线环境。 |
4445
| **环境变量** | `QSL_CLOUD_PROVIDER=env` | 密钥从环境变量读取;其余操作使用本地文件系统。适合 CI 场景。 |
4546

src/quant_platform_kit/cloud/aws_provider.py

Lines changed: 98 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -242,21 +242,65 @@ def resolve_instance_ip(
242242

243243

244244
class AwsDeploymentContext:
245-
"""AWS deployment context."""
245+
"""AWS deployment context.
246+
247+
通过 ECS metadata / EC2 IMDSv2 获取运行时身份信息。
248+
249+
注意:fetch_id_token 的语义与 GCP 不完全对等 ——
250+
AWS 没有 GCP ID Token 的标准替代品。
251+
此实现返回当前 EC2/ECS 实例的身份文档,
252+
或 STS GetCallerIdentity 作为 fallback。
253+
"""
246254

247255
@property
248256
def project_id(self) -> str:
249-
return os.environ.get("AWS_ACCOUNT_ID", "")
257+
# ECS task ARN 解析 → account_id
258+
task_arn = os.environ.get("ECS_CONTAINER_METADATA_URI_V4", "")
259+
if task_arn:
260+
parts = task_arn.split(":") if ":" in task_arn else []
261+
if len(parts) >= 5:
262+
return parts[4]
263+
return _resolve_aws_account_id()
250264

251265
@property
252266
def region(self) -> str | None:
253267
return _resolve_aws_region()
254268

255269
def fetch_id_token(self, audience: str) -> str:
256-
raise NotImplementedError(
257-
"AWS DeploymentContext.fetch_id_token is not implemented. "
258-
"Use a service-specific mechanism (e.g., Cognito, STS, or IAM roles)."
259-
)
270+
"""获取当前 AWS 环境的身份凭证。
271+
272+
在 ECS/EC2 上返回实例身份文档的 JSON string;
273+
本地开发环境 fallback 到 STS GetCallerIdentity。
274+
275+
``audience`` 参数在 AWS 无直接对应,此方法不会使用它。
276+
如果调用方依赖 audience 进行令牌验证,
277+
建议使用 Cognito 或自建 OIDC provider。
278+
"""
279+
import json as _json
280+
try:
281+
return _fetch_ecs_identity()
282+
except Exception:
283+
pass
284+
try:
285+
return _fetch_ec2_identity()
286+
except Exception:
287+
pass
288+
try:
289+
import boto3
290+
sts = boto3.client("sts", region_name=_resolve_aws_region())
291+
identity = sts.get_caller_identity()
292+
return _json.dumps({
293+
"account": identity.get("Account", ""),
294+
"arn": identity.get("Arn", ""),
295+
"user_id": identity.get("UserId", ""),
296+
})
297+
except Exception as exc:
298+
raise RuntimeError(
299+
"AwsDeploymentContext.fetch_id_token: unable to resolve AWS identity. "
300+
"Ensure the process runs on EC2, ECS, or has valid AWS credentials. "
301+
"Note: GCP-style audience-based ID tokens are not natively supported on AWS; "
302+
"consider using Cognito or a custom OIDC setup for audienced tokens."
303+
) from exc
260304

261305

262306
# ══════════════════════════════════════════════════════════════════════
@@ -279,6 +323,54 @@ def _resolve_aws_region() -> str:
279323
return "us-east-1"
280324

281325

326+
def _resolve_aws_account_id() -> str:
327+
"""Resolve AWS account ID from env vars, STS, or metadata."""
328+
env = os.environ.get("AWS_ACCOUNT_ID")
329+
if env:
330+
return env
331+
try:
332+
import boto3
333+
sts = boto3.client("sts", region_name=_resolve_aws_region())
334+
return sts.get_caller_identity().get("Account", "")
335+
except Exception:
336+
return ""
337+
338+
339+
def _fetch_ecs_identity() -> str:
340+
"""Fetch identity document from ECS container metadata endpoint (v4)."""
341+
import urllib.request
342+
metadata_uri = os.environ.get("ECS_CONTAINER_METADATA_URI_V4", "")
343+
if not metadata_uri:
344+
raise RuntimeError("Not running on ECS (ECS_CONTAINER_METADATA_URI_V4 not set)")
345+
req = urllib.request.Request(metadata_uri)
346+
with urllib.request.urlopen(req, timeout=3) as resp:
347+
return resp.read().decode("utf-8")
348+
349+
350+
def _fetch_ec2_identity() -> str:
351+
"""Fetch identity document from EC2 IMDSv2."""
352+
import urllib.request
353+
# Step 1: get IMDSv2 token
354+
token_req = urllib.request.Request(
355+
"http://169.254.169.254/latest/api/token",
356+
headers={"X-aws-ec2-metadata-token-ttl-seconds": "60"},
357+
method="PUT",
358+
)
359+
try:
360+
with urllib.request.urlopen(token_req, timeout=2) as resp:
361+
token = resp.read().decode("utf-8")
362+
except Exception:
363+
raise RuntimeError("Not running on EC2 (IMDSv2 unreachable)")
364+
365+
# Step 2: fetch identity document
366+
id_req = urllib.request.Request(
367+
"http://169.254.169.254/latest/dynamic/instance-identity/document",
368+
headers={"X-aws-ec2-metadata-token": token},
369+
)
370+
with urllib.request.urlopen(id_req, timeout=3) as resp:
371+
return resp.read().decode("utf-8")
372+
373+
282374
def _dynamodb_serialize(value):
283375
"""Convert Python values to DynamoDB-compatible format."""
284376
if isinstance(value, (str, bool, int, float, type(None))):

src/quant_platform_kit/cloud/env_provider.py

Lines changed: 14 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,20 @@
11
"""
22
Environment Variable provider — 从 os.environ 读取配置。
3-
适合简单部署场景或 CI 环境,无需云服务 SDK。
3+
4+
适合简单部署场景或 CI 环境,无需任何云服务 SDK。
45
56
用法:
67
QSL_CLOUD_PROVIDER=env
78
密钥通过环境变量注入(大写+下划线格式)
8-
对象存储通过本地文件系统(类似 local_provider)
9+
对象存储通过本地文件系统(与 local_provider 共享实现)
10+
11+
设计说明:
12+
- SecretStore: 从环境变量读取(优先 QSL_SECRET_<NAME>,其次 <NAME>)
13+
- SecretStoreReadWrite: 只读,写操作 noop(env vars 不可运行时写入)
14+
- ObjectStore / DocumentStore / ComputeDiscovery / DeploymentContext:
15+
直接复用 local_provider 实现。原因:env provider 的定位是"零依赖",
16+
不引入任何云 SDK。对象存储和文档数据库不适合通过环境变量操作,
17+
local 文件系统是开发者/CI 场景下最合理的后端。
918
"""
1019

1120
from __future__ import annotations
@@ -21,7 +30,8 @@
2130
LocalDeploymentContext,
2231
)
2332

24-
# Re-use local implementations for object store / doc store / compute
33+
# Re-use local implementations — by design, not a gap.
34+
# In CI and local dev, a real cloud bucket is unnecessary; a tmp dir suffices.
2535
ObjectStore = LocalObjectStore
2636
DocumentStore = LocalDocumentStore
2737
ComputeDiscovery = LocalComputeDiscovery
@@ -54,13 +64,12 @@ def get_secret(self, secret_name: str, *, project_id: str | None = None) -> str:
5464

5565

5666
class EnvSecretStoreReadWrite:
57-
"""只读 + 占位写操作(env var 不支持写入返回 noop)。"""
67+
"""只读 + 占位写操作(env var 不支持运行时写入)。"""
5868

5969
def get_secret(self, secret_name: str, *, project_id: str | None = None) -> str:
6070
return EnvSecretStore().get_secret(secret_name, project_id=project_id)
6171

6272
def create_secret(self, secret_name: str, payload: str, *, project_id: str | None = None) -> str:
63-
# env vars 不可写,仅记录日志
6473
import logging
6574
logging.getLogger(__name__).warning(
6675
"EnvSecretStore: create_secret is a no-op (env vars cannot be written at runtime). "

src/quant_platform_kit/cloud/ports.py

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -48,9 +48,23 @@ def destroy_latest_secret(self, secret_name: str, *, project_id: str | None = No
4848
@runtime_checkable
4949
class ObjectStore(Protocol):
5050
"""对象存储接口。URI 格式由实现方决定:
51-
- GCP: gs://bucket/key
52-
- AWS: s3://bucket/key
53-
- Local: file:///absolute/path 或 /absolute/path
51+
52+
GCP provider:
53+
gs://bucket-name/path/to/key — Cloud Storage
54+
55+
AWS provider:
56+
s3://bucket-name/path/to/key — S3
57+
58+
Local / Env provider:
59+
/absolute/path/to/file — 本地文件(绝对路径)
60+
file:///absolute/path/to/file — 同上
61+
gs://bucket/key — 映射到 ~/.qsl/storage/gs/bucket/key
62+
s3://bucket/key — 映射到 ~/.qsl/storage/s3/bucket/key
63+
64+
注意:URI scheme 与 provider 必须匹配:
65+
- ``s3://`` URI + AWS provider → S3 操作
66+
- ``s3://`` URI + Local provider → 本地文件模拟(便于开发)
67+
- ``gs://`` URI + AWS provider → ValueError(不跨云互通)
5468
"""
5569

5670
def read_text(self, uri: str) -> str:

0 commit comments

Comments
 (0)