Skip to content

Commit bb64ebe

Browse files
author
developerworks
committed
Add chaos/soak reliability test suite (006-7)
- Add chaos test suite with 11 fault-injection scenarios (child_panic_storm, child_block_forever, child_ignore_cancel, rapid_failure_10k, slow_event_subscriber, command_channel_full, ipc_connection_storm, socket_path_contention, relay_crash_loop, clock_step_backward, runtime_starvation_probe) - Add soak test framework with MetricsCollector (platform-conditional RSS), SteadyTrafficGenerator, SoakReport generator, and SoakRuntime orchestrator - Add ScenarioVerdict JSON judgement schema and schema validation tests - Add shared fixtures: FixtureChildSpawner, FixtureEventThrottle, FixtureIpcStress (+RateLimiter, ClientClassification), FixtureClockController, FixtureRuntimeProbe - Add Cargo.toml entries for chaos_suite and soak_suite test binaries - Add spec artifacts: plan.md, research.md, data-model.md, quickstart.md, contracts, tasks.md (35 tasks all completed) - Update .specify/feature.json to point to 006-7 - Register chaos/soak archive paths in 006-2 QualityGateOutcome foreign-key column
1 parent 680e0c0 commit bb64ebe

46 files changed

Lines changed: 2861 additions & 68 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.specify/feature.json‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
2-
"feature_directory": "specs/006-6-config-dynamic-children",
3-
"active_feature_directory": "specs/006-6-config-dynamic-children",
2+
"feature_directory": "specs/006-7-chaos-soak-reliability",
3+
"active_feature_directory": "specs/006-7-chaos-soak-reliability",
44
"feature_directories": [
55
"specs/005-1-failure-policy-reliability",
66
"specs/005-2-work-role-defaults",
@@ -9,6 +9,7 @@
99
"specs/006-3-lifecycle-shutdown-realism",
1010
"specs/006-4-restart-policy-production",
1111
"specs/006-5-typed-events-observability",
12-
"specs/006-6-config-dynamic-children"
12+
"specs/006-6-config-dynamic-children",
13+
"specs/006-7-chaos-soak-reliability"
1314
]
1415
}

‎Cargo.toml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -264,3 +264,11 @@ path = "src/dashboard/tests/dashboard_module_test.rs"
264264
[[test]]
265265
name = "supervisor_generation_fencing_test"
266266
path = "src/tests/supervisor_generation_fencing_test.rs"
267+
268+
[[test]]
269+
name = "chaos_suite"
270+
path = "tests/chaos_suite.rs"
271+
272+
[[test]]
273+
name = "soak_suite"
274+
path = "tests/soak_suite.rs"

‎specs/006-2-release-supply-chain-gates/spec.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@
1010

1111
本切片与 specs/006-8-product-bundle-runbooks/spec.md 分工如下: 本切片只约束发布工程能力与记录模板, 006-8 约束对外交付目录与值守文档载体. 闸口是否通过只写在发布流水与归档路径上, 006-8 正文不重复罗列工具命令行.
1212

13+
**混沌与浸泡归档引用**: `specs/006-7-chaos-soak-reliability` 切片的混沌套件 JSON 判决书与 SoakReport Markdown 归档路径登记在本切片 `QualityGateOutcome`(质量闸口结果) 的外链列中. 归档格式分别遵循 `specs/006-7-chaos-soak-reliability/contracts/chaos-scenario-verdict.md`(JSON 判决书 schema) 和 `specs/006-7-chaos-soak-reliability/contracts/soak-report-format.md`(SoakReport 格式契约). CI nightly 混沌套件输出位于 stdout(由 CI 日志捕获), 浸泡报告写入 `artifacts/validation/soak-<timestamp>.md`. 这两个归档路径必须随发布记录一同登记, 不得仅依靠个人笔记本留存.
14+
1315
## User Scenarios & Testing (用户场景和测试) _(mandatory (必填))_
1416

1517
### User Story 1 (用户故事一) - 版本可追溯 (Priority (优先级): P1)

‎specs/006-7-chaos-soak-reliability/contracts/chaos-scenario-verdict.md‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
# Contract(契约): ChaosScenario Verdict JSON Schema(混沌场景判决书 JSON 模式)
2+
3+
**Status(状态)**: Draft(草稿) | **Version(版本)**: 1.0.0
4+
**Applies to(适用范围)**: `tests/chaos/verdict.rs` 的序列化实现
5+
6+
## 1. Schema(模式)
7+
8+
```json
9+
{
10+
"$schema": "https://json-schema.org/draft/2020-12/schema",
11+
"title": "ChaosScenarioVerdict",
12+
"description": "Verdict for a single chaos scenario run",
13+
"type": "object",
14+
"required": [
15+
"scenario_id",
16+
"semver",
17+
"passed",
18+
"thresholds",
19+
"started_at_unix_nanos",
20+
"duration_ns"
21+
],
22+
"properties": {
23+
"scenario_id": {
24+
"type": "string",
25+
"pattern": "^[a-z][a-z0-9_]*$",
26+
"description": "Scenario identifier in snake_case"
27+
},
28+
"semver": {
29+
"type": "string",
30+
"pattern": "^\\d+\\.\\d+\\.\\d+$",
31+
"description": "Semantic version from CARGO_PKG_VERSION"
32+
},
33+
"passed": {
34+
"type": "boolean",
35+
"description": "Overall pass/fail for this scenario"
36+
},
37+
"thresholds": {
38+
"type": "object",
39+
"additionalProperties": {
40+
"$ref": "#/$defs/ThresholdResult"
41+
},
42+
"description": "Per-threshold measurement results"
43+
},
44+
"started_at_unix_nanos": {
45+
"type": "integer",
46+
"minimum": 0,
47+
"description": "Unix timestamp in nanoseconds when the scenario started"
48+
},
49+
"duration_ns": {
50+
"type": "integer",
51+
"minimum": 0,
52+
"description": "Duration of the scenario in nanoseconds"
53+
},
54+
"error": {
55+
"type": ["string", "null"],
56+
"description": "Error message if the scenario failed unexpectedly"
57+
}
58+
},
59+
"$defs": {
60+
"ThresholdResult": {
61+
"type": "object",
62+
"required": ["value", "limit", "passed"],
63+
"properties": {
64+
"value": {
65+
"type": "number",
66+
"description": "Actual measured value"
67+
},
68+
"limit": {
69+
"type": "number",
70+
"description": "Threshold limit"
71+
},
72+
"passed": {
73+
"type": "boolean",
74+
"description": "Whether value <= limit (or meets other pass criteria)"
75+
}
76+
}
77+
}
78+
}
79+
}
80+
```
81+
82+
## 2. 序列化要求
83+
84+
- JSON 输出必须使用 `serde_json::to_string` 或等效工具序列化.
85+
- 顶层字段顺序不重要, 但 `scenario_id` 和 `semver` 应放在前两个字段便于人类阅读.
86+
- `error` 字段: 通过时为 `null`, 失败时填充可读错误信息.
87+
- `thresholds` 的 key 使用 snake_case, 与 ChaosScenario 表的 metric 名称一致.
88+
89+
## 3. 消费端契约
90+
91+
- CI nightly 脚本通过 `jq -e '.passed == true'` 逐条判定 JSON 判决书.
92+
- 任何一条判决书 `passed == false` 则 CI 任务失败.
93+
- `error` 字段不为 `null` 时视为未通过, 等价于 `passed == false`.

‎specs/006-7-chaos-soak-reliability/contracts/soak-report-format.md‎

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
# Contract(契约): SoakReport Markdown Format(浸泡报告 Markdown 格式)
2+
3+
**Status(状态)**: Draft(草稿) | **Version(版本)**: 1.0.0
4+
**Applies to(适用范围)**: `tests/soak/report.rs` 的格式化实现
5+
6+
## 1. 文件命名
7+
8+
```
9+
artifacts/validation/soak-{YYYYMMDD}-{HHMMSS}.md
10+
```
11+
12+
时间戳使用测试窗口结束时间的 UTC.
13+
14+
## 2. Markdown 结构
15+
16+
### 2.1 Metadata 段落
17+
18+
```markdown
19+
# SoakReport
20+
21+
## Metadata
22+
23+
- **Window**: {start_utc} - {end_utc}
24+
- **Commit**: {commit_hash}
25+
- **Hardware**: {hardware_description}
26+
```
27+
28+
- `start_utc` / `end_utc`: ISO 8601 格式, 如 `2026-05-19T00:00:00Z`.
29+
- `commit_hash`: `git rev-parse HEAD` 的全量 SHA.
30+
- `hardware_description`: 自由文本, 如 `macOS Apple Silicon, 16GB`.
31+
32+
### 2.2 Thresholds 表格
33+
34+
```markdown
35+
## Thresholds
36+
37+
| Metric | p99 | Avg | Max | Limit | Passed |
38+
| ---------------------- | ----- | ----- | ----- | ----- | ---------- |
39+
| p99_latency_ms | {val} | {val} | {val} | {val} | true/false |
40+
| rss_growth_mb_per_hour | {val} | {val} | {val} | {val} | true/false |
41+
| fd_count_drift | {val} | {val} | {val} | {val} | true/false |
42+
| event_gap_total | {val} | {val} | {val} | {val} | true/false |
43+
| shutdown_success_ratio | {val} | {val} | {val} | {val} | true/false |
44+
```
45+
46+
- 数值格式: 浮点数保留 2 位小数, 整数保留整数.
47+
- `Passed` 列: `true`(通过) 或 `false`(越界).
48+
49+
### 2.3 Violations 段落
50+
51+
```markdown
52+
## Violations
53+
54+
| Metric | Actual | Limit | Blocking | Exemption Ticket |
55+
| ------ | ------ | ----- | -------- | ---------------- |
56+
| {name} | {val} | {val} | yes/no | {ticket_id} or - |
57+
```
58+
59+
- 如果无越界, 写 `(none)`.
60+
- `Blocking` 标记条件: 连续 5 个采样窗口越阈.
61+
- 非 blocking 的越界必须挂豁免工单编号.
62+
63+
### 2.4 Exemptions 段落
64+
65+
```markdown
66+
## Exemptions
67+
68+
| Ticket ID | Metric | Reason | Expiry |
69+
| --------- | ------ | ------ | ------ |
70+
| {id} | {name} | {text} | {date} |
71+
```
72+
73+
- 如果没有豁免, 写 `(none)`.
74+
- `Expiry` 使用 ISO 8601 日期格式.
75+
76+
### 2.5 Attachments 段落
77+
78+
```markdown
79+
## Attachments
80+
81+
| File | SHA-256 |
82+
| --------------------- | ------- |
83+
| p99_latency_curve.png | {hash} |
84+
| rss_curve.png | {hash} |
85+
| fd_count_curve.png | {hash} |
86+
```
87+
88+
- 附件不存在时写 `(not generated)` 而不是跳过行.
89+
- 曲线 PNG 由 CI 后处理脚本(python/matplotlib)根据 CSV 数据生成, 不在 Rust 测试二进制中生成.
90+
91+
## 3. CI 集成
92+
93+
- SoakReport 作为 CI artifact 归档, 保留 90 天.
94+
- 归档路径: `artifacts/validation/soak-*`.
95+
- SHA-256 哈希写入 CI 日志, 供后续追溯.

0 commit comments

Comments
 (0)