From 3e9892e18a7e64f3cc9b88aaa538ec2c3a947c35 Mon Sep 17 00:00:00 2001 From: hermes agent Date: Tue, 28 Apr 2026 01:36:18 +0400 Subject: [PATCH 01/96] =?UTF-8?q?docs(backlog):=20detailed=20specs=20for?= =?UTF-8?q?=20section=2008=20=E2=80=94=20database=20&=20ops=20(TASK-306..3?= =?UTF-8?q?35)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 30 task specs covering migrations (indexes, new tables for slashes/contract code+storage+metadata/state snapshots/peers), DB observability (pool metrics, query histogram, slow-query log), read-replica routing, backup + restore tooling, migration CLI helpers, Redis cache warming + pub/sub bridge, SSE replica pinning, worker leader election, stuck-job recovery, dead-letter queue, and a job retry dashboard endpoint. First section of the spec-expansion plan; written before chain/vm/api sections so later specs can cite migration filenames directly. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/backlog/queue.md | 536 +++++++++++ docs/backlog/queue/08-database-ops.md | 1226 +++++++++++++++++++++++++ 2 files changed, 1762 insertions(+) create mode 100644 docs/backlog/queue.md create mode 100644 docs/backlog/queue/08-database-ops.md diff --git a/docs/backlog/queue.md b/docs/backlog/queue.md new file mode 100644 index 00000000..fae72990 --- /dev/null +++ b/docs/backlog/queue.md @@ -0,0 +1,536 @@ +# Tier-3 → 1000 Commit Backlog + +Source-of-truth task queue. Each unchecked item = one commit on this branch. +The paced-push script (`backend/scripts/paced-push.ts`) reads `data/push_pointer.txt` +and advances `origin/main` by `PUSH_RATE_PER_DAY / (1440/PUSH_INTERVAL_MIN)` commits per fire. + +**Target: 60 commits/day → ~8.2 days to clear.** + +## Chain & consensus + +- [ ] 1. Block.fromJSON deserializer +- [ ] 2. Wire /api/mesh/block to call chain.addBlock after fromJSON +- [ ] 3. Header-only sync endpoint /api/mesh/headers?from=&to= +- [ ] 4. Bulk block fetch /api/mesh/blocks?from=&to= with 100-block cap +- [ ] 5. Peer head poller (query each peer's /api/mesh/head every 30s) +- [ ] 6. Auto-sync on start: pick highest-height peer, pull missing +- [ ] 7. Reorg-on-sync: walk back to common ancestor +- [ ] 8. Finalized block flag at depth N-12 in chainState +- [ ] 9. Reject reorg attempts past finality depth (409) +- [ ] 10. VRF-style proposer rotation hash(prev_hash + height) mod n +- [ ] 11. validator_slashes table + read endpoint +- [ ] 12. Slash on equivocation (same height, different blocks) +- [ ] 13. validators.stake column + weighted producer selection +- [ ] 14. Quorum weight by stake instead of head count +- [ ] 15. Block timestamp drift check (>30s future = reject) +- [ ] 16. Min block time enforcement (<2s after parent = reject) +- [ ] 17. Difficulty retarget every 100 blocks +- [ ] 18. Persist mempool to disk on shutdown, restore on boot +- [ ] 19. Tx replacement-by-fee (same nonce, higher gasPrice) +- [ ] 20. Mempool size cap 10k with lowest-gasPrice eviction +- [ ] 21. Pending tx TTL 1h +- [ ] 22. Block size limit 1MB serialized +- [ ] 23. logs_topic0_idx index migration +- [ ] 24. /api/logs?fromBlock=&toBlock=&address=&topic0= +- [ ] 25. /api/logs/bloom-check helper endpoint +- [ ] 26. Block uncles tracking +- [ ] 27. GHOST fork-choice weighting +- [ ] 28. /api/chain/export?from=&to= NDJSON stream +- [ ] 29. backend/scripts/import-chain.ts +- [ ] 30. Genesis parameterization via genesis.json +- [ ] 31. Genesis hash verification at boot +- [ ] 32. Validator handoff record on rotation +- [ ] 33. Per-block VRF beacon for randomness +- [ ] 34. State pruning for zero/dead accounts +- [ ] 35. State snapshot every 10k blocks +- [ ] 36. /api/mesh/snapshot/:height +- [ ] 37. Tx fee distribution: 80% producer, 20% burned +- [ ] 38. Burn counter on chain stats +- [ ] 39. Per-validator block reward via env +- [ ] 40. Coinbase tx representation in receipts +- [ ] 41. Migration 0002: block_hash index +- [ ] 42. Migration: tx (from_address, nonce) compound index +- [ ] 43. Account-history rebuild script +- [ ] 44. CLI: npm run verify-chain +- [ ] 45. State root mismatch alarm event +- [ ] 46. Receipt root verification at sync +- [ ] 47. SSE /api/logs/subscribe?topic0= +- [ ] 48. SSE /api/mempool/subscribe +- [ ] 49. SSE /api/forks/subscribe +- [ ] 50. Per-block aggregate gas price stats +- [ ] 51. /api/chain/tps?window=60 +- [ ] 52. /api/chain/block-times histogram +- [ ] 53. Validator uptime metric +- [ ] 54. Mempool depth chart endpoint +- [ ] 55. /api/tx/simulate (VM dry-run) +- [ ] 56. /api/tx/estimate-gas +- [ ] 57. /api/account/:addr/next-nonce +- [ ] 58. /api/account/:addr/history paginated +- [ ] 59. /api/validator/:addr/blocks paginated +- [ ] 60. /api/chain/reorgs (last 50) + +## VM expansion + +- [ ] 61. MUL opcode +- [ ] 62. DIV opcode +- [ ] 63. MOD opcode +- [ ] 64. EQ / LT / GT comparison ops +- [ ] 65. AND / OR / NOT bitwise ops +- [ ] 66. JUMP / JUMPI control flow +- [ ] 67. JUMPDEST validation pass +- [ ] 68. SLOAD opcode (read storage) +- [ ] 69. Storage persistence to contract_storage table +- [ ] 70. CALL opcode +- [ ] 71. RETURN opcode + return-data buffer +- [ ] 72. CALLDATA opcode +- [ ] 73. CALLER / ORIGIN ops +- [ ] 74. VALUE op (msg.value) +- [ ] 75. BALANCE(addr) op +- [ ] 76. BLOCKNUMBER / TIMESTAMP / DIFFICULTY ops +- [ ] 77. SHA256 / KECCAK precompile +- [ ] 78. ECRECOVER precompile +- [ ] 79. CREATE opcode (contract deployment) +- [ ] 80. Contract address derivation keccak(sender+nonce) +- [ ] 81. contract_code table + storage +- [ ] 82. Code-loaded execution path in BlockProducer +- [ ] 83. /api/tx/:hash/trace endpoint +- [ ] 84. /api/tx/:hash/gas-profile per-op breakdown +- [ ] 85. STATICCALL (read-only nested) +- [ ] 86. SELFDESTRUCT op +- [ ] 87. SSTORE refund for zero-set +- [ ] 88. Cold/warm SLOAD pricing +- [ ] 89. Memory model byte-addressable scratch +- [ ] 90. MLOAD / MSTORE / MSTORE8 +- [ ] 91. Memory expansion gas +- [ ] 92. Stack depth limit 1024 +- [ ] 93. Call depth limit 1024 +- [ ] 94. REVERT with return data +- [ ] 95. Try-catch CALL semantics +- [ ] 96. EVENT opcode separate from LOG +- [ ] 97. Event ABI registry per contract +- [ ] 98. /api/contract/:addr/source verifier +- [ ] 99. /api/contract/:addr/disasm endpoint +- [ ] 100. /api/contract/:addr/storage browser +- [ ] 101. contract_metadata table +- [ ] 102. VM unit test fixtures (30 sample programs) +- [ ] 103. /docs/vm spec +- [ ] 104. Compiler stub: tiny DSL → JSON-op +- [ ] 105. Sample contracts: counter / erc20-like / multisig / vrf + +## Wallet & accounts + +- [ ] 106. HD wallet derivation BIP32-style +- [ ] 107. Mnemonic export endpoint +- [ ] 108. Mnemonic import + recovery +- [ ] 109. Watch-only address mode +- [ ] 110. Multi-sig wallet primitive +- [ ] 111. Wallet name aliases +- [ ] 112. ENS-like /api/names/:name resolver +- [ ] 113. Reverse name lookup +- [ ] 114. Wallet activity feed (sends + receives + events) +- [ ] 115. CSV export of wallet history +- [ ] 116. /api/wallet/:addr/qr.png +- [ ] 117. Wallet contact book +- [ ] 118. Token balance aggregation across deployed tokens +- [ ] 119. Approve / transferFrom flow for tokens +- [ ] 120. Allowance lookup endpoint +- [ ] 121. Token transfer history per-account +- [ ] 122. Faucet rate-limit by IP not just address +- [ ] 123. Faucet captcha hook +- [ ] 124. Faucet drip dynamic amount +- [ ] 125. Faucet pool refill schedule +- [ ] 126. Wallet send batch +- [ ] 127. Tx scheduling (broadcast at future height) +- [ ] 128. Tx replacement UI flow (cancel-by-replace) +- [ ] 129. Hardware-key signing protocol stub +- [ ] 130. Session key delegation +- [ ] 131. Account abstraction stub: paymaster +- [ ] 132. Wallet-side mempool view +- [ ] 133. Wallet recovery via social guardians +- [ ] 134. Per-account gas budget cap +- [ ] 135. Wallet password-encrypted export PBKDF2+AES +- [ ] 136. Wallet import from JSON +- [ ] 137. Address validity checker endpoint +- [ ] 138. Vanity address generator script +- [ ] 139. Bulk address generator for testing +- [ ] 140. Wallet metrics endpoint + +## API & explorer + +- [ ] 141. /api/openapi.json generation +- [ ] 142. Swagger UI at /docs +- [ ] 143. /api/v1/* version prefix + deprecation headers +- [ ] 144. Rate-limit headers +- [ ] 145. CORS allowlist via env +- [ ] 146. Request-ID middleware +- [ ] 147. Structured access log NDJSON +- [ ] 148. Slow-request log >1s +- [ ] 149. /health/live + /health/ready + /health/deep +- [ ] 150. /api/build (commit sha + build time) +- [ ] 151. /api/flags feature-flag endpoint +- [ ] 152. /api/metrics Prometheus text +- [ ] 153. Block search by height range with filters +- [ ] 154. Tx search by from/to/value range +- [ ] 155. Top accounts by balance +- [ ] 156. Top accounts by tx count +- [ ] 157. Validator leaderboard +- [ ] 158. /api/network/stats dashboard endpoint +- [ ] 159. Block detail with full receipts inline +- [ ] 160. Tx detail with decoded log events +- [ ] 161. /api/contract/:addr/events feed +- [ ] 162. Address tag system +- [ ] 163. Tag suggestion endpoint +- [ ] 164. Top gas spenders last 24h +- [ ] 165. /api/reorg/:id detail page +- [ ] 166. /api/mempool?limit=200 snapshot +- [ ] 167. /api/mempool/:hash pending tx by hash +- [ ] 168. Cancel pending tx endpoint +- [ ] 169. Bulk tx submit endpoint +- [ ] 170. Idempotent tx submit (dedup on hash) +- [ ] 171. WebSocket equivalents of all SSE channels +- [ ] 172. Socket.io rooms per address +- [ ] 173. SSE event replay since cursor +- [ ] 174. GraphQL gateway over REST +- [ ] 175. tRPC endpoint mirror +- [ ] 176. JSON-RPC eth_blockNumber + eth_getBalance +- [ ] 177. JSON-RPC eth_call via VM +- [ ] 178. JSON-RPC eth_sendRawTransaction +- [ ] 179. JSON-RPC subscriptions +- [ ] 180. Postman/Bruno collection generator + +## Agent worker + +- [ ] 181. Task priority queue (urgent / normal / chore) +- [ ] 182. Task dependency graph +- [ ] 183. Task estimated + actual effort tracking +- [ ] 184. Task retry on transient failure +- [ ] 185. Task timeout 30min +- [ ] 186. Per-task token budget +- [ ] 187. Cumulative daily token budget +- [ ] 188. /api/agent/tokens/stream SSE +- [ ] 189. Cost estimate per task +- [ ] 190. Task rejection reason logging +- [ ] 191. Skill registry persistence +- [ ] 192. Skill versioning + hot-reload +- [ ] 193. Per-skill rate limit +- [ ] 194. /api/skills discovery +- [ ] 195. Skill source viewer +- [ ] 196. /api/agent/tools?since= log dump +- [ ] 197. Verification: typecheck after every code edit +- [ ] 198. Verification: tests on changed packages +- [ ] 199. Verification: prettier +- [ ] 200. Auto-rollback on verification failure +- [ ] 201. PR-mode toggle +- [ ] 202. PR template generator from task +- [ ] 203. PR auto-link to task ID +- [ ] 204. Squash-merge agent for stacked commits +- [ ] 205. Branch-per-task workflow +- [ ] 206. Auto-rebase on main before push +- [ ] 207. Conflict-resolution prompt to LLM +- [ ] 208. Commitlint enforcement +- [ ] 209. Co-author tag on every agent commit +- [ ] 210. Sign-off (DCO) tag +- [ ] 211. Agent identity rotation per file area +- [ ] 212. Per-area expertise routing +- [ ] 213. Cross-task memory +- [ ] 214. Per-task post-mortem 1-line +- [ ] 215. Learning corpus index + +## Frontend / HUD + +- [ ] 216. Tx detail modal +- [ ] 217. Block detail drawer +- [ ] 218. Account detail panel +- [ ] 219. Mempool live-tail panel +- [ ] 220. Reorg history widget +- [ ] 221. Validator board with rotating producer indicator +- [ ] 222. Quorum-vote live ticker +- [ ] 223. VM execution trace viewer +- [ ] 224. Contract storage browser +- [ ] 225. Event log viewer with topic filter +- [ ] 226. Gas-price chart last 1000 blocks +- [ ] 227. Block-time chart +- [ ] 228. TPS sparkline in header +- [ ] 229. Network peer map (graph view) +- [ ] 230. Peer latency table +- [ ] 231. Wallet panel: HD address tree +- [ ] 232. Send-tx form with gas estimator +- [ ] 233. Token balance grid +- [ ] 234. Approve / transfer modal +- [ ] 235. Faucet button + cooldown timer +- [ ] 236. Address book panel +- [ ] 237. Name registry browse +- [ ] 238. Theme: high-contrast variant +- [ ] 239. Theme: amber CRT variant +- [ ] 240. Theme picker dropdown +- [ ] 241. Compact mode +- [ ] 242. Stage agent reasoning panel +- [ ] 243. Per-task progress bar in HUD +- [ ] 244. Cost ticker (today's spend) +- [ ] 245. SSE reconnect with exponential backoff +- [ ] 246. SSE channel selector +- [ ] 247. Keyboard shortcuts overlay (?) +- [ ] 248. Command palette cmd-k +- [ ] 249. Block search bar +- [ ] 250. Address search with autocomplete +- [ ] 251. Tx hash paste auto-route +- [ ] 252. URL deep-link state restore +- [ ] 253. Shareable view URLs preserve filters +- [ ] 254. Local hud preferences in localStorage +- [ ] 255. Server-synced HUD prefs per API key +- [ ] 256. Toast notifications for own-wallet activity +- [ ] 257. Browser push notification opt-in +- [ ] 258. Embed-mode iframe +- [ ] 259. Mobile-responsive collapse +- [ ] 260. PWA manifest + offline shell +- [ ] 261. Service worker for static assets +- [ ] 262. Reusable chart component +- [ ] 263. A11y pass: aria-labels everywhere +- [ ] 264. Keyboard-trap audit on modals +- [ ] 265. Focus-visible styling pass + +## Docs & site + +- [ ] 266. Landing page tagline rewrite +- [ ] 267. /docs/architecture rewrite +- [ ] 268. /docs/consensus page +- [ ] 269. /docs/vm spec page +- [ ] 270. /docs/api reference +- [ ] 271. /docs/sdk page +- [ ] 272. SDK quickstart code samples (TS + Python) +- [ ] 273. SDK npm package skeleton +- [ ] 274. SDK chain client class +- [ ] 275. SDK wallet helper +- [ ] 276. SDK VM helper for op programs +- [ ] 277. SDK signed-tx builder +- [ ] 278. Python SDK skeleton +- [ ] 279. Rust SDK skeleton +- [ ] 280. CLI tool `hermes` npm bin +- [ ] 281. CLI: hermes balance +- [ ] 282. CLI: hermes send +- [ ] 283. CLI: hermes call +- [ ] 284. CLI: hermes deploy +- [ ] 285. CLI: hermes node start (local node) +- [ ] 286. /examples/counter walkthrough +- [ ] 287. /examples/erc20-like walkthrough +- [ ] 288. /examples/multisig walkthrough +- [ ] 289. /examples/oracle walkthrough +- [ ] 290. Tutorial: build your first contract +- [ ] 291. Tutorial: run your own validator +- [ ] 292. Tutorial: query the chain +- [ ] 293. Tutorial: submit a tx from a script +- [ ] 294. Whitepaper draft v0.1 +- [ ] 295. Glossary page +- [ ] 296. FAQ page +- [ ] 297. Roadmap page +- [ ] 298. Contributing guide +- [ ] 299. Style guide for agent-authored code +- [ ] 300. Code-of-conduct +- [ ] 301. Threat model document +- [ ] 302. Disclosure / security.txt +- [ ] 303. Bug bounty page +- [ ] 304. Status page stub +- [ ] 305. Press kit (logos, screenshots) + +## Database & ops + +- [ ] 306. Migration: index transactions(block_height) +- [ ] 307. Migration: index transactions(from_address, nonce) +- [ ] 308. Migration: index accounts(balance DESC) +- [ ] 309. Migration: index receipts(status) +- [ ] 310. Migration: GIN index on receipts.logs_json +- [ ] 311. Migration: validators.stake numeric column +- [ ] 312. Migration: validator_slashes table +- [ ] 313. Migration: contract_code table +- [ ] 314. Migration: contract_storage table +- [ ] 315. Migration: contract_metadata table +- [ ] 316. Migration: state_snapshots table +- [ ] 317. Migration: peers table mirror of peers.json +- [ ] 318. Migration: rename chat_logs → agent_chat_logs +- [ ] 319. DB connection-pool tuning + monitoring +- [ ] 320. DB query-time histogram metric +- [ ] 321. DB slow-query log +- [ ] 322. Read-replica routing for heavy reads +- [ ] 323. pg_dump backup script to S3 +- [ ] 324. Restore script + smoke test +- [ ] 325. CLI: npm run migrate:down NNNN +- [ ] 326. Migration dry-run mode +- [ ] 327. Schema-diff against prod CLI +- [ ] 328. Redis cache warmer at boot +- [ ] 329. Redis key TTL audit +- [ ] 330. Redis pub/sub channel for cross-replica events +- [ ] 331. Two-replica web service: pin SSE to one +- [ ] 332. Worker leader election +- [ ] 333. Stuck-job recovery +- [ ] 334. Dead-letter queue for failed agent tasks +- [ ] 335. Job retry dashboard + +## Security + +- [ ] 336. CSRF protection on POST endpoints +- [ ] 337. Helmet middleware (CSP, HSTS, frameguard) +- [ ] 338. SQL-injection audit pass +- [ ] 339. Input length caps on every endpoint +- [ ] 340. JSON body size limit +- [ ] 341. Per-endpoint rate limit overrides +- [ ] 342. API-key scope chain:write vs chain:read +- [ ] 343. API-key expiry default 90d +- [ ] 344. API-key rotation endpoint +- [ ] 345. API-key audit log +- [ ] 346. Admin-token rotation flow +- [ ] 347. Failed-auth lockout (5 in 1min → 15min) +- [ ] 348. Suspicious-activity feed +- [ ] 349. Tx replay protection: chainId +- [ ] 350. Tx replay protection: per-key nonce window +- [ ] 351. Wallet-export rate limit 1/min +- [ ] 352. Mnemonic display obscured + reveal button +- [ ] 353. Server log redaction (no keys/sigs/mnemonics) +- [ ] 354. Secrets scan in CI (gitleaks) +- [ ] 355. Dependency audit on PR (npm audit) +- [ ] 356. CodeQL workflow on push +- [ ] 357. Snyk integration +- [ ] 358. CSP nonce per request +- [ ] 359. Subresource integrity on CDN +- [ ] 360. HTTPS-only redirect middleware +- [ ] 361. SameSite=strict cookies +- [ ] 362. Session fixation defense +- [ ] 363. Password-strength meter +- [ ] 364. Encryption at rest for stored mnemonics +- [ ] 365. KMS integration stub +- [ ] 366. Tor / VPN flag (informational) +- [ ] 367. ip2geo on auth log +- [ ] 368. Threat-feed integration +- [ ] 369. Cert-pinning notes for mobile +- [ ] 370. Disclosure response template + +## Testing + +- [ ] 371. Unit: ValidatorManager rotation +- [ ] 372. Unit: ValidatorManager quorum thresholds +- [ ] 373. Unit: Interpreter happy path +- [ ] 374. Unit: Interpreter out-of-gas +- [ ] 375. Unit: Interpreter REVERT +- [ ] 376. Unit: GasMeter charging +- [ ] 377. Unit: PeerRegistry stale eviction +- [ ] 378. Unit: parseVmProgram edge cases +- [ ] 379. Unit: TransactionPool nonce window +- [ ] 380. Unit: TransactionPool replacement-by-fee +- [ ] 381. Unit: StateManager.applyTransaction +- [ ] 382. Unit: StateManager.revertBlock +- [ ] 383. Unit: createReceipt + bloom +- [ ] 384. Unit: Chain.handleReorg +- [ ] 385. Unit: ForkManager fork-choice +- [ ] 386. Unit: Block.isValid +- [ ] 387. Unit: signature verify ed25519 +- [ ] 388. Unit: faucet cooldown +- [ ] 389. Unit: api-key permission gating +- [ ] 390. Unit: migration runner ordering +- [ ] 391. Integration: boot + apply migration + insert + read +- [ ] 392. Integration: produce 10 blocks, verify receipts persist +- [ ] 393. Integration: reorg verify state matches canonical +- [ ] 394. Integration: VM tx end-to-end via /api/transactions +- [ ] 395. Integration: peer announce + list +- [ ] 396. Integration: faucet → send → balance +- [ ] 397. Integration: api-key creation gated +- [ ] 398. Integration: SSE stream emits all event types +- [ ] 399. Integration: socket.io rooms +- [ ] 400. Load: 1000 tx/sec submission +- [ ] 401. Load: 100 concurrent SSE clients +- [ ] 402. Load: peer-mesh announce flood +- [ ] 403. Fuzz: VM with random op sequences +- [ ] 404. Fuzz: tx body fields +- [ ] 405. Fuzz: API JSON inputs +- [ ] 406. Property: state root invariance under reorder-then-reorg +- [ ] 407. Property: gas accounting never exceeds limit +- [ ] 408. Snapshot: API response shapes +- [ ] 409. Snapshot: HUD components +- [ ] 410. CI workflow runs all of the above on PR + +## DX & tooling + +- [ ] 411. ESLint config tightening +- [ ] 412. Prettier config +- [ ] 413. Husky pre-commit +- [ ] 414. lint-staged setup +- [ ] 415. Commitlint conventional-commits +- [ ] 416. Renovate bot config +- [ ] 417. Dependabot config +- [ ] 418. tsconfig strict mode on +- [ ] 419. tsconfig noUncheckedIndexedAccess +- [ ] 420. Path aliases (@chain, @api) +- [ ] 421. Build-time env validation (zod) +- [ ] 422. dotenv example file refresh +- [ ] 423. Docker Compose backend+postgres+redis +- [ ] 424. Devcontainer config +- [ ] 425. Makefile shortcuts +- [ ] 426. justfile alternative +- [ ] 427. nvmrc file +- [ ] 428. Volta pin +- [ ] 429. README badges +- [ ] 430. Architecture Mermaid diagram +- [ ] 431. Block production sequence diagram +- [ ] 432. Reorg sequence diagram +- [ ] 433. Agent task sequence diagram +- [ ] 434. Logo refresh +- [ ] 435. Favicon refresh +- [ ] 436. og:image generator +- [ ] 437. Sitemap.xml +- [ ] 438. robots.txt +- [ ] 439. Lighthouse score ≥ 90 on landing +- [ ] 440. Bundle-size budget enforcement +- [ ] 441. Tree-shake unused exports +- [ ] 442. Dynamic imports for HUD heavy panels +- [ ] 443. Frontend Sentry integration +- [ ] 444. Backend Sentry integration +- [ ] 445. Logflare / Axiom integration + +## Ecosystem stubs + +- [ ] 446. Block explorer minimal v2 +- [ ] 447. Validator dashboard standalone +- [ ] 448. Chain stats embedded widget +- [ ] 449. Telegram bot /balance + /tx +- [ ] 450. Discord bot equivalent +- [ ] 451. Slack notifier for own-wallet +- [ ] 452. Email notifier (SES) for own-wallet +- [ ] 453. Twilio SMS notifier +- [ ] 454. /api/webhooks subscription +- [ ] 455. Zapier connector spec +- [ ] 456. n8n node +- [ ] 457. Grafana dashboard JSON +- [ ] 458. Datadog monitor JSON +- [ ] 459. Prometheus alert rules +- [ ] 460. PagerDuty service mapping notes +- [ ] 461. Runbook: db unreachable +- [ ] 462. Runbook: agent stuck +- [ ] 463. Runbook: chain halted +- [ ] 464. Runbook: peer mesh partitioned +- [ ] 465. Runbook: out-of-disk +- [ ] 466. Chaos test: kill worker mid-block +- [ ] 467. Chaos test: drop db connection +- [ ] 468. Chaos test: clock skew +- [ ] 469. Disaster-recovery dry-run +- [ ] 470. Multi-region deploy notes +- [ ] 471. CDN caching headers tuning +- [ ] 472. Static-asset versioning +- [ ] 473. SDK npm publish workflow +- [ ] 474. CLI npm publish workflow +- [ ] 475. Brew tap formula + +## Final polish + +- [ ] 476. CHANGELOG file (auto from conventional commits) +- [ ] 477. v0.3 release tag +- [ ] 478. Release notes blog post +- [ ] 479. Demo screencast +- [ ] 480. Onboarding video +- [ ] 481. Public roadmap board +- [ ] 482. Issue templates (bug/feature/security) +- [ ] 483. PR template +- [ ] 484. Discussion categories on GitHub +- [ ] 485. RSS feed for blog/changelog +- [ ] 486. Newsletter signup endpoint +- [ ] 487. Discord invite on landing +- [ ] 488. Twitter card meta refresh +- [ ] 489. Social proof: live commit counter +- [ ] 490. Tagline rotation A/B test diff --git a/docs/backlog/queue/08-database-ops.md b/docs/backlog/queue/08-database-ops.md new file mode 100644 index 00000000..9f05e2b5 --- /dev/null +++ b/docs/backlog/queue/08-database-ops.md @@ -0,0 +1,1226 @@ +# Section 08 — Database & Ops Specs (TASK-306..335) + +30 tasks. Migrations, query observability, replica/cache infrastructure, backup/restore, job-queue hardening. Authored first because later sections cite these migration filenames and helper functions by name. + +**Preconditions used throughout this section:** +- Migration runner contract at [backend/src/database/migrations.ts:25-60](backend/src/database/migrations.ts#L25-L60). Files named `NNNN_slug.sql`, split on literal `-- down:` line, `-- up:` half is what runs in production. +- Existing migrations: [backend/src/database/migrations/0001_receipts.sql](backend/src/database/migrations/0001_receipts.sql). +- Schema source: [backend/src/database/schema.ts](backend/src/database/schema.ts) — every CREATE TABLE has `IF NOT EXISTS`. New tables go in NEW migration files, not edits to schema.ts. +- DB wrapper: [backend/src/database/db.ts:81-94](backend/src/database/db.ts#L81-L94) exposes `db.query(sql, params) → {rows, rowCount}` and `db.exec(sql)` for multi-statement. +- Redis wrapper: [backend/src/database/db.ts:123-222](backend/src/database/db.ts#L123-L222) exposes `cache.get/set/getJSON/setJSON/incr/hget/hset/hgetall`. + +--- + +### TASK-306 — Migration 0002: index transactions(block_height) + +**Section:** db +**Effort:** S +**Depends on:** none +**Type:** migration + +**Goal** +The `transactions` table from [schema.ts:24-56](backend/src/database/schema.ts#L24-L56) has no index on `block_height`. Block-detail endpoints currently scan the full table to load a block's txs. Add a btree index so block lookups stay O(log n) as the chain grows. + +**Files** +- new: `backend/src/database/migrations/0002_tx_block_height_idx.sql` + +**Reuses** +- Pattern from [backend/src/database/migrations/0001_receipts.sql:17-19](backend/src/database/migrations/0001_receipts.sql#L17-L19) — `CREATE INDEX IF NOT EXISTS ... ON ... (...);` + +**Migration SQL** +```sql +-- up: +CREATE INDEX IF NOT EXISTS idx_transactions_block_height + ON transactions(block_height); + +-- down: +DROP INDEX IF EXISTS idx_transactions_block_height; +``` + +**Implementation sketch** +- Drop file in `backend/src/database/migrations/` with the exact name above. +- Verify lexical sort places it after 0001. +- No code changes; runner picks it up on next boot via `applyPendingMigrations()`. + +**Acceptance** +- [ ] `backend/src/database/migrations/0002_tx_block_height_idx.sql` exists with both `-- up:` and `-- down:` blocks. +- [ ] Boot logs `[MIGRATIONS] 0002_tx_block_height_idx applied in ms`. +- [ ] `SELECT * FROM pg_indexes WHERE indexname='idx_transactions_block_height'` returns one row. + +**Verification** +- Cold boot against an empty PG: `npm run dev` (backend), inspect logs for migration line. +- Hot boot (migration already applied): boot still succeeds, logs `All N migration(s) already applied`. +- `EXPLAIN ANALYZE SELECT * FROM transactions WHERE block_height = 100;` shows `Index Scan` not `Seq Scan`. + +--- + +### TASK-307 — Migration 0003: compound index transactions(from_address, nonce) + +**Section:** db +**Effort:** S +**Depends on:** none +**Type:** migration + +**Goal** +Mempool admission and `/api/account/:addr/next-nonce` (TASK-057) both compute "max nonce seen from this sender." Without a compound index, that's a full table scan per call. Add `(from_address, nonce DESC)` to make it an index-only lookup. + +**Files** +- new: `backend/src/database/migrations/0003_tx_from_nonce_idx.sql` + +**Reuses** +- Same `CREATE INDEX IF NOT EXISTS` pattern from 0001. + +**Migration SQL** +```sql +-- up: +CREATE INDEX IF NOT EXISTS idx_transactions_from_nonce + ON transactions(from_address, nonce DESC); + +-- down: +DROP INDEX IF EXISTS idx_transactions_from_nonce; +``` + +**Implementation sketch** +- Drop the SQL file. +- No code changes. + +**Acceptance** +- [ ] Index visible via `\d transactions` (psql) under `Indexes:`. +- [ ] `EXPLAIN SELECT MAX(nonce) FROM transactions WHERE from_address = '0xabc'` shows `Index Scan using idx_transactions_from_nonce`. + +**Verification** +- Cold boot, scan logs for migration line. + +--- + +### TASK-308 — Migration 0004: index accounts(balance DESC) for top-balances + +**Section:** db +**Effort:** S +**Depends on:** none +**Type:** migration + +**Goal** +The "Top accounts by balance" endpoint (TASK-155) ranks every account by balance. Without an index, that's a full sort over the accounts table on every call. Add a descending btree so the top-N is an index-range scan. + +**Files** +- new: `backend/src/database/migrations/0004_accounts_balance_idx.sql` + +**Reuses** +- `CREATE INDEX` pattern from 0001. + +**Migration SQL** +```sql +-- up: +CREATE INDEX IF NOT EXISTS idx_accounts_balance_desc + ON accounts(balance DESC); + +-- down: +DROP INDEX IF EXISTS idx_accounts_balance_desc; +``` + +**Implementation sketch** +- The `accounts` table from [schema.ts:58-66](backend/src/database/schema.ts#L58-L66) stores balance as TEXT (decimal-as-string). PostgreSQL btree sorts TEXT lexicographically — incorrect for numeric ordering. Two options: + 1. **Recommended:** add an additional column `balance_numeric NUMERIC` populated by trigger or ALTER, index that. + 2. Cast index: `CREATE INDEX ... ON accounts ((balance::numeric) DESC)` — works on PG. +- Pick option 2 (zero schema change). Document the cast in spec. + +**Migration SQL (revised, option 2)** +```sql +-- up: +CREATE INDEX IF NOT EXISTS idx_accounts_balance_desc + ON accounts ((balance::numeric) DESC); + +-- down: +DROP INDEX IF EXISTS idx_accounts_balance_desc; +``` + +**Acceptance** +- [ ] Index exists. +- [ ] `EXPLAIN SELECT address, balance FROM accounts ORDER BY balance::numeric DESC LIMIT 100;` uses the index. + +**Verification** +- After a few thousand tx, top-100 query returns in <50ms. + +--- + +### TASK-309 — Migration 0005: index receipts(status) for failure-rate queries + +**Section:** db +**Effort:** S +**Depends on:** TASK-306 implicitly (migrations are sequential) +**Type:** migration + +**Goal** +The `receipts` table from 0001 has indexes on block_number / from / to but not on `status`. "Failure rate over last N blocks" and "latest reverts" queries scan the full table. Add an index. + +**Files** +- new: `backend/src/database/migrations/0005_receipts_status_idx.sql` + +**Migration SQL** +```sql +-- up: +CREATE INDEX IF NOT EXISTS idx_receipts_status + ON receipts(status, block_number DESC); + +-- down: +DROP INDEX IF EXISTS idx_receipts_status; +``` + +**Implementation sketch** +- Compound `(status, block_number DESC)` so "latest 100 failed" = single range scan. +- No code changes. + +**Acceptance** +- [ ] Index applied. +- [ ] `EXPLAIN SELECT * FROM receipts WHERE status != 1 ORDER BY block_number DESC LIMIT 100;` uses it. + +**Verification** +- Inspect `pg_indexes` after boot. + +--- + +### TASK-310 — Migration 0006: GIN index on receipts.logs_json + +**Section:** db +**Effort:** S +**Depends on:** none +**Type:** migration + +**Goal** +The `/api/logs` endpoint (TASK-024) filters by topic / address inside the `logs_json` text column. Without a GIN index over the parsed JSON, every log query is a full receipts scan. Convert query path to JSONB and add a GIN index for membership queries. + +**Files** +- new: `backend/src/database/migrations/0006_receipts_logs_gin.sql` + +**Migration SQL** +```sql +-- up: +ALTER TABLE receipts + ADD COLUMN IF NOT EXISTS logs_jsonb JSONB + GENERATED ALWAYS AS (logs_json::jsonb) STORED; +CREATE INDEX IF NOT EXISTS idx_receipts_logs_gin + ON receipts USING GIN (logs_jsonb); + +-- down: +DROP INDEX IF EXISTS idx_receipts_logs_gin; +ALTER TABLE receipts DROP COLUMN IF EXISTS logs_jsonb; +``` + +**Implementation sketch** +- Use a generated column so existing writes through `storeReceipt()` in [TransactionReceipt.ts:205-245](backend/src/blockchain/TransactionReceipt.ts#L205-L245) keep going to `logs_json` (text) and PG materializes `logs_jsonb` automatically. +- TASK-024 will query `logs_jsonb @> '[{"address":"0x..."}]'` style. + +**Acceptance** +- [ ] Generated column populated for all existing rows. +- [ ] GIN index visible in `\d receipts`. +- [ ] `EXPLAIN SELECT * FROM receipts WHERE logs_jsonb @> '[{"address":"0xabc"}]'` uses index. + +**Verification** +- Run migration on a DB with existing receipts, count `WHERE logs_jsonb IS NOT NULL` matches `COUNT(*)`. + +--- + +### TASK-311 — Migration 0007: validators.stake column for weighted quorum + +**Section:** db +**Effort:** S +**Depends on:** none +**Type:** migration + +**Goal** +TASK-013 (weighted producer selection) and TASK-014 (stake-weighted quorum) need a `stake` column on the `validators` table from [schema.ts:68-87](backend/src/database/schema.ts#L68-L87). Add it with a default of 1 so existing rows keep current head-count quorum behavior. + +**Files** +- new: `backend/src/database/migrations/0007_validators_stake.sql` + +**Migration SQL** +```sql +-- up: +ALTER TABLE validators + ADD COLUMN IF NOT EXISTS stake NUMERIC NOT NULL DEFAULT 1; + +-- down: +ALTER TABLE validators DROP COLUMN IF EXISTS stake; +``` + +**Implementation sketch** +- NUMERIC chosen for arbitrary-precision (matches existing balance/value text-as-numeric pattern). +- DEFAULT 1 means TASK-014's `Math.ceil(2 * sumStake / 3)` reduces to the current `Math.ceil(2n/3)` until stakes are explicitly set. + +**Acceptance** +- [ ] Column visible in `\d validators`. +- [ ] All existing rows have `stake = 1`. + +**Verification** +- `SELECT address, stake FROM validators` shows the new column. + +--- + +### TASK-312 — Migration 0008: validator_slashes table + +**Section:** db +**Effort:** S +**Depends on:** TASK-311 (validators.stake referenced) +**Type:** migration + +**Goal** +TASK-011 (slashing record) and TASK-012 (slash on equivocation) need a persistent table of slashing events. One row per offense with the offending validator, height, evidence pointer, and stake decrement applied. + +**Files** +- new: `backend/src/database/migrations/0008_validator_slashes.sql` + +**Migration SQL** +```sql +-- up: +CREATE TABLE IF NOT EXISTS validator_slashes ( + id BIGSERIAL PRIMARY KEY, + validator_address TEXT NOT NULL, + block_height BIGINT NOT NULL, + reason TEXT NOT NULL, + evidence_json TEXT NOT NULL DEFAULT '{}', + stake_before NUMERIC NOT NULL, + stake_after NUMERIC NOT NULL, + slashed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP +); +CREATE INDEX IF NOT EXISTS idx_slashes_validator + ON validator_slashes(validator_address); +CREATE INDEX IF NOT EXISTS idx_slashes_height + ON validator_slashes(block_height); + +-- down: +DROP INDEX IF EXISTS idx_slashes_height; +DROP INDEX IF EXISTS idx_slashes_validator; +DROP TABLE IF EXISTS validator_slashes; +``` + +**Implementation sketch** +- TASK-011 will expose `GET /api/validator/:addr/slashes` reading from this table. +- TASK-012 will INSERT into this table when equivocation is detected, then `UPDATE validators SET stake = stake - `. + +**Acceptance** +- [ ] Table + 2 indexes present. +- [ ] FK pattern (informal — no hard FK to validators.address since that's not unique-indexed in the original schema; document the soft join). + +**Verification** +- `INSERT INTO validator_slashes (validator_address, block_height, reason, stake_before, stake_after) VALUES ('0xabc', 100, 'equivocation', 100, 90);` succeeds. + +--- + +### TASK-313 — Migration 0009: contract_code table + +**Section:** db +**Effort:** S +**Depends on:** none +**Type:** migration + +**Goal** +TASK-079 (CREATE opcode), TASK-081 (contract code storage), and TASK-082 (code-loaded execution) need persistent contract bytecode keyed by contract address. The `accounts` table has a `code` TEXT column already but it's never populated — moving code to its own table avoids bloating account rows and lets us add code-specific metadata later. + +**Files** +- new: `backend/src/database/migrations/0009_contract_code.sql` + +**Migration SQL** +```sql +-- up: +CREATE TABLE IF NOT EXISTS contract_code ( + address TEXT PRIMARY KEY, + code_hash TEXT NOT NULL, + bytecode TEXT NOT NULL, + deployed_at_block BIGINT NOT NULL, + deployed_by TEXT NOT NULL, + deployed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP +); +CREATE INDEX IF NOT EXISTS idx_contract_code_hash + ON contract_code(code_hash); +CREATE INDEX IF NOT EXISTS idx_contract_code_deployer + ON contract_code(deployed_by); + +-- down: +DROP INDEX IF EXISTS idx_contract_code_deployer; +DROP INDEX IF EXISTS idx_contract_code_hash; +DROP TABLE IF EXISTS contract_code; +``` + +**Implementation sketch** +- `bytecode` stored as JSON-op string (matches the `vm:` prefix format from [Interpreter.ts](backend/src/vm/Interpreter.ts)) — TEXT not BYTEA. +- `code_hash` enables dedup detection for identical contracts. +- TASK-082 will look up code via `SELECT bytecode FROM contract_code WHERE address = $1` before treating a tx as a plain transfer. + +**Acceptance** +- [ ] Table + 2 indexes present. +- [ ] Migration runs after 0008 (sequential). + +**Verification** +- `\d contract_code` shows expected schema. + +--- + +### TASK-314 — Migration 0010: contract_storage table + +**Section:** db +**Effort:** S +**Depends on:** TASK-313 +**Type:** migration + +**Goal** +TASK-068 (SLOAD) and TASK-069 (storage persistence) need durable per-contract key/value storage. The Interpreter currently keeps storage in-memory inside the execution result and discards it. Persist it. + +**Files** +- new: `backend/src/database/migrations/0010_contract_storage.sql` + +**Migration SQL** +```sql +-- up: +CREATE TABLE IF NOT EXISTS contract_storage ( + contract_address TEXT NOT NULL, + storage_key TEXT NOT NULL, + storage_value TEXT NOT NULL, + updated_at_block BIGINT NOT NULL, + updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (contract_address, storage_key) +); +CREATE INDEX IF NOT EXISTS idx_contract_storage_block + ON contract_storage(updated_at_block); + +-- down: +DROP INDEX IF EXISTS idx_contract_storage_block; +DROP TABLE IF EXISTS contract_storage; +``` + +**Implementation sketch** +- Composite PK lets SLOAD do `WHERE contract_address = $1 AND storage_key = $2` as an index-only lookup. +- `updated_at_block` index supports state-snapshot logic from TASK-035. +- TASK-069 will `INSERT ... ON CONFLICT (contract_address, storage_key) DO UPDATE SET storage_value = EXCLUDED.storage_value`. + +**Acceptance** +- [ ] Table with composite PK present. +- [ ] Block-index in place. + +**Verification** +- Insert two rows with same address+key but different values, second is upsert not duplicate-key error. + +--- + +### TASK-315 — Migration 0011: contract_metadata table + +**Section:** db +**Effort:** S +**Depends on:** TASK-313 +**Type:** migration + +**Goal** +TASK-097 (event ABI registry), TASK-098 (source verifier), TASK-101 (contract metadata) need per-contract metadata: human-readable name, JSON ABI, source URL, verifier status. Separate from `contract_code` so verification can be done lazily without touching the bytecode row. + +**Files** +- new: `backend/src/database/migrations/0011_contract_metadata.sql` + +**Migration SQL** +```sql +-- up: +CREATE TABLE IF NOT EXISTS contract_metadata ( + address TEXT PRIMARY KEY, + name TEXT, + abi_json TEXT, + source_url TEXT, + source_verified BOOLEAN NOT NULL DEFAULT FALSE, + verifier_notes TEXT, + created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP +); +CREATE INDEX IF NOT EXISTS idx_contract_metadata_verified + ON contract_metadata(source_verified) WHERE source_verified = TRUE; + +-- down: +DROP INDEX IF EXISTS idx_contract_metadata_verified; +DROP TABLE IF EXISTS contract_metadata; +``` + +**Implementation sketch** +- Partial index on `source_verified = TRUE` keeps "list verified contracts" query cheap. +- ABI stored as JSON text; clients parse. +- Soft-link to `contract_code.address` (no hard FK so metadata can be created speculatively). + +**Acceptance** +- [ ] Table + partial index present. + +**Verification** +- `INSERT INTO contract_metadata (address, name) VALUES ('0xabc', 'MyContract')` works. + +--- + +### TASK-316 — Migration 0012: state_snapshots table + +**Section:** db +**Effort:** S +**Depends on:** none +**Type:** migration + +**Goal** +TASK-035 (state snapshot every 10k blocks) and TASK-036 (`/api/mesh/snapshot/:height` fast-sync) need a place to store periodic state checkpoints — full state-root + serialized state diff blob, keyed by height. + +**Files** +- new: `backend/src/database/migrations/0012_state_snapshots.sql` + +**Migration SQL** +```sql +-- up: +CREATE TABLE IF NOT EXISTS state_snapshots ( + height BIGINT PRIMARY KEY, + state_root TEXT NOT NULL, + account_count INTEGER NOT NULL, + storage_count INTEGER NOT NULL, + snapshot_blob BYTEA NOT NULL, + created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP +); + +-- down: +DROP TABLE IF EXISTS state_snapshots; +``` + +**Implementation sketch** +- BYTEA blob holds gzipped JSON of (accounts + contract_storage) at that height. +- TASK-035 will run as a cron during BlockProducer post-commit when `height % 10000 === 0`. +- TASK-036 will stream the blob over HTTP for peer fast-sync. + +**Acceptance** +- [ ] Table present, height as PK. + +**Verification** +- `INSERT` a 1MB blob, `SELECT octet_length(snapshot_blob) FROM state_snapshots` returns expected size. + +--- + +### TASK-317 — Migration 0013: peers table mirror of peers.json + +**Section:** db +**Effort:** S +**Depends on:** none +**Type:** migration + +**Goal** +The current peer registry persists to `data/peers.json` (file-backed) per [PeerRegistry.ts:60-79](backend/src/network/PeerRegistry.ts#L60-L79). For multi-replica deploys, the JSON file is local-only. Mirror to a `peers` table so all replicas see the same peer set. Keep the JSON as a write-through cache for fast cold-boot. + +**Files** +- new: `backend/src/database/migrations/0013_peers.sql` + +**Migration SQL** +```sql +-- up: +CREATE TABLE IF NOT EXISTS peers ( + peer_id TEXT PRIMARY KEY, + url TEXT NOT NULL, + chain_height BIGINT NOT NULL DEFAULT 0, + public_key TEXT NOT NULL DEFAULT '', + last_seen_ms BIGINT NOT NULL, + first_seen_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP +); +CREATE INDEX IF NOT EXISTS idx_peers_last_seen + ON peers(last_seen_ms DESC); + +-- down: +DROP INDEX IF EXISTS idx_peers_last_seen; +DROP TABLE IF EXISTS peers; +``` + +**Implementation sketch** +- After this migration, [PeerRegistry.registerPeer()](backend/src/network/PeerRegistry.ts) gets a follow-up commit (separate task in section 04) to UPSERT to PG too. +- Eviction loop will `DELETE FROM peers WHERE last_seen_ms < NOW() - 180000`. + +**Acceptance** +- [ ] Table + index present. + +**Verification** +- After commit lands, manual `INSERT` works. + +--- + +### TASK-318 — Migration 0014: rename chat_logs → agent_chat_logs + +**Section:** db +**Effort:** S +**Depends on:** none +**Type:** migration + +**Goal** +The `chat_logs` table from [schema.ts:115-128](backend/src/database/schema.ts#L115-L128) is misnamed — it stores agent conversations, not generic chat. Rename for consistency with `agent_memory`, `agent_completed_tasks`, etc. Use a view to keep old reads working during the rollout window. + +**Files** +- new: `backend/src/database/migrations/0014_rename_chat_logs.sql` + +**Migration SQL** +```sql +-- up: +ALTER TABLE chat_logs RENAME TO agent_chat_logs; +CREATE OR REPLACE VIEW chat_logs AS SELECT * FROM agent_chat_logs; + +-- down: +DROP VIEW IF EXISTS chat_logs; +ALTER TABLE agent_chat_logs RENAME TO chat_logs; +``` + +**Implementation sketch** +- View preserves backwards-compat for any forgotten callsite. Dropping the view is a separate later task. +- Indexes from the original (chat_logs_validator_address, chat_logs_created_at) get auto-renamed by ALTER TABLE … RENAME. + +**Acceptance** +- [ ] `\d agent_chat_logs` shows the table. +- [ ] `\d chat_logs` shows it as a VIEW. +- [ ] `SELECT * FROM chat_logs LIMIT 1` and `SELECT * FROM agent_chat_logs LIMIT 1` both work. + +**Verification** +- Boot succeeds (no callsites broken). +- Audit query `grep -rn "chat_logs" backend/src` returns hits for view-friendly read paths only. + +--- + +### TASK-319 — DB connection-pool tuning + monitoring + +**Section:** db +**Effort:** M +**Depends on:** none +**Type:** edit + +**Goal** +The PG pool from [db.ts:15-21](backend/src/database/db.ts#L15-L21) hardcodes `max: 20`. For Railway production with multiple replicas + worker, that's both under- and over-provisioned for different loads. Make pool size configurable via env, expose pool metrics (`waiting`, `idle`, `total`) for `/api/metrics`. + +**Files** +- edit: `backend/src/database/db.ts:15-21` — read `PG_POOL_MAX` (default 20), `PG_POOL_IDLE_MS` (default 30000), `PG_POOL_CONNECT_MS` (default 2000) from env. +- edit: same file — add `db.poolStats(): { total: number, idle: number, waiting: number }` reading from `pool.totalCount`, `pool.idleCount`, `pool.waitingCount`. + +**Reuses** +- node-postgres `Pool` instance fields (totalCount/idleCount/waitingCount). + +**API contract** +```ts +db.poolStats() → { total: 12, idle: 8, waiting: 0 } +``` + +**Implementation sketch** +- Add 3 env reads at module top. +- Add public method `poolStats` that returns 0/0/0 when no pool (in-memory mode). +- TASK-152 (`/api/metrics`) will export these as Prometheus gauges `hermes_pg_pool_{total,idle,waiting}`. + +**Acceptance** +- [ ] Env override works (`PG_POOL_MAX=50 npm run dev` boots with max=50). +- [ ] `db.poolStats()` returns sensible numbers under load. +- [ ] No breaking change for callers. + +**Verification** +- Hit a load test (TASK-400), watch `db.poolStats().waiting` rise then fall. + +--- + +### TASK-320 — DB query-time histogram metric + +**Section:** db +**Effort:** M +**Depends on:** TASK-319 +**Type:** edit + +**Goal** +We have no visibility into PG query latency. Wrap `db.query()` and `db.exec()` to record per-call duration into a histogram exposed at `/api/metrics`. Bucket boundaries: 1ms, 5ms, 10ms, 50ms, 100ms, 500ms, 1s, 5s. + +**Files** +- new: `backend/src/database/queryMetrics.ts` — exports `recordQuery(durationMs: number)`, `getHistogram(): { buckets: number[], counts: number[], sum: number, count: number }`. +- edit: `backend/src/database/db.ts:81-94` — wrap `query()` and `exec()` to call `recordQuery(Date.now() - start)`. + +**Reuses** +- Prometheus histogram convention (cumulative buckets). + +**API contract** +```ts +recordQuery(12) // ms +getHistogram() +→ { buckets: [1,5,10,50,100,500,1000,5000], counts: [0,0,12,40,8,1,0,0], sum: 23456, count: 61 } +``` + +**Implementation sketch** +- Use a flat array of bucket counts; increment the smallest bucket where `durationMs <= boundary`. +- Cumulative form computed at read time for Prometheus output (TASK-152). +- Sum + count accumulators for mean derivation. + +**Acceptance** +- [ ] Every successful PG query lands in the histogram. +- [ ] Failed queries also recorded (under separate counter `pg_query_errors_total`). + +**Verification** +- After 100 queries, `getHistogram().count === 100`. +- Slow query intentionally injected (`SELECT pg_sleep(2)`) lands in 1s+ bucket. + +--- + +### TASK-321 — DB slow-query log + +**Section:** db +**Effort:** S +**Depends on:** TASK-320 +**Type:** edit + +**Goal** +Queries above a threshold (default 1s) should be logged with the SQL + parameter shape (not values, to avoid leaking PII) for diagnosis. Use the wrapper from TASK-320. + +**Files** +- edit: `backend/src/database/db.ts` — extend the wrapper to `console.warn('[PG SLOW]', { sql, paramCount, durationMs })` when over threshold. + +**Reuses** +- Wrapper added in TASK-320. + +**Implementation sketch** +- Threshold via env `PG_SLOW_QUERY_MS` (default 1000). +- SQL truncated to 200 chars in log. +- Param values NEVER logged — only `params.length`. + +**Acceptance** +- [ ] Slow query logs include `[PG SLOW]` prefix. +- [ ] Param values absent from log line. + +**Verification** +- Inject `SELECT pg_sleep(2)`, observe log. + +--- + +### TASK-322 — Read-replica routing for heavy reads + +**Section:** db +**Effort:** L +**Depends on:** TASK-319 +**Type:** edit + +**Goal** +Single PG instance handles all reads + writes. As traffic grows, heavy reads (block listings, account history) should hit a read replica when one is available, so writes aren't latency-impacted. Add an optional `READ_DATABASE_URL` env; if set, route SELECT-only queries through a second pool. + +**Files** +- edit: `backend/src/database/db.ts` — add `readPool` second Pool when env present. +- edit: same — add `db.queryRead(sql, params)` method that prefers `readPool`, falls back to primary. + +**Reuses** +- Same Pool config conventions from TASK-319. + +**API contract** +```ts +db.queryRead('SELECT * FROM blocks WHERE height = $1', [100]) → {rows, rowCount} +``` + +**Implementation sketch** +- Detect `READ_DATABASE_URL`; if present, instantiate second Pool with same settings. +- `queryRead` tries readPool first, on connection error falls back to primary pool. +- Existing `db.query()` always hits primary (safe default). +- Audit endpoints (TASK-153, TASK-154, TASK-155) will be updated to use `queryRead` in their respective sections. + +**Acceptance** +- [ ] Without env, behavior unchanged. +- [ ] With env, `queryRead` hits replica. +- [ ] Replica unreachable → falls back to primary, logs warning. + +**Verification** +- Local: spin up two PG containers, point `READ_DATABASE_URL` at the second, observe `pg_stat_activity` on each. + +--- + +### TASK-323 — pg_dump backup script to S3 + +**Section:** db +**Effort:** M +**Depends on:** none +**Type:** new-file +**Type:** script + +**Goal** +We have no automated DB backup. Add a script that runs `pg_dump`, gzips, uploads to S3 with date-stamped key. Designed to be invoked from a cron (Railway, GH Action, or local). + +**Files** +- new: `backend/scripts/backup-db.ts` — entry point. +- new: `backend/scripts/README.md` — operator docs (or appended to existing README). + +**Reuses** +- Node `child_process.spawn` for `pg_dump`. +- AWS SDK v3 `@aws-sdk/client-s3` (add to deps if not present). + +**Env** +- `DATABASE_URL` — source. +- `S3_BACKUP_BUCKET` — target bucket. +- `S3_BACKUP_PREFIX` — key prefix (default `hermes-backups/`). +- `AWS_REGION`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` — AWS auth. + +**Implementation sketch** +- Spawn `pg_dump --no-owner --no-acl ` → pipe through `gzip` → `PutObjectCommand`. +- Key = `${prefix}${YYYY-MM-DD}/hermes-${HH:MM:SS}.sql.gz`. +- Log size, duration, key on success; non-zero exit on error. +- Optionally enforce retention: list keys older than 30d, delete (behind `--prune` flag). + +**Acceptance** +- [ ] `npm run backup` (script alias added in package.json) uploads a fresh dump. +- [ ] Key format matches spec. +- [ ] Script exits 1 on failure. + +**Verification** +- Run against a local PG + Localstack S3. + +--- + +### TASK-324 — Restore script + smoke test + +**Section:** db +**Effort:** M +**Depends on:** TASK-323 +**Type:** script + +**Goal** +A backup is only useful if restore works. Add a script that downloads the latest (or a specified) backup, restores into a target DB, and runs a smoke query (`SELECT COUNT(*) FROM blocks`, `SELECT COUNT(*) FROM transactions`). + +**Files** +- new: `backend/scripts/restore-db.ts` + +**Reuses** +- `@aws-sdk/client-s3` from TASK-323. +- `psql --dbname=$URL -f -` via spawn. + +**Env** +- `RESTORE_DATABASE_URL` — target (REFUSED if equals `DATABASE_URL` and `--force` not set, to prevent accidental prod restore). +- Other S3 env from TASK-323. + +**Implementation sketch** +- `npm run restore -- --key ` — restore that specific backup. +- `npm run restore -- --latest` — restore most recent. +- After restore, run smoke queries; print row counts. +- Refuse to overwrite a non-empty target unless `--force`. + +**Acceptance** +- [ ] Round-trip works: backup, drop target, restore, counts match. +- [ ] Smoke output printed. +- [ ] Safety check rejects identical source/target. + +**Verification** +- Backup local DB, create empty `hermes_restored`, restore into it, compare row counts. + +--- + +### TASK-325 — CLI: npm run migrate:down NNNN + +**Section:** db +**Effort:** S +**Depends on:** none +**Type:** edit + script + +**Goal** +The migration runner from [migrations.ts:110-154](backend/src/database/migrations.ts#L110-L154) only runs `up`. The `down:` half is parsed and ignored. Add a CLI for operators to roll back a specific migration locally. + +**Files** +- new: `backend/scripts/migrate-down.ts` — parse argv, find matching migration file, run its down half, delete row from `schema_migrations`. +- edit: `backend/package.json:scripts` — add `"migrate:down": "ts-node backend/scripts/migrate-down.ts"`. + +**Reuses** +- Migration loader logic from `loadMigrationsFrom()` in [migrations.ts:42-61](backend/src/database/migrations.ts#L42-L61). Export it (currently file-internal). + +**Implementation sketch** +- Usage: `npm run migrate:down 0007`. +- Refuses to run in production unless `FORCE_PROD_DOWN=1` env set. +- Wraps the down-block in a transaction so partial failure rolls back. +- Removes corresponding row from `schema_migrations`. + +**Acceptance** +- [ ] Running with valid NNNN executes the down block. +- [ ] `schema_migrations` row removed. +- [ ] Re-running `npm run dev` re-applies the up. + +**Verification** +- Apply 0007, run down, verify column dropped, verify row gone, restart, verify column added again. + +--- + +### TASK-326 — Migration dry-run mode + +**Section:** db +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +`applyPendingMigrations()` always writes. Operators can't preview what would change. Add a dry-run mode that prints the SQL each pending migration would execute, then exits without writing. + +**Files** +- edit: `backend/src/database/migrations.ts:110` — add an optional `{ dryRun?: boolean }` arg. +- new: `backend/scripts/migrate-status.ts` — prints applied + pending, with `--dry-run` flag invoking `applyPendingMigrations({ dryRun: true })`. +- edit: `backend/package.json:scripts` — add `"migrate:status": "ts-node backend/scripts/migrate-status.ts"`. + +**Reuses** +- Existing `migrationStatus()` from [migrations.ts:156-167](backend/src/database/migrations.ts#L156-L167). + +**Implementation sketch** +- In dry-run, replace `db.exec(migration.up)` with `console.log('[DRY-RUN]', migration.name, '\n', migration.up)`. +- Skip the INSERT into schema_migrations. +- Lock still acquired/released so behavior matches real run. + +**Acceptance** +- [ ] `npm run migrate:status -- --dry-run` prints SQL without applying. +- [ ] Real boot still applies migrations as normal. + +**Verification** +- Add a stub migration, dry-run, observe SQL printed, observe nothing in `schema_migrations`. + +--- + +### TASK-327 — Schema-diff against prod CLI + +**Section:** db +**Effort:** M +**Depends on:** none +**Type:** script + +**Goal** +Drift between dev schema and prod is invisible. Add a script that connects to two DB URLs (dev + prod), introspects every table + index + column, and prints a diff. Read-only; never writes. + +**Files** +- new: `backend/scripts/schema-diff.ts` +- edit: `backend/package.json:scripts` — add `"schema:diff": "ts-node backend/scripts/schema-diff.ts"`. + +**Reuses** +- `pg` Client (not Pool — short-lived, two distinct URLs). + +**Implementation sketch** +- Args: `npm run schema:diff -- --left $DEV_URL --right $PROD_URL`. +- For each side, query `information_schema.tables`, `.columns`, `pg_indexes`. +- Build a normalized fingerprint per table (sorted columns, sorted indexes). +- Diff fingerprints; print added/removed/changed. +- Exit 1 if differences found (CI-friendly). + +**Acceptance** +- [ ] Reports identical schemas as "no diff". +- [ ] Reports an extra column on one side as a single line `+ accounts.foo TEXT`. + +**Verification** +- Diff against itself → no output, exit 0. +- Apply a migration on one only → diff shows the change. + +--- + +### TASK-328 — Redis cache warmer at boot + +**Section:** db +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Cold boot leaves Redis empty; the first ~30s of traffic causes burst PG reads as the cache populates. Pre-warm with the things `chainState` (line 225 of db.ts) is going to read anyway: latest 100 blocks, top 50 accounts by balance, last block height. + +**Files** +- new: `backend/src/database/cacheWarmer.ts` — exports `warmCache(): Promise`. +- edit: `backend/src/api/server.ts` — invoke after `applyPendingMigrations()` and before route mounting. + +**Reuses** +- `cache.setJSON()` from [db.ts:148-170](backend/src/database/db.ts#L148-L170). +- `chainState` getters from [db.ts:225-280](backend/src/database/db.ts#L225-L280). + +**Implementation sketch** +- Warmer runs once at boot, behind `CACHE_WARMER_ENABLED=true` env (default off in dev, on in production). +- Reads: + - Last 100 blocks → `cache.setJSON('block:height:N', ..., 300)`. + - Top 50 accounts → `cache.setJSON('top_accounts', ..., 60)`. +- All warming work runs in parallel via `Promise.all`. +- Logs total entries warmed + duration. + +**Acceptance** +- [ ] Warmer boots cleanly even with empty PG. +- [ ] Logs count + duration. +- [ ] Doesn't block the server from starting (parallel-with-listen). + +**Verification** +- Boot with empty cache, confirm `cache.get('block:height:1')` returns the block right after boot completes. + +--- + +### TASK-329 — Redis key TTL audit + +**Section:** db +**Effort:** S +**Depends on:** none +**Type:** script + +**Goal** +The `cache` wrapper accepts an optional TTL but many callsites pass none, leaking memory long-term. Add a script that scans every `cache.set/setJSON/hset` callsite in `backend/src/` and reports the ones missing a TTL. + +**Files** +- new: `backend/scripts/audit-redis-ttl.ts` + +**Reuses** +- Plain `fs.readdirSync` recursion + regex matching. + +**Implementation sketch** +- Walk `backend/src/`, read every .ts. +- Match `cache.set(`, `cache.setJSON(`, `cache.hset(` invocations. +- Inspect arg list: third arg present = TTL set; absent = leak risk. +- Print one line per leak: `path:line — cache.set('foo', ...)` (no TTL). +- Exit 1 if any leaks found (CI-friendly). + +**Acceptance** +- [ ] Script runs, prints exactly one line per missing-TTL callsite. +- [ ] Recognizes inline string keys + variable keys. + +**Verification** +- Add a test fixture with one missing-TTL call, run script, see it. + +--- + +### TASK-330 — Redis pub/sub channel for cross-replica events + +**Section:** db +**Effort:** M +**Depends on:** none +**Type:** new-file + +**Goal** +Today's [EventBus](backend/src/events/EventBus.ts) is in-process. SSE clients on replica A miss events emitted on replica B. Add a Redis pub/sub bridge: emit local → republish to channel; subscribe channel → emit local. Replicas converge on the same event stream. + +**Files** +- new: `backend/src/events/RedisBridge.ts` — exports `attachRedisBridge(eventBus: EventBus, redis: Redis): { detach(): void }`. +- edit: `backend/src/api/server.ts` — invoke after eventBus + redis are constructed. + +**Reuses** +- ioredis duplicated client pattern (one publisher + one subscriber, since subscriber blocks). +- Existing event names from grep over `eventBus.emit(`. + +**Implementation sketch** +- Whitelist of event types that are safe to bridge (block_produced, network_message, ci_results, consensus_quorum) — others stay local. +- Loop-prevention: tag bridged messages with `_origin = REPLICA_ID`; ignore if echoed back. +- Channel name: `hermes:events:v1`. + +**Acceptance** +- [ ] Two-replica test: emit on A, observe on B (within 100ms). +- [ ] No infinite loop. +- [ ] Non-whitelisted events stay local. + +**Verification** +- Local: run two backend instances pointing at same Redis, watch B's logs when A produces a block. + +--- + +### TASK-331 — Two-replica web service: pin SSE to one replica + +**Section:** db +**Effort:** M +**Depends on:** TASK-330 +**Type:** edit + +**Goal** +With two web replicas behind a load balancer, SSE clients can land on either. Without sticky sessions, a reconnect lands on the other replica with no event history. Either: (a) sticky sessions via cookie, or (b) only one replica accepts SSE. Pick (b) since LB config may not be in our hands. + +**Files** +- edit: `backend/src/api/server.ts` — `/api/agent/stream` and other SSE routes return 503 if `process.env.SSE_REPLICA !== 'true'`. + +**Reuses** +- Existing SSE route handlers. + +**Implementation sketch** +- Env `SSE_REPLICA=true` set on exactly one replica via Railway service config. +- 503 response includes header `X-SSE-Failover: true` so clients can retry against a different host (when we add multiple). +- HUD reconnect logic (TASK-245) handles the 503. + +**Acceptance** +- [ ] Default replica returns 503 on `/api/agent/stream`. +- [ ] Replica with env=true serves SSE normally. + +**Verification** +- Two-replica Railway deploy: only the one with env serves the HUD ticker. + +--- + +### TASK-332 — Worker leader election + +**Section:** db +**Effort:** M +**Depends on:** none +**Type:** new-file + +**Goal** +Two worker replicas would both run the AgentWorker loop simultaneously and produce duplicate commits. Add a leader-election lease via Redis SETNX with TTL. Only the leader runs the agent loop; followers no-op. + +**Files** +- new: `backend/src/agent/leaderElection.ts` — exports `acquireLeadership(redis: Redis, leaseId: string): Promise` and `renewLease(): Promise`. +- edit: `backend/src/worker.ts` — wrap the agent loop start with leadership check; renew every 10s; release on shutdown. + +**Reuses** +- ioredis `set('key', val, 'PX', ttlMs, 'NX')` semantics. + +**Implementation sketch** +- Key: `hermes:worker:leader`. +- Value: this replica's hostname + pid. +- TTL: 30s; renewal every 10s with `EXPIRE` (only if value matches our id, via Lua script for atomicity). +- On lease loss (renew returns false), kill the agent loop, fall back to "follower" state. + +**Acceptance** +- [ ] Single worker: gets leader, agent runs. +- [ ] Two workers: only one gets leader; the other logs `[LEADER] follower mode`. +- [ ] Kill the leader: the follower picks up within ~30s. + +**Verification** +- Local: run worker.ts twice against same Redis, observe. + +--- + +### TASK-333 — Stuck-job recovery + +**Section:** db +**Effort:** M +**Depends on:** TASK-332 +**Type:** edit + +**Goal** +If the agent worker crashes mid-task, the task stays in "in_progress" forever and never gets retried. Add a recovery sweep that, on worker startup (and once every 5 min), finds tasks in `agent_tasks` with `status='in_progress'` and `started_at < NOW() - 1 hour`, resets them to `status='pending'` and increments a `recovery_count`. + +**Files** +- new: `backend/src/database/migrations/0015_agent_tasks_recovery.sql` — `ALTER TABLE agent_tasks ADD COLUMN IF NOT EXISTS recovery_count INTEGER NOT NULL DEFAULT 0;` +- edit: `backend/src/agent/AgentWorker.ts` — add `recoverStuckTasks()` method, call from constructor + interval. + +**Reuses** +- Existing PG access via `db.query`. +- Pattern from existing AgentWorker query callsites. + +**Implementation sketch** +- Recovery query: `UPDATE agent_tasks SET status='pending', recovery_count=recovery_count+1 WHERE status='in_progress' AND started_at < NOW() - INTERVAL '1 hour' RETURNING id`. +- Log each recovered task ID. +- After 5 recoveries on the same task, mark it `status='abandoned'` (separate hard-stop to avoid infinite retry loops). + +**Acceptance** +- [ ] Stuck tasks return to pending. +- [ ] `recovery_count` increments. +- [ ] After 5 recoveries, task marked abandoned and logged. + +**Verification** +- Insert a fake stuck row, run AgentWorker, observe. + +--- + +### TASK-334 — Dead-letter queue for failed agent tasks + +**Section:** db +**Effort:** M +**Depends on:** TASK-333 +**Type:** edit + migration + +**Goal** +Tasks that fail repeatedly get marked `failed` but stay in the same table, cluttering the active queue. Move them to a dead-letter table with the failure reason for triage. + +**Files** +- new: `backend/src/database/migrations/0016_dead_letter_tasks.sql` +- edit: `backend/src/agent/AgentWorker.ts` — when `failure_count >= 3`, move row to `dead_letter_tasks` and delete from `agent_tasks`. + +**Migration SQL** +```sql +-- up: +CREATE TABLE IF NOT EXISTS dead_letter_tasks ( + id TEXT PRIMARY KEY, + original_task_json TEXT NOT NULL, + last_error TEXT, + failure_count INTEGER NOT NULL, + moved_to_dlq_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP +); +CREATE INDEX IF NOT EXISTS idx_dlq_moved_at + ON dead_letter_tasks(moved_to_dlq_at DESC); + +-- down: +DROP INDEX IF EXISTS idx_dlq_moved_at; +DROP TABLE IF EXISTS dead_letter_tasks; +``` + +**Implementation sketch** +- Move = `INSERT INTO dead_letter_tasks (id, original_task_json, last_error, failure_count) SELECT id, row_to_json(t.*)::text, last_error, failure_count FROM agent_tasks t WHERE id = $1; DELETE FROM agent_tasks WHERE id = $1;` in one transaction. + +**Acceptance** +- [ ] Task that fails 3x lands in DLQ. +- [ ] Active queue no longer contains it. + +**Verification** +- Insert a synthetic always-failing task, run agent, watch it move. + +--- + +### TASK-335 — Job retry dashboard + +**Section:** db +**Effort:** M +**Depends on:** TASK-334 +**Type:** new-file + +**Goal** +Surface DLQ + recovery stats in the HUD. Endpoint that returns counts of pending / in_progress / failed / abandoned / dead-lettered, plus the 20 most recent DLQ entries. + +**Files** +- new: `backend/src/api/jobs.ts` — Express router. +- edit: `backend/src/api/server.ts` — mount at `/api/jobs`. + +**Reuses** +- Express router pattern from [backend/src/api/wallet.ts](backend/src/api/wallet.ts). + +**API contract** +``` +GET /api/jobs/stats +→ 200 { + pending: 10, + in_progress: 1, + failed: 2, + abandoned: 0, + dead_lettered: 5 +} + +GET /api/jobs/dlq?limit=20 +→ 200 { items: [ { id, last_error, failure_count, moved_to_dlq_at } ] } + +POST /api/jobs/dlq/:id/retry +→ 200 { restored: true } +→ 404 { error: 'not in dlq' } +``` + +**Implementation sketch** +- `/stats` is one query against `agent_tasks` group-by status + one count from `dead_letter_tasks`. +- `/dlq` reads recent rows. +- `/dlq/:id/retry` moves row back to `agent_tasks` with `status='pending'`, `failure_count=0`. +- Retry endpoint is admin-gated via `requireApiKey('jobs:write')` from [auth.ts](backend/src/api/auth.ts). + +**Acceptance** +- [ ] `/stats` returns counts matching DB ground truth. +- [ ] `/dlq/:id/retry` moves row, returns 200. +- [ ] Auth on retry rejected without admin scope. + +**Verification** +- `curl /api/jobs/stats` against running backend. + +--- + +## Summary + +| TASK | Title | Effort | +|---|---|---| +| 306 | Index transactions(block_height) | S | +| 307 | Index transactions(from_address, nonce) | S | +| 308 | Index accounts(balance DESC) | S | +| 309 | Index receipts(status) | S | +| 310 | GIN index on receipts.logs_json | S | +| 311 | validators.stake column | S | +| 312 | validator_slashes table | S | +| 313 | contract_code table | S | +| 314 | contract_storage table | S | +| 315 | contract_metadata table | S | +| 316 | state_snapshots table | S | +| 317 | peers table | S | +| 318 | Rename chat_logs | S | +| 319 | Pool tuning + monitoring | M | +| 320 | Query-time histogram | M | +| 321 | Slow-query log | S | +| 322 | Read-replica routing | L | +| 323 | pg_dump → S3 | M | +| 324 | Restore script + smoke test | M | +| 325 | migrate:down CLI | S | +| 326 | Migration dry-run mode | S | +| 327 | Schema-diff CLI | M | +| 328 | Redis cache warmer | S | +| 329 | Redis TTL audit script | S | +| 330 | Redis pub/sub bridge | M | +| 331 | SSE replica pinning | M | +| 332 | Worker leader election | M | +| 333 | Stuck-job recovery | M | +| 334 | Dead-letter queue | M | +| 335 | Job retry dashboard | M | + +13 small, 14 medium, 1 large. Net effort ≈ 30 commits over ~2 days at the 60/day cadence. From 5021ad786d3877052225d0f21e1a6e7ecac46a8e Mon Sep 17 00:00:00 2001 From: hermes agent Date: Tue, 28 Apr 2026 01:44:06 +0400 Subject: [PATCH 02/96] =?UTF-8?q?docs(backlog):=20detailed=20specs=20for?= =?UTF-8?q?=20section=2001=20=E2=80=94=20chain=20&=20consensus=20(TASK-001?= =?UTF-8?q?..060)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 60 task specs covering Block.fromJSON deserializer (unblocks gossip-apply), peer head poller + auto-sync + reorg-on-sync, finality flag at depth-12 + finality-violation rejection, VRF-style proposer rotation, equivocation slashing, stake-weighted quorum, block timestamp drift + min-time checks, difficulty retargeting, mempool persistence + RBF + size cap + TTL, log indexing + filter endpoint + bloom helper, GHOST fork-choice with uncle tracking, chain export/import, genesis parameterization + hash verify, beacon randomness, state pruning + snapshots + fast-sync endpoint, fee distribution (80/20 producer/burn) + coinbase receipt, account-history rebuild + verify-chain CLI, state-root mismatch alarm, receipt-root verify on sync, three SSE streams (logs/mempool/forks), per-block gas stats, tps/block-times/uptime endpoints, mempool depth chart, tx simulate/estimate-gas, account next-nonce + history, validator blocks, reorg log. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/backlog/queue/01-chain-consensus.md | 1815 ++++++++++++++++++++++ 1 file changed, 1815 insertions(+) create mode 100644 docs/backlog/queue/01-chain-consensus.md diff --git a/docs/backlog/queue/01-chain-consensus.md b/docs/backlog/queue/01-chain-consensus.md new file mode 100644 index 00000000..6b604081 --- /dev/null +++ b/docs/backlog/queue/01-chain-consensus.md @@ -0,0 +1,1815 @@ +# Section 01 — Chain & Consensus Specs (TASK-001..060) + +60 tasks. Block deserialization for gossip-apply, peer sync, finality, validator rotation/slashing, mempool hardening, log indexing, fork accounting, fee burning, scripts/CLIs, and a batch of read endpoints (`tps`, `block-times`, account/validator history, reorg log). + +**Preconditions used throughout this section:** +- Chain core: [Chain.ts](backend/src/blockchain/Chain.ts) — `addBlock` (line 193) routes non-canonical blocks via `forkManager.addBlock()` (line 205); `handleReorg(newBlocks, commonAncestorHeight)` at line 361 reverts state then re-applies; `getChainLength()` (line 308) returns synthetic wall-clock height. +- BlockProducer: [BlockProducer.ts](backend/src/blockchain/BlockProducer.ts) `produceBlock()` loop at line 70. +- Consensus singletons: `proofOfAI`, `difficultyManager`, `forkManager` from [Consensus.ts](backend/src/blockchain/Consensus.ts). `ForkManager.isFinalized(hash)` at line 239 uses `FORK_CHOICE_DEPTH = 6`. +- TransactionPool: [TransactionPool.ts](backend/src/blockchain/TransactionPool.ts) — `addTransaction` (line 68), `evictInvalid` (line 143), `readmitOrphaned` (line 168), `validateTransaction` (line 184). +- StateManager: [StateManager.ts](backend/src/blockchain/StateManager.ts) — `applyTransaction`, `revertBlock`, `applyBlockReward`, `commitBlock`, `getStateRoot`. +- Receipts: [TransactionReceipt.ts](backend/src/blockchain/TransactionReceipt.ts) — `createReceipt`, `storeReceipt`, `loadReceipt`, `loadBlockReceipts`, `calculateReceiptsRoot`. +- Block class: [Block.ts](backend/src/blockchain/Block.ts) — constructor `(height, parentHash, producer, transactions, difficulty)`, `isValid(prev?)`, `toJSON()`. +- Crypto: [Crypto.ts](backend/src/blockchain/Crypto.ts) — `verifyTransactionSignature(tx)`, `verify(message, sig, pubkey)`, `sha256Base58`. +- Mesh router: [backend/src/network/api.ts](backend/src/network/api.ts) — `/api/mesh/*` routes; `POST /api/mesh/block` currently returns `accepted:false`. +- Event bus: [EventBus.ts](backend/src/events/EventBus.ts) — `eventBus.emit(name, payload)`. Existing events listed in section 08 preamble. +- SSE pattern: [server.ts:971-1057](backend/src/api/server.ts#L971-L1057) — pattern for new SSE endpoints in TASK-047/048/049. + +--- + +### TASK-001 — Block.fromJSON deserializer + +**Section:** chain +**Effort:** M +**Depends on:** none +**Type:** edit + +**Goal** +[Block.ts](backend/src/blockchain/Block.ts) only has `toJSON()` (line 173). Gossip-apply (TASK-002) and chain import (TASK-029) need the inverse: take the JSON shape produced by `toJSON()` and reconstruct an equivalent Block instance whose `.header.hash` matches the original. + +**Files** +- edit: `backend/src/blockchain/Block.ts` — add `static fromJSON(json: any): Block` after the existing `toJSON` method. + +**Reuses** +- The Block constructor's hash calculation in [Block.ts:103-115](backend/src/blockchain/Block.ts#L103-L115). +- `setStateRoot()` (line 143) for re-applying the deserialized state root. + +**Implementation sketch** +- Validate that `json` has every required header field; throw `Error('fromJSON: missing field ')` with explicit field name on miss. +- Reconstruct each transaction by parsing `value`, `gasPrice`, `gasLimit` back from string → bigint. +- Construct `new Block(json.height, json.parentHash, json.producer, txs, json.difficulty)`. +- Override generated `timestamp`, `nonce`, `gasUsed`, `gasLimit` via direct header assignment. +- Call `setStateRoot(json.stateRoot)` so the hash recomputes. +- Override `transactionsRoot` and `receiptsRoot` to the deserialized values, then recompute hash. +- Final assertion: `block.header.hash === json.hash`; throw `Error('fromJSON: hash mismatch — header field tampered')` if not. + +**Acceptance** +- [ ] Round-trip property: `Block.fromJSON(b.toJSON()).header.hash === b.header.hash` for any locally-produced block. +- [ ] Tampered field (e.g. mutate `json.height`) → fromJSON throws `hash mismatch`. +- [ ] Empty transactions list works. + +**Verification** +- Local script: produce a block, JSON it, fromJSON it, compare every header field. + +--- + +### TASK-002 — Wire /api/mesh/block to call chain.addBlock after fromJSON + +**Section:** chain +**Effort:** S +**Depends on:** TASK-001 +**Type:** edit + +**Goal** +[network/api.ts:55-73](backend/src/network/api.ts#L55-L73) currently returns `accepted: false` because the deserializer was missing. With TASK-001, complete the gossip path: deserialize → addBlock → respond with the accept/reorg outcome. + +**Files** +- edit: `backend/src/network/api.ts:55-73` — replace the conservative stub with real acceptance. + +**Reuses** +- `Block.fromJSON` from TASK-001. +- `chain.addBlock(block)` from [Chain.ts:193](backend/src/blockchain/Chain.ts#L193). + +**API contract** +``` +POST /api/mesh/block +body: { ...Block.toJSON output... } +→ 200 { accepted: true, head: { height, hash } } +→ 409 { accepted: false, reason: 'parent unknown' | 'finalized-conflict' | 'invalid' } +→ 400 { accepted: false, reason: 'fromJSON failed: ' } +``` + +**Implementation sketch** +- Try `Block.fromJSON(req.body)` inside try/catch → 400 on throw. +- Call `chain.addBlock(block)`. +- 200 with new head if accepted; 409 with reason otherwise. +- Continue emitting `mesh_block_received` for observability. + +**Acceptance** +- [ ] Valid block from a peer at our parent → accepted. +- [ ] Block with unknown parent → 409 `parent unknown`. +- [ ] Tampered block → 400 with deserializer message. + +**Verification** +- Local two-node sim: A produces, B receives via curl. + +--- + +### TASK-003 — Header-only sync endpoint /api/mesh/headers + +**Section:** chain +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Full-block sync is bandwidth-heavy. Peers that only need to verify the chain tip / measure honest-majority can pull headers only. Add a header-range fetch. + +**Files** +- edit: `backend/src/network/api.ts` — add `GET /api/mesh/headers`. + +**Reuses** +- `chain.getBlockByHeight(h)` from [Chain.ts:296](backend/src/blockchain/Chain.ts#L296). + +**API contract** +``` +GET /api/mesh/headers?from=100&to=200 +→ 200 { headers: [ { height, hash, parentHash, producer, timestamp, stateRoot, transactionsRoot, receiptsRoot, gasUsed, gasLimit, difficulty } ... ] } +→ 400 { error: 'from > to' | 'range exceeds 1000' } +``` + +**Implementation sketch** +- Hard cap range: `to - from <= 1000`. +- Loop `from..=to`, look up each block, push its header (not transactions). +- Skip-and-continue on missing heights (don't 500). +- Convert `gasUsed`/`gasLimit` bigint → string. + +**Acceptance** +- [ ] Range under 1000 returns headers. +- [ ] Range over 1000 returns 400. +- [ ] Out-of-range heights silently skipped, not erroring. + +**Verification** +- `curl /api/mesh/headers?from=0&to=5` returns 6 headers (genesis + 5). + +--- + +### TASK-004 — Bulk block fetch /api/mesh/blocks + +**Section:** chain +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +TASK-003's headers tell a peer what's missing; this endpoint serves the actual blocks for sync. + +**Files** +- edit: `backend/src/network/api.ts` — add `GET /api/mesh/blocks`. + +**Reuses** +- `chain.getBlockByHeight()` and `block.toJSON()`. + +**API contract** +``` +GET /api/mesh/blocks?from=100&to=200 +→ 200 { blocks: [ { ...toJSON() } ... ] } +→ 400 { error: 'range exceeds 100' } // tighter cap than headers +``` + +**Implementation sketch** +- Cap range: `to - from <= 100` (full blocks are bigger than headers). +- Same skip-on-missing semantics as TASK-003. + +**Acceptance** +- [ ] Returns full block objects compatible with TASK-001's `fromJSON`. +- [ ] Range >100 returns 400. + +**Verification** +- Round-trip: pull blocks via this endpoint, fromJSON each, hash matches original. + +--- + +### TASK-005 — Peer head poller + +**Section:** chain +**Effort:** M +**Depends on:** none +**Type:** new-file + +**Goal** +A peer that announces never updates its `chainHeight` again until we ask. We need a 30s poller that hits each peer's `/api/mesh/head` and refreshes their `chainHeight` in our [PeerRegistry](backend/src/network/PeerRegistry.ts). + +**Files** +- new: `backend/src/network/headPoller.ts` — exports `startHeadPoller(): {stop()}`. +- edit: `backend/src/api/server.ts` — start poller after mesh router mount. + +**Reuses** +- `peerRegistry.listPeers()` and `registerPeer({...input, lastSeenMs: now})`. +- Native `fetch()`. + +**Implementation sketch** +- Every 30s: `Promise.allSettled(peers.map(p => fetch(p.url + '/api/mesh/head')))`. +- On 200: update peer's `chainHeight` via `registerPeer()` (which doubles as a heartbeat). +- On error: do NOT decrement lastSeen; eviction loop handles dead peers. +- Skip self (compare against `HERMES_PUBLIC_URL`). + +**Acceptance** +- [ ] Two-node setup: heights converge in PeerRegistry within ~60s of one node producing a new block. +- [ ] Dead peer doesn't crash the loop. + +**Verification** +- Run two backends locally, watch peerRegistry.listPeers() over time. + +--- + +### TASK-006 — Auto-sync on start: pick highest-height peer, pull missing + +**Section:** chain +**Effort:** M +**Depends on:** TASK-001, TASK-002, TASK-004 +**Type:** new-file + +**Goal** +Cold-boot replica is at height 0 (or wherever DB left off). It should detect peers, find the highest known head, and pull missing blocks until caught up — automatically. + +**Files** +- new: `backend/src/network/syncManager.ts` — exports `runInitialSync(chain): Promise<{synced: number, from: string|null}>`. +- edit: `backend/src/api/server.ts` — invoke after `chain.initialize()` and after first peer announce returns (give it 5s for peers to reply). + +**Reuses** +- `peerRegistry.listPeers()`, `chain.getChainLength()`, `chain.addBlock()`, `Block.fromJSON()`. + +**Implementation sketch** +- Wait 5s after boot for at least one peer announce. +- If no peers, no-op (we may BE the seed). +- Else: pick `peers.sort((a,b) => b.chainHeight - a.chainHeight)[0]`. +- If their height ≤ ours, no-op. +- Loop: pull `[ourHeight+1 .. min(theirHeight, ourHeight+100)]` via `/api/mesh/blocks`, fromJSON each, addBlock. +- If addBlock returns false, log + skip (will be retried next sync tick). +- After full catch-up, log `[SYNC] caught up to height N from peer X`. + +**Acceptance** +- [ ] Empty replica + one full peer: replica catches up to peer's height. +- [ ] No peers: silent no-op, doesn't block boot. + +**Verification** +- Drop a fresh DB, start backend with bootstrap peer pointing at a populated peer. + +--- + +### TASK-007 — Reorg-on-sync: walk back to common ancestor + +**Section:** chain +**Effort:** L +**Depends on:** TASK-006 +**Type:** edit + +**Goal** +TASK-006 assumes the local chain is a strict prefix of the peer's. If they've diverged, we need to find the common ancestor (via header probes), then reorg from there. Otherwise we'd just see "parent unknown" rejections forever. + +**Files** +- edit: `backend/src/network/syncManager.ts` — extend with `findCommonAncestor(peer)` + reorg-walk path. + +**Reuses** +- `Chain.findCommonAncestor` exists at [Chain.ts:476](backend/src/blockchain/Chain.ts#L476) but works on in-memory blocks; we need a peer-side variant. +- `chain.handleReorg(newBlocks, commonAncestorHeight)` at [Chain.ts:361](backend/src/blockchain/Chain.ts#L361). + +**Implementation sketch** +- `findCommonAncestor(peerUrl)`: + - Pull peer's headers in chunks of 100, walking back from peer head. + - For each header, check if `chain.getBlockByHash(header.hash)` exists locally. + - First match is the ancestor. If walk exhausts without match, ancestor = height 0 (genesis). +- After ancestor found: pull peer's full blocks for `[ancestor+1 .. peerHead]`, then call `chain.handleReorg(newBlocks, ancestor)`. +- Add a max-depth guard (default 1000) — refuse to reorg deeper than that without operator confirmation (env `MAX_REORG_DEPTH`). + +**Acceptance** +- [ ] Two nodes diverge at height 50, sync converges them on the longer chain. +- [ ] Reorg deeper than `MAX_REORG_DEPTH` is refused with `[SYNC] refusing reorg of depth N > max M`. + +**Verification** +- Local: produce different chains on two nodes, then bring up syncManager. + +--- + +### TASK-008 — Finalized block flag at depth N-12 + +**Section:** chain +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +[ForkManager.isFinalized()](backend/src/blockchain/Consensus.ts) uses depth-6. Hermeschain wants 12 (matching common L1 conventions). Centralize the constant, expose finality height in chainState, surface in `/api/status`. + +**Files** +- edit: `backend/src/blockchain/Consensus.ts` — change `FORK_CHOICE_DEPTH` from 6 → 12 (or via env `FINALITY_DEPTH`). +- edit: `backend/src/database/db.ts:225-280` (chainState) — add `setFinalizedHeight(h)`, `getFinalizedHeight()`. +- edit: `backend/src/blockchain/Chain.ts` — after each successful `addBlock`, write finalized height = `chainLength - 12` to chainState. + +**Reuses** +- `chainState` Redis-backed store. + +**Implementation sketch** +- After `addBlock` succeeds and updates `chainState.saveBlockHeight()`, additionally call `chainState.setFinalizedHeight(Math.max(0, this.getChainLength() - 12))`. +- Surface in `/api/status` payload. + +**Acceptance** +- [ ] After 100 blocks, finalized height = 88. +- [ ] Visible in `/api/status` JSON. + +**Verification** +- `curl /api/status | jq .finalizedHeight`. + +--- + +### TASK-009 — Reject reorg attempts past finality depth + +**Section:** chain +**Effort:** S +**Depends on:** TASK-008 +**Type:** edit + +**Goal** +Reorgs deeper than the finality depth violate finality guarantees. Refuse them in `addBlock` and `handleReorg`. + +**Files** +- edit: `backend/src/blockchain/Chain.ts:193,361` — pre-check reorg depth against finalized height. + +**Reuses** +- `chainState.getFinalizedHeight()` from TASK-008. +- `findCommonAncestor` at [Chain.ts:476](backend/src/blockchain/Chain.ts#L476). + +**Implementation sketch** +- In `addBlock` fork-route: before calling `forkManager.addBlock`, compute the would-be reorg depth. If `commonAncestorHeight < finalizedHeight`, log and return false with reason `finality-violation`. +- Same check in `handleReorg`. +- Caller endpoints (TASK-002) translate this to 409 with `reason: 'finalized-conflict'`. + +**Acceptance** +- [ ] Reorg at depth ≥ finality depth: refused. +- [ ] Shallow reorg: works as before. + +**Verification** +- Force a deep reorg via `/api/mesh/block` with a competing chain at depth 20; expect 409. + +--- + +### TASK-010 — VRF-style proposer rotation hash(prev_hash + height) mod n + +**Section:** chain +**Effort:** S +**Depends on:** TASK-013 (need stake column for validator listing) +**Type:** edit + +**Goal** +Current rotation in [ValidatorManager.ts](backend/src/validators/ValidatorManager.ts) uses `height % n`. That's predictable; replace with a deterministic hash-based rotation that uses parent hash entropy too: `producer = validators[ hash(prevHash + height) mod n ]`. + +**Files** +- edit: `backend/src/validators/ValidatorManager.ts` — `selectProducer` accepts `(nextHeight, parentHash)`; use sha256 → bigint → mod. +- edit: `backend/src/blockchain/BlockProducer.ts:75` — pass parent hash to selectProducer. + +**Reuses** +- `sha256Base58` from [Crypto.ts:194](backend/src/blockchain/Crypto.ts#L194). + +**Implementation sketch** +- `const seed = sha256(`${parentHash}:${nextHeight}`)`. +- `const idx = Number(BigInt('0x' + seed.slice(0, 16)) % BigInt(validatorOrder.length))`. +- For backwards-compat, when `parentHash` not provided, fall back to `height % n`. + +**Acceptance** +- [ ] Same (parentHash, height) → same producer. +- [ ] Different parentHash → distribution flat over many trials. +- [ ] No-arg call works as before. + +**Verification** +- Unit test: 1000 selections across (parentHash, height) pairs cover all validators ±10%. + +--- + +### TASK-011 — validator_slashes read endpoint + +**Section:** chain +**Effort:** S +**Depends on:** TASK-312 (table exists) +**Type:** edit + +**Goal** +Surface the slashing log built by TASK-012. Two endpoints: per-validator history and global recent. + +**Files** +- new: handlers in `backend/src/api/server.ts` (or new `backend/src/api/slashing.ts` router). + +**Reuses** +- `db.query` against `validator_slashes`. + +**API contract** +``` +GET /api/validator/:addr/slashes +→ 200 { slashes: [ { id, block_height, reason, evidence_json, stake_before, stake_after, slashed_at } ] } + +GET /api/slashing/recent?limit=50 +→ 200 { items: [...same shape, plus validator_address] } +``` + +**Implementation sketch** +- Cap `limit` at 200. +- Order DESC on `slashed_at`. + +**Acceptance** +- [ ] Empty → returns `[]`. +- [ ] After a slash event lands → entry visible. + +**Verification** +- Insert a row manually, curl endpoint. + +--- + +### TASK-012 — Slash on equivocation + +**Section:** chain +**Effort:** L +**Depends on:** TASK-011, TASK-008 +**Type:** edit + +**Goal** +If the same validator signs two different blocks at the same height (equivocation), that's a slash-able offense. Detect it and reduce stake. + +**Files** +- new: `backend/src/blockchain/equivocationDetector.ts` — exports `detectEquivocation(block: Block): Promise<{equivocated: bool, otherHash?: string}>`. +- edit: `backend/src/blockchain/Chain.ts:193` — invoke detector on every accepted block; on hit, write to `validator_slashes` and decrement stake. + +**Reuses** +- `chain.getBlockByHash`, validators table. + +**Implementation sketch** +- For each accepted block at height H, query if we already have ANOTHER hash at H signed by the same producer (in `blocks` table). +- If yes: insert into `validator_slashes` with evidence `{firstHash, secondHash, height}`. +- `UPDATE validators SET stake = stake - LEAST(stake, 10) WHERE address = $1`. +- Emit `validator_slashed` event. +- Slashing capped at zero (no negative stake). + +**Acceptance** +- [ ] Same producer at same height with different hashes → slash row inserted, stake reduced. +- [ ] Same producer at different heights → no slash. + +**Verification** +- Force two blocks at same height through addBlock with same producer; check `validator_slashes`. + +--- + +### TASK-013 — validators.stake column + weighted producer selection + +**Section:** chain +**Effort:** M +**Depends on:** TASK-311 (column exists) +**Type:** edit + +**Goal** +Combine TASK-311's column with TASK-010's hash-based rotation to make heavier-staked validators proposer more often. Stake-weighted lottery instead of uniform. + +**Files** +- edit: `backend/src/validators/ValidatorManager.ts:selectProducer` — replace mod-n with stake-weighted pick. + +**Reuses** +- `validatorOrder`, plus query `SELECT address, stake FROM validators`. + +**Implementation sketch** +- Cache stakes in memory; refresh every 100 blocks (cheap query). +- `cumulative = []`; build a running-sum array of stakes in validatorOrder order. +- `seed mod totalStake` → first index where `cumulative[i] > seed`. +- Same fallback to mod-n when stakes unavailable. + +**Acceptance** +- [ ] Validator with 2x stake produces ~2x the blocks over many rounds. +- [ ] Single-validator unchanged. + +**Verification** +- Unit test: 10k draws with stakes [1,2,3] yield ~16/33/50% distribution. + +--- + +### TASK-014 — Quorum weight by stake instead of head count + +**Section:** chain +**Effort:** M +**Depends on:** TASK-013 +**Type:** edit + +**Goal** +[ValidatorManager.getConsensus()](backend/src/validators/ValidatorManager.ts) currently counts head approvals: `Math.ceil(n * 2/3)`. With stake, threshold should be `Math.ceil(totalStake * 2/3)` and approvals tallied by stake. + +**Files** +- edit: `backend/src/validators/ValidatorManager.ts:getConsensus` — change tally + threshold logic. + +**Reuses** +- Cached stakes from TASK-013. + +**Implementation sketch** +- `const total = validators.reduce((s, v) => s + getStake(v), 0n)`. +- `const required = (total * 2n + 2n) / 3n` (BigInt ceiling-divide). +- Approval count by sum of approver stakes. +- Default-stake-1 still produces head-count behavior. + +**Acceptance** +- [ ] All-stake-1 quorum behaves identically to current. +- [ ] Validator with 51% stake can finalize alone. + +**Verification** +- Unit test with mixed stakes. + +--- + +### TASK-015 — Block timestamp drift check (>30s future = reject) + +**Section:** chain +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +[Block.isValid](backend/src/blockchain/Block.ts#L153) checks monotonicity vs parent but not absolute future-time. A producer with a fast clock could mint future blocks. Reject blocks with `timestamp > now + 30000`. + +**Files** +- edit: `backend/src/blockchain/Block.ts:153-171` — add a `now` parameter, default `Date.now()`, check drift. + +**Implementation sketch** +- New signature: `isValid(previousBlock?: Block, now: number = Date.now()): boolean`. +- Add: `if (this.header.timestamp > now + 30_000) return false;`. +- Update existing callers in BlockProducer / ForkManager to pass `Date.now()` or accept default. + +**Acceptance** +- [ ] Block with future timestamp 60s ahead → isValid returns false. +- [ ] Block with timestamp now+5s → still valid. + +**Verification** +- Unit test: synthesize a future-time block. + +--- + +### TASK-016 — Min block time enforcement (<2s after parent = reject) + +**Section:** chain +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +A producer with a backwards clock could mint sub-second blocks. Enforce a 2s minimum delta vs parent. + +**Files** +- edit: `backend/src/blockchain/Block.ts:isValid` — add `if (previousBlock && this.header.timestamp - previousBlock.header.timestamp < 2000) return false;`. + +**Implementation sketch** +- Replace existing strict `<=` check (line 166) with new `< 2000ms` rule. + +**Acceptance** +- [ ] Block 1000ms after parent → invalid. +- [ ] Block 5000ms after parent → valid. + +**Verification** +- Unit. + +--- + +### TASK-017 — Difficulty retarget every 100 blocks + +**Section:** chain +**Effort:** M +**Depends on:** none +**Type:** edit + +**Goal** +[DifficultyManager](backend/src/blockchain/Consensus.ts) has `adjustDifficulty` but it's never invoked. Periodically retarget based on observed block time vs the 10s target. + +**Files** +- edit: `backend/src/blockchain/Consensus.ts:DifficultyManager` — add `retarget(observedAvgMs: number, targetMs: number = 10_000): void`. +- edit: `backend/src/blockchain/BlockProducer.ts` — after every 100 blocks, compute observed avg over last 100 timestamps and call `retarget`. + +**Reuses** +- `chain.getRecentBlocks(100)` for the observed window. + +**Implementation sketch** +- `observedAvg = (blocks[99].ts - blocks[0].ts) / 99`. +- `factor = observedAvg / target`. +- New difficulty = clamp(`current * factor`, [1, 1e9]); adjust by at most 4× per retarget to avoid oscillation. + +**Acceptance** +- [ ] After 100 blocks at 5s avg, difficulty ~halves. +- [ ] Bounded swing (max 4× per period). + +**Verification** +- Run BlockProducer with synthetic timestamps; observe difficulty adjustment. + +--- + +### TASK-018 — Persist mempool to disk on shutdown, restore on boot + +**Section:** chain +**Effort:** M +**Depends on:** none +**Type:** edit + +**Goal** +Mempool sits in `pendingTransactions: Map`. Process restart drops it. Persist to disk on shutdown, restore on boot so users don't have to resubmit. + +**Files** +- edit: `backend/src/blockchain/TransactionPool.ts` — add `dumpToDisk(): Promise` and `restoreFromDisk(): Promise`. +- edit: `backend/src/api/server.ts` — call dumpToDisk in graceful-shutdown hook (already exists from prior work). +- edit: `backend/src/blockchain/TransactionPool.ts:initialize` — call restoreFromDisk after the existing init. + +**Reuses** +- `addTransaction` for re-validation on restore. +- `data/` directory. + +**Implementation sketch** +- Dump path: `data/mempool.json` containing array of tx JSON (with bigints stringified). +- On restore: read file, for each tx call `addTransaction` (which re-validates), count successful re-admits, log. +- File deleted after successful restore so partial restart doesn't double-replay. + +**Acceptance** +- [ ] Submit 5 txs, shutdown gracefully, restart, all 5 still pending. +- [ ] Crashed shutdown (no dump) → graceful no-op. + +**Verification** +- Manual restart sequence. + +--- + +### TASK-019 — Tx replacement-by-fee (same nonce, higher gasPrice replaces) + +**Section:** chain +**Effort:** M +**Depends on:** none +**Type:** edit + +**Goal** +Standard "RBF": if a sender submits a new tx with the same nonce as a pending one, accept only if `newGasPrice >= oldGasPrice * 1.1`. Otherwise reject as stale. + +**Files** +- edit: `backend/src/blockchain/TransactionPool.ts:addTransaction` (line 68) — pre-check for existing-nonce conflict. + +**Implementation sketch** +- On `addTransaction`, find existing tx in `pendingTransactions` with same `from` + `nonce`. +- If found: require `tx.gasPrice >= existing.gasPrice * 11n / 10n`. Else reject `replacement gas price too low`. +- If accepted: delete the old tx (and its DB row) before inserting the new. + +**Acceptance** +- [ ] Resubmit same nonce + same price → rejected. +- [ ] Resubmit same nonce + 11% higher price → old replaced. +- [ ] Resubmit same nonce + 9% higher price → rejected. + +**Verification** +- Three sequential submits with the conditions above. + +--- + +### TASK-020 — Mempool size cap 10k with lowest-gasPrice eviction + +**Section:** chain +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Unbounded mempool is a DoS vector. Cap at 10k. When full, evict the lowest-gasPrice pending tx to make room (assuming the new one is higher). + +**Files** +- edit: `backend/src/blockchain/TransactionPool.ts:addTransaction` — after validation, check size; evict if needed. + +**Implementation sketch** +- `MEMPOOL_MAX = 10000` (env override `MEMPOOL_MAX_SIZE`). +- If at cap: find min-gasPrice pending; if `newTx.gasPrice > min.gasPrice`, evict min, insert new. Else reject `mempool full and price too low`. + +**Acceptance** +- [ ] Filling to 10k + 1 with high-price tx → one evicted. +- [ ] Filling to 10k + 1 with too-low tx → reject. + +**Verification** +- Stress test. + +--- + +### TASK-021 — Pending tx TTL 1h + +**Section:** chain +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +[TransactionPool.clearExpired](backend/src/blockchain/TransactionPool.ts#L279) is a stub returning 0. Implement: drop pending txs older than 1h (configurable). + +**Files** +- edit: `backend/src/blockchain/TransactionPool.ts:clearExpired` — real implementation. +- edit: same — start a 1-min interval calling it. + +**Implementation sketch** +- Each pending tx already has a timestamp (`addedAt`); add the field if missing. +- `clearExpired(ageMs = 3_600_000)`: iterate, delete if `now - addedAt > ageMs`. +- Return count cleared. +- Log when count > 0. + +**Acceptance** +- [ ] Tx older than 1h disappears from pool. +- [ ] Tx within window stays. + +**Verification** +- Inject older `addedAt`, run clearExpired. + +--- + +### TASK-022 — Block size limit 1MB serialized + +**Section:** chain +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Currently only gas-bounded. A pathological block could have 10k tiny txs that fit gas but blow up serialized size. Add a 1MB cap. + +**Files** +- edit: `backend/src/blockchain/BlockProducer.ts:produceBlock` — track running serialized size of pushed txs; stop including more when ≥ 1MB. + +**Implementation sketch** +- After each tx accepted into validTxs: add `Buffer.byteLength(JSON.stringify(tx))`. +- Stop when ≥ 1_048_576. +- Log `[PRODUCER] block size cap reached at X bytes`. + +**Acceptance** +- [ ] Producing a block with > 1MB worth of txs caps at the limit. + +**Verification** +- Synthetic load test. + +--- + +### TASK-023 — logs_topic0_idx index migration + +**Section:** chain +**Effort:** S +**Depends on:** TASK-310 (logs_jsonb GIN exists) +**Type:** migration + +**Goal** +TASK-310 adds GIN over the full logs_json. Topic-0 (event signature) lookups dominate; add a more specific index to make them fast. + +**Files** +- new: `backend/src/database/migrations/0017_logs_topic0_idx.sql` + +**Migration SQL** +```sql +-- up: +CREATE INDEX IF NOT EXISTS idx_receipts_logs_topic0 + ON receipts USING GIN ((logs_jsonb -> 'topics' -> 0)); + +-- down: +DROP INDEX IF EXISTS idx_receipts_logs_topic0; +``` + +**Acceptance** +- [ ] Index visible. +- [ ] `EXPLAIN SELECT * FROM receipts WHERE logs_jsonb @> '[{"topics":["0xabc"]}]'` uses it. + +--- + +### TASK-024 — /api/logs?fromBlock=&toBlock=&address=&topic0= + +**Section:** chain +**Effort:** M +**Depends on:** TASK-023 +**Type:** new-file + +**Goal** +Standard log filtering endpoint matching `eth_getLogs` semantics. Filter by block range, contract address, and indexed topic0. + +**Files** +- new: `backend/src/api/logs-query.ts` — exports router. +- edit: `backend/src/api/server.ts` — mount at `/api/logs` (separate from existing `/api/logs` which is the agent log feed; check if collision — if so use `/api/chain/logs`). + +**API contract** +``` +GET /api/chain/logs?fromBlock=100&toBlock=200&address=0xabc&topic0=0xdef&limit=100 +→ 200 { logs: [ { address, topics, data, blockNumber, transactionHash, logIndex } ] } +``` + +**Implementation sketch** +- All filter params optional except limit (default 100, max 1000). +- Build dynamic SQL using indexed columns first. +- JSONB containment for topic0; address filter via `logs_jsonb @> '[{"address":"..."}]'`. +- Block range via `WHERE block_number BETWEEN $f AND $t`. + +**Acceptance** +- [ ] No filters: returns most recent 100 logs. +- [ ] address filter: only logs from that contract. +- [ ] topic0 filter: only matching event signatures. + +**Verification** +- Curl with various filter combinations after seed data. + +--- + +### TASK-025 — /api/logs/bloom-check helper + +**Section:** chain +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Block headers carry a bloom filter ([Receipt.ts:logsBloom](backend/src/blockchain/TransactionReceipt.ts)). Clients can pre-filter blocks by checking the bloom before fetching full receipts. Expose the membership check. + +**Files** +- new endpoint in `backend/src/api/logs-query.ts`. + +**Reuses** +- `bloomContains` from [TransactionReceipt.ts:75-88](backend/src/blockchain/TransactionReceipt.ts#L75-L88). + +**API contract** +``` +GET /api/chain/logs/bloom-check?height=100&item=0xabc +→ 200 { mightContain: true|false } +``` + +**Implementation sketch** +- Look up the block by height, read its `logsBloom` (combined from all receipts at calculateReceiptsRoot time — may need to also store on block header). +- Return `bloomContains(bloomHex, item)`. + +**Acceptance** +- [ ] Definitely-not-present item → false. +- [ ] Present item → true. +- [ ] Note false-positive rate inherent to bloom. + +**Verification** +- Insert a known log, check both true and false cases. + +--- + +### TASK-026 — Block uncles tracking + +**Section:** chain +**Effort:** M +**Depends on:** TASK-007 +**Type:** edit + +**Goal** +Track orphaned-but-valid blocks (uncles) for the GHOST fork-choice rule (TASK-027). Currently orphans are pruned. Keep them in a side table. + +**Files** +- new: `backend/src/database/migrations/0018_uncles.sql` — `uncles(block_hash, parent_hash, height, producer, included_in_block_hash, found_at)`. +- edit: `backend/src/blockchain/Chain.ts:handleReorg` — when reverting a block, record it as an uncle of the new canonical block. + +**Implementation sketch** +- On reorg, the orphaned blocks become uncles of the same-height canonical block. +- Insert one row per orphaned block. +- `included_in_block_hash` = canonical at that height. + +**Acceptance** +- [ ] After reorg, `uncles` table has rows for the orphaned chain. + +**Verification** +- Force reorg, query uncles. + +--- + +### TASK-027 — GHOST fork-choice weighting + +**Section:** chain +**Effort:** L +**Depends on:** TASK-026 +**Type:** edit + +**Goal** +[ForkManager.addBlock](backend/src/blockchain/Consensus.ts) currently picks longest chain. GHOST: weight = (canonical depth) + (uncle count under this subtree). The heaviest subtree wins, not just the longest. + +**Files** +- edit: `backend/src/blockchain/Consensus.ts:ForkManager.addBlock` — replace longest-chain logic with subtree-weight logic. + +**Reuses** +- `uncles` table from TASK-026. + +**Implementation sketch** +- For each fork tip, compute weight = blocks-in-fork + uncles-in-fork. +- Switch canonical iff candidate weight > current weight (strict). +- Tie-break by lower-hash for determinism. + +**Acceptance** +- [ ] Two equal-length forks, the one with more uncles becomes canonical. + +**Verification** +- Synthetic test with engineered uncle counts. + +--- + +### TASK-028 — /api/chain/export?from=&to= NDJSON stream + +**Section:** chain +**Effort:** M +**Depends on:** none +**Type:** edit + +**Goal** +Operators need a fast way to bulk-export a height range for backup, analytics, or seeding a new node. + +**Files** +- new endpoint in `backend/src/api/server.ts` (or a new chain-tools router). + +**API contract** +``` +GET /api/chain/export?from=0&to=100000 +→ 200 (text/plain; application/x-ndjson) + {"type":"block", ...block.toJSON()} + {"type":"receipt", ...receipt} + ... line-delimited +``` + +**Implementation sketch** +- Use Node `Readable` stream; write one JSON-line per block, then per receipt. +- Don't buffer — write+flush as you go. +- Cap range to 1M blocks (way bigger than current chain but still bounded). + +**Acceptance** +- [ ] Streams without buffering full result. +- [ ] Each line valid JSON parsable independently. + +**Verification** +- `curl /api/chain/export?from=0&to=10 | jq -c` parses cleanly. + +--- + +### TASK-029 — backend/scripts/import-chain.ts + +**Section:** chain +**Effort:** M +**Depends on:** TASK-001, TASK-028 +**Type:** script + +**Goal** +The inverse of TASK-028: read NDJSON, fromJSON each block, addBlock, plus apply receipts via storeReceipt. + +**Files** +- new: `backend/scripts/import-chain.ts` +- edit: `backend/package.json:scripts` — `"chain:import": "ts-node backend/scripts/import-chain.ts"`. + +**Implementation sketch** +- Args: `npm run chain:import -- --file path.ndjson --from-stdin`. +- Stream-parse each line, dispatch by type. +- On addBlock failure (parent unknown), buffer + retry once full file read; if still failing, exit 1. + +**Acceptance** +- [ ] Import + export round-trip preserves all blocks + receipts. + +**Verification** +- Export from one DB, import to fresh DB, compare counts. + +--- + +### TASK-030 — Genesis parameterization via genesis.json + +**Section:** chain +**Effort:** M +**Depends on:** none +**Type:** edit + +**Goal** +[Chain.ts](backend/src/blockchain/Chain.ts) hardcodes genesis time and producer. Move to a tracked `genesis.json` so testnets/forks can use different params without code change. + +**Files** +- new: `backend/genesis.json` — `{ chainId: "hermes-mainnet-1", genesisTimestamp: 1776067200000, genesisProducer: "...", initialAllocations: [{address, balance}, ...] }`. +- edit: `backend/src/blockchain/Chain.ts` — read genesis.json at boot, override defaults. +- edit: `backend/src/blockchain/StateManager.ts:initialize` — apply initialAllocations if state empty. + +**Implementation sketch** +- Path resolved relative to repo root or backend/ dir; env `HERMES_GENESIS_FILE` overrides. +- Strict JSON shape; throw on malformed. + +**Acceptance** +- [ ] Default genesis.json reproduces current behavior. +- [ ] Custom genesis with different chainId boots a different chain. + +**Verification** +- Cold boot with default genesis, observe height-0 state matches expectation. + +--- + +### TASK-031 — Genesis hash verification at boot + +**Section:** chain +**Effort:** S +**Depends on:** TASK-030 +**Type:** edit + +**Goal** +On boot, the chain's genesis block hash must match `genesisHash` in genesis.json. Mismatch = wrong chain config; halt boot rather than corrupt. + +**Files** +- edit: `backend/src/blockchain/Chain.ts:initialize` — compute genesis hash from in-memory genesis block, compare to genesis.json field. + +**Implementation sketch** +- `genesis.json` gets a `genesisHash` field (deterministic from other fields). +- After loading/creating genesis: assert hash equality. Throw with both hashes printed on mismatch. + +**Acceptance** +- [ ] Matching → boots silently. +- [ ] Mismatched → throws on boot, process exits. + +**Verification** +- Tweak genesis.json hash, observe boot failure. + +--- + +### TASK-032 — Validator handoff record on rotation + +**Section:** chain +**Effort:** S +**Depends on:** TASK-013 +**Type:** edit + +**Goal** +When the producer changes between blocks (rotation), record the handoff in `consensus_events` for an audit trail. + +**Files** +- edit: `backend/src/blockchain/BlockProducer.ts` — after block accepted, if `block.producer != lastBlock.producer`, INSERT a `consensus_events` row. + +**Reuses** +- Existing `consensus_events` table. + +**Implementation sketch** +- `event_type = 'producer_handoff'`, metadata `{ from, to, height }`. + +**Acceptance** +- [ ] Multi-validator chain shows handoff rows in consensus_events. + +**Verification** +- After 10 blocks with rotation, query event_type='producer_handoff'. + +--- + +### TASK-033 — Per-block VRF beacon for randomness + +**Section:** chain +**Effort:** M +**Depends on:** none +**Type:** edit + +**Goal** +Contracts (TASK-076 BLOCKNUMBER/TIMESTAMP/DIFFICULTY) and validators want a tamper-resistant per-block randomness beacon. Use the producer's signature over `(prevHash || height)` as the beacon. + +**Files** +- new: `backend/src/blockchain/Beacon.ts` — `computeBeacon(block, producerKeypair): string`. +- edit: `backend/src/blockchain/Block.ts:BlockHeader` — add `beacon?: string` field. +- edit: BlockProducer to attach beacon at production time. + +**Reuses** +- `sign()` from [Crypto.ts:97](backend/src/blockchain/Crypto.ts#L97). + +**Implementation sketch** +- `beacon = sign(`${prevHash}:${height}`, producerPriv)` — verifiable by anyone. +- VM consumes `beacon` for randomness opcode. + +**Acceptance** +- [ ] Beacon present on all newly-produced blocks. +- [ ] Verifiable: `verify(message, beacon, producerPub) === true`. + +**Verification** +- Inspect `block.header.beacon`. + +--- + +### TASK-034 — State pruning for zero/dead accounts + +**Section:** chain +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Accounts that have been emptied (balance=0, nonce=0, no code, no storage) bloat the state. Periodically prune them. + +**Files** +- new: `backend/src/blockchain/statePruner.ts` +- edit: `backend/src/blockchain/StateManager.ts` — invoke pruner every 1000 blocks. + +**Implementation sketch** +- `DELETE FROM accounts WHERE balance = '0' AND nonce = 0 AND (code IS NULL OR code = '') AND storage IS NULL OR storage = '{}'`. +- Log count pruned. +- Skip if it would touch active recent accounts (last 100 blocks of activity) — be conservative. + +**Acceptance** +- [ ] Empty accounts vanish after pruning sweep. +- [ ] Active accounts untouched. + +**Verification** +- Manually empty an account, run pruner, observe. + +--- + +### TASK-035 — State snapshot every 10k blocks + +**Section:** chain +**Effort:** M +**Depends on:** TASK-316 (state_snapshots table) +**Type:** edit + +**Goal** +Capture full account + contract_storage state every 10k blocks for fast-sync (TASK-036). + +**Files** +- new: `backend/src/blockchain/snapshotWriter.ts` +- edit: BlockProducer to invoke after `commitBlock` when `height % 10000 === 0`. + +**Implementation sketch** +- Snapshot blob = gzipped JSON of all rows in `accounts` + `contract_storage` at this height. +- Write to `state_snapshots` table. +- Skip if snapshot already exists for that height (idempotent). + +**Acceptance** +- [ ] After 10k blocks, exactly one row in `state_snapshots`. +- [ ] Blob inflates to a JSON object with `accounts[]` and `storage[]`. + +**Verification** +- Run to height 10000, query state_snapshots. + +--- + +### TASK-036 — /api/mesh/snapshot/:height fast-sync + +**Section:** chain +**Effort:** S +**Depends on:** TASK-035 +**Type:** edit + +**Goal** +Serve the snapshot blob for a given height so a fresh peer can hydrate without replaying 10k blocks. + +**Files** +- edit: `backend/src/network/api.ts` — `GET /api/mesh/snapshot/:height`. + +**API contract** +``` +GET /api/mesh/snapshot/10000 +→ 200 (Content-Type: application/octet-stream) + +→ 404 { error: 'snapshot not found' } +``` + +**Acceptance** +- [ ] Returns blob for known snapshot heights. +- [ ] 404 otherwise. + +**Verification** +- `curl /api/mesh/snapshot/10000 | gunzip | jq '.accounts | length'`. + +--- + +### TASK-037 — Tx fee distribution: 80% producer, 20% burned + +**Section:** chain +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Currently `applyBlockReward` adds the fixed BLOCK_REWARD. Tx fees are not separately accounted. Split them: 80% credited to producer (along with reward), 20% burned (subtracted from supply). + +**Files** +- edit: `backend/src/blockchain/StateManager.ts:applyBlockReward` — accept additional `feeTotal: bigint`; credit producer with `reward + (fee * 80n / 100n)`; track burn separately in chainState. +- edit: `backend/src/blockchain/BlockProducer.ts` — accumulate per-tx fees (`tx.gasPrice * gasUsed`), pass into applyBlockReward. +- edit: `chainState` — add `incrementBurn(amount: bigint)`, `getTotalBurn(): bigint`. + +**Acceptance** +- [ ] Producer receives reward + 80% of fees. +- [ ] Total burn tracked. + +**Verification** +- After known txs, compare producer balance increment to expected. + +--- + +### TASK-038 — Burn counter on chain stats + +**Section:** chain +**Effort:** S +**Depends on:** TASK-037 +**Type:** edit + +**Goal** +Surface the cumulative burn in `/api/status` and a new `/api/chain/burn` for easy querying. + +**Files** +- edit: `backend/src/api/server.ts:/api/status` — add `totalBurn` field. +- new endpoint `GET /api/chain/burn` → `{ totalBurn: '', burnRatePerBlock: '' }`. + +**Acceptance** +- [ ] After fee-bearing blocks, totalBurn > 0. + +**Verification** +- `curl /api/chain/burn`. + +--- + +### TASK-039 — Per-validator block reward via env + +**Section:** chain +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +[BlockProducer.BLOCK_REWARD](backend/src/blockchain/BlockProducer.ts#L20) is a hardcoded `10e18`. Allow override via env per environment (testnet vs mainnet). + +**Files** +- edit: `backend/src/blockchain/BlockProducer.ts:18-20` — read `HERMES_BLOCK_REWARD_WEI` env. + +**Implementation sketch** +- Default unchanged. +- Parse as bigint. Throw on non-numeric. + +**Acceptance** +- [ ] No env: 10e18. +- [ ] `HERMES_BLOCK_REWARD_WEI=5000000000000000000` → 5e18. + +**Verification** +- Boot with env, watch first block reward. + +--- + +### TASK-040 — Coinbase tx representation in receipts + +**Section:** chain +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Block reward is a state credit but no tx represents it. Block explorers can't show "where did this 10 OPEN come from?" Add a synthetic coinbase tx with hash `coinbase:${blockHeight}` and a receipt. + +**Files** +- edit: `backend/src/blockchain/BlockProducer.ts` — after applyBlockReward, create + storeReceipt for a synthetic coinbase tx. + +**Implementation sketch** +- Synthetic tx: `from='0x0000...coinbase'`, `to=producer.address`, `value=BLOCK_REWARD + producer fee share`, `hash=sha256('coinbase:'+blockHeight)`, `data='coinbase'`. +- Receipt at index 0; shifts other tx indices. +- Don't add to TransactionPool (it's already executed). + +**Acceptance** +- [ ] Each block has a coinbase receipt. +- [ ] Sum of receipt values = BLOCK_REWARD + 80% fees. + +**Verification** +- Curl receipts for a recent block. + +--- + +### TASK-041 — Migration: index blocks(hash) + +**Section:** chain +**Effort:** S +**Depends on:** none +**Type:** migration + +**Goal** +`getBlockByHash` performs a full table scan without an index on the hash column. Add it. + +**Files** +- new: `backend/src/database/migrations/0019_blocks_hash_idx.sql` + +**Migration SQL** +```sql +-- up: +CREATE INDEX IF NOT EXISTS idx_blocks_hash ON blocks(hash); + +-- down: +DROP INDEX IF EXISTS idx_blocks_hash; +``` + +**Acceptance** +- [ ] Index visible. + +**Verification** +- `EXPLAIN SELECT * FROM blocks WHERE hash = 'abc';` uses it. + +--- + +### TASK-042 — Tx (from_address, nonce) compound index + +**Section:** chain +**Effort:** S +**Depends on:** TASK-307 (already in section 08) +**Type:** migration + +**Goal** +This task is satisfied by TASK-307 in section 08 (`backend/src/database/migrations/0003_tx_from_nonce_idx.sql`). No additional migration here — note dependency only. + +**Files** +- (covered by TASK-307) + +**Acceptance** +- [ ] When section 08 lands, this task closes automatically. + +**Verification** +- See TASK-307. + +--- + +### TASK-043 — Account-history rebuild script + +**Section:** chain +**Effort:** M +**Depends on:** none +**Type:** script + +**Goal** +If state_change events are lost (e.g. fresh DB) but blocks/receipts survive, rebuild per-account history by replaying all txs. + +**Files** +- new: `backend/scripts/rebuild-account-history.ts` +- edit: `backend/package.json:scripts` — `"chain:rebuild-history": "..."`. + +**Implementation sketch** +- Iterate blocks 0..N. +- For each tx: apply to a fresh in-memory state, log state_change. +- After full replay, write summary per address: total_in, total_out, tx_count. + +**Acceptance** +- [ ] Output matches a from-scratch chain. +- [ ] Idempotent. + +**Verification** +- Run script, sample one account, compare to live state. + +--- + +### TASK-044 — CLI: npm run verify-chain + +**Section:** chain +**Effort:** M +**Depends on:** none +**Type:** script + +**Goal** +Sanity check: walk the chain from genesis, verify (parent linkage, block hashes, signatures, state-root reproducibility). Operators run before trusting a chain copy. + +**Files** +- new: `backend/scripts/verify-chain.ts` +- edit: `backend/package.json:scripts` — `"verify-chain": "..."`. + +**Implementation sketch** +- For each block: `Block.fromJSON(toJSON()).header.hash === current.hash`. +- For each tx: `verifyTransactionSignature` returns true. +- For each receipt: matches `calculateReceiptsRoot`. +- Log pass/fail per block; final summary. +- Exit 1 on any failure. + +**Acceptance** +- [ ] Healthy chain → all pass, exit 0. +- [ ] Tampered block → fail, exit 1. + +**Verification** +- Run on dev chain. + +--- + +### TASK-045 — State root mismatch alarm event + +**Section:** chain +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +If `commitBlock`'s computed state root doesn't match the value the producer baked into the block header (e.g. desync between producer and validator), emit a loud alarm. + +**Files** +- edit: `backend/src/blockchain/Chain.ts:addBlock` — after applying tx to local state, compare `stateManager.calculateStateRoot()` vs `block.header.stateRoot`. On mismatch, emit `state_root_mismatch` event. + +**Implementation sketch** +- Don't reject the block (it may be ours); just emit the event for monitoring. +- Payload: `{ blockHeight, blockHash, expected, actual, producer }`. + +**Acceptance** +- [ ] Event fires on mismatch. +- [ ] Doesn't fire on match. + +**Verification** +- Inject a mismatched root manually, observe event. + +--- + +### TASK-046 — Receipt root verification at sync + +**Section:** chain +**Effort:** S +**Depends on:** TASK-007 +**Type:** edit + +**Goal** +On gossip-applied blocks, recompute receipts root from the block's receipts and compare to header. Reject mismatches as malformed. + +**Files** +- edit: `backend/src/network/api.ts:/api/mesh/block` — after fromJSON, before addBlock, recompute receipts root, compare. Reject if mismatch. + +**Acceptance** +- [ ] Block with valid receipts root → accepted. +- [ ] Tampered receipts → rejected. + +**Verification** +- Round-trip with intact, then tampered receipts. + +--- + +### TASK-047 — SSE /api/logs/subscribe?topic0= + +**Section:** chain +**Effort:** M +**Depends on:** none +**Type:** new-file + +**Goal** +Real-time log stream filtered by topic0. Clients (HUD, contracts, oracles) subscribe and receive new logs as blocks land. + +**Files** +- new: `backend/src/api/log-stream.ts` +- mount in server.ts at `/api/chain/logs/subscribe`. + +**Reuses** +- `eventBus.on('block_produced', cb)` — extract logs from each block's receipts. +- SSE pattern from [server.ts:971-1057](backend/src/api/server.ts#L971-L1057). + +**API contract** +``` +GET /api/chain/logs/subscribe?topic0=0xabc +→ Server-Sent Events + data: { address, topics, data, blockNumber, transactionHash } +``` + +**Acceptance** +- [ ] Filter by topic0 works. +- [ ] Reconnect resumes from `Last-Event-ID`. + +**Verification** +- `curl -N /api/chain/logs/subscribe`, produce a log-bearing block, observe. + +--- + +### TASK-048 — SSE /api/mempool/subscribe + +**Section:** chain +**Effort:** S +**Depends on:** none +**Type:** new-file + +**Goal** +Stream `transaction_added` and `transaction_removed` events for live mempool view. + +**Files** +- new: `backend/src/api/mempool-stream.ts` +- mount at `/api/mempool/subscribe`. + +**Reuses** +- `eventBus.on('transaction_added')` and a new `transaction_removed` event emitted by `removeTransactions`. + +**Acceptance** +- [ ] Submit tx → event arrives. +- [ ] Tx mined → removal event. + +**Verification** +- `curl -N` while submitting txs. + +--- + +### TASK-049 — SSE /api/forks/subscribe + +**Section:** chain +**Effort:** S +**Depends on:** TASK-026 +**Type:** new-file + +**Goal** +Stream `chain_reorg` events with depth + payload. + +**Files** +- new: `backend/src/api/fork-stream.ts` +- mount at `/api/forks/subscribe`. + +**Acceptance** +- [ ] Reorg → event delivered. + +**Verification** +- Force reorg, watch stream. + +--- + +### TASK-050 — Per-block aggregate gas price stats + +**Section:** chain +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Each block's receipts include `gasUsed` per tx and `gasPrice` per tx. Aggregate to `{ p50, p95, max, mean }` for charting (TASK-226). + +**Files** +- new: `backend/src/api/chain-stats.ts` — `GET /api/chain/gas-stats?height=N` or `?fromHeight=&toHeight=`. + +**Implementation sketch** +- For a single block: aggregate over receipts. +- For a range: same per-block, return array. +- Sort gasPrice values, pick percentiles. + +**Acceptance** +- [ ] Returns numeric stats. + +**Verification** +- Curl after a couple blocks. + +--- + +### TASK-051 — /api/chain/tps?window=60 + +**Section:** chain +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +[chain.getRecentTps](backend/src/blockchain/Chain.ts#L344) already exists. Just expose it. + +**Files** +- edit: `backend/src/api/server.ts` or `chain-stats.ts` — `GET /api/chain/tps?window=60`. + +**Reuses** +- `chain.getRecentTps(windowSec)`. + +**API contract** +``` +GET /api/chain/tps?window=60 +→ 200 { tps: 3.4, window_sec: 60 } +``` + +**Acceptance** +- [ ] Default window 60. +- [ ] Range param honored. + +**Verification** +- Curl. + +--- + +### TASK-052 — /api/chain/block-times histogram + +**Section:** chain +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Distribution of block times (parent.ts → child.ts) over last N blocks. + +**Files** +- new endpoint in `backend/src/api/chain-stats.ts`. + +**API contract** +``` +GET /api/chain/block-times?limit=1000 +→ 200 { buckets: [...], counts: [...], mean: 10.2, p95: 13.0 } +``` + +**Implementation sketch** +- Pull last N blocks, compute deltas, bucket. + +**Acceptance** +- [ ] Returns histogram. + +**Verification** +- Curl after 100+ blocks. + +--- + +### TASK-053 — Validator uptime metric + +**Section:** chain +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Per-validator uptime = (blocks produced when scheduled) / (blocks scheduled). Surface via existing validators endpoint. + +**Files** +- edit: `backend/src/api/server.ts:/api/validators` — extend response with `uptime: number` (0..1). + +**Implementation sketch** +- Scheduled = floor(chainHeight / numValidators) per validator (rough; precise needs producer rotation accounting). +- Actual = `blocks_produced` from `validators` table. +- `uptime = actual / scheduled`. + +**Acceptance** +- [ ] Validator with 100% production → uptime = 1. + +**Verification** +- Curl. + +--- + +### TASK-054 — Mempool depth chart endpoint + +**Section:** chain +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Time-series of mempool size for the HUD chart (TASK-219). + +**Files** +- new: `backend/src/blockchain/mempoolHistory.ts` — sample every 10s, keep ring buffer of last 360 samples (1h). +- new endpoint `GET /api/mempool/history`. + +**Acceptance** +- [ ] Returns last hour of mempool depth. + +**Verification** +- Curl. + +--- + +### TASK-055 — /api/tx/simulate (VM dry-run) + +**Section:** chain +**Effort:** M +**Depends on:** none +**Type:** edit + +**Goal** +Run a tx through the VM without committing. Returns the would-be receipt + state changes. + +**Files** +- new endpoint `POST /api/tx/simulate` in `backend/src/api/server.ts`. + +**Reuses** +- `interpreter.execute()` from [vm/Interpreter.ts](backend/src/vm/Interpreter.ts). +- `parseVmProgram`. + +**API contract** +``` +POST /api/tx/simulate +body: { from, to, value, gasPrice, gasLimit, nonce, data } +→ 200 { gasUsed: '...', logs: [...], status: 'success'|'revert', error?: string } +``` + +**Implementation sketch** +- For VM tx: parse program, run interpreter against a snapshot of current state. +- For plain transfer: just check balance/nonce, return predicted gasUsed = 21000. + +**Acceptance** +- [ ] Simulation matches actual on-chain execution for VM tx. + +**Verification** +- Submit same tx via simulate then via /api/transactions; compare. + +--- + +### TASK-056 — /api/tx/estimate-gas + +**Section:** chain +**Effort:** S +**Depends on:** TASK-055 +**Type:** edit + +**Goal** +Subset of simulate that returns just the gasUsed estimate. + +**Files** +- new endpoint `POST /api/tx/estimate-gas`. + +**Implementation sketch** +- Wrap TASK-055 internals; return only `{ gasEstimate: '' }`. + +**Acceptance** +- [ ] Returns numeric estimate. + +**Verification** +- Curl with a sample VM tx. + +--- + +### TASK-057 — /api/account/:addr/next-nonce + +**Section:** chain +**Effort:** S +**Depends on:** TASK-307 (compound index) +**Type:** edit + +**Goal** +`next_nonce = max(chain_nonce, max_pending_nonce) + 1`. Wallets need this to avoid nonce conflicts. + +**Files** +- new endpoint in `backend/src/api/server.ts`. + +**Reuses** +- `stateManager.getNonce(addr)`, query `MAX(nonce)` over pending. + +**API contract** +``` +GET /api/account/:addr/next-nonce +→ 200 { address, nextNonce: 42 } +``` + +**Acceptance** +- [ ] No pending → returns chain_nonce + 1. +- [ ] Pending tx → returns max(pending) + 1. + +**Verification** +- Curl after submitting a few pending. + +--- + +### TASK-058 — /api/account/:addr/history paginated + +**Section:** chain +**Effort:** M +**Depends on:** TASK-307 +**Type:** edit + +**Goal** +Per-account tx history with cursor pagination. + +**Files** +- new endpoint in `backend/src/api/server.ts`. + +**API contract** +``` +GET /api/account/:addr/history?cursor=&limit=50 +→ 200 { items: [...txs...], next_cursor: "..." | null } +``` + +**Implementation sketch** +- Cursor = base64-encoded `block_height:tx_index` of last item. +- Query `WHERE (from_address = $1 OR to_address = $1) AND (block_height, tx_index) < (cursor) ORDER BY block_height DESC, tx_index DESC LIMIT $2`. + +**Acceptance** +- [ ] First page returns latest. +- [ ] Cursor walks back. + +**Verification** +- Walk through pages. + +--- + +### TASK-059 — /api/validator/:addr/blocks paginated + +**Section:** chain +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +List blocks produced by a specific validator, paginated. + +**Files** +- new endpoint. + +**API contract** +``` +GET /api/validator/:addr/blocks?cursor=&limit=50 +→ 200 { items: [{height, hash, timestamp, transactionCount}], next_cursor } +``` + +**Acceptance** +- [ ] Returns blocks where producer = addr. + +**Verification** +- Curl for known validator. + +--- + +### TASK-060 — /api/chain/reorgs (last 50) + +**Section:** chain +**Effort:** S +**Depends on:** none +**Type:** new-file + +**Goal** +Persist reorg events (currently only emitted on event bus) to a `reorg_log` table; expose recent ones. + +**Files** +- new: `backend/src/database/migrations/0020_reorg_log.sql` +- new: handler in `backend/src/api/server.ts` — `GET /api/chain/reorgs?limit=50`. +- edit: `backend/src/blockchain/Chain.ts:handleReorg` — INSERT a row per reorg. + +**Migration SQL** +```sql +-- up: +CREATE TABLE IF NOT EXISTS reorg_log ( + id BIGSERIAL PRIMARY KEY, + occurred_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + depth INTEGER NOT NULL, + orphaned_count INTEGER NOT NULL, + added_count INTEGER NOT NULL, + new_height BIGINT NOT NULL, + common_ancestor_height BIGINT NOT NULL, + metadata_json TEXT +); +CREATE INDEX IF NOT EXISTS idx_reorg_log_occurred + ON reorg_log(occurred_at DESC); + +-- down: +DROP INDEX IF EXISTS idx_reorg_log_occurred; +DROP TABLE IF EXISTS reorg_log; +``` + +**Acceptance** +- [ ] Reorg records inserted. +- [ ] Endpoint returns recent. + +**Verification** +- Force reorg, curl endpoint. + +--- + +## Summary + +60 tasks: 38 small, 18 medium, 4 large. Mix of new endpoints (~20), edits to existing chain code (~25), new modules (~10), and a handful of migrations + scripts. From cdd9ab8f3391caeab538a69cb930acb20c52f253 Mon Sep 17 00:00:00 2001 From: hermes agent Date: Tue, 28 Apr 2026 01:47:30 +0400 Subject: [PATCH 03/96] =?UTF-8?q?docs(backlog):=20detailed=20specs=20for?= =?UTF-8?q?=20section=2002=20=E2=80=94=20VM=20(TASK-061..105)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 45 task specs extending the JSON-op interpreter into a usable contract VM: arithmetic (MUL/DIV/MOD), comparisons (EQ/LT/GT), bitwise (AND/OR/NOT), control flow (JUMP/JUMPI/JUMPDEST), storage (SLOAD/SSTORE persistence, cold/warm pricing, zero-set refund), cross-contract CALL/STATICCALL/RETURN + returndata buffer, calldata + caller/origin/value/balance/block-context opcodes, hashing precompiles (SHA256/KECCAK), ECRECOVER (Ed25519 variant), contract deployment (CREATE + address derivation), code-loaded execution path in BlockProducer, trace + gas-profile endpoints, SELFDESTRUCT, memory model (MLOAD/MSTORE/MSTORE8 + expansion gas), stack + call depth limits, REVERT with returndata, EVENT opcode + ABI registry, source verifier, disasm + storage browser endpoints, 30-fixture VM unit test suite, /docs/vm spec page, tiny DSL compiler stub, four worked sample contracts. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/backlog/queue/02-vm.md | 1230 +++++++++++++++++++++++++++++++++++ 1 file changed, 1230 insertions(+) create mode 100644 docs/backlog/queue/02-vm.md diff --git a/docs/backlog/queue/02-vm.md b/docs/backlog/queue/02-vm.md new file mode 100644 index 00000000..d6cf7555 --- /dev/null +++ b/docs/backlog/queue/02-vm.md @@ -0,0 +1,1230 @@ +# Section 02 — VM Specs (TASK-061..105) + +45 tasks. Extends the JSON-op interpreter from [Interpreter.ts](backend/src/vm/Interpreter.ts) into a usable contract VM: arithmetic + comparisons + control flow + storage + cross-contract calls + events + memory + deployment + tooling. Targets correctness over performance. + +**Preconditions used throughout this section:** +- VM core: [Interpreter.ts](backend/src/vm/Interpreter.ts) — `execute(ops, gasLimit, ctx)` returns `{status, gasUsed, logs, storage, error?}`. Op shape `{ op: 'NAME', args?: any }`. +- Gas: [GasMeter.ts](backend/src/vm/GasMeter.ts) — `GAS_COSTS` table, `meter.charge(amount): bool`, `logGasCost(bytes)`. +- Re-export: [vm/index.ts](backend/src/vm/index.ts). +- Block dispatch: [BlockProducer.ts:115-160](backend/src/blockchain/BlockProducer.ts#L115-L160) — VM tx detection via `parseVmProgram(tx.data)`. +- Receipts: [TransactionReceipt.ts:Log](backend/src/blockchain/TransactionReceipt.ts) interface. +- Storage table from TASK-314: `contract_storage(contract_address, storage_key, storage_value, updated_at_block)`. +- Code table from TASK-313: `contract_code(address, code_hash, bytecode, deployed_at_block, deployed_by)`. + +--- + +### TASK-061 — MUL opcode + cost + +**Section:** vm +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Multiply top two stack items, push result. Standard arithmetic. + +**Files** +- edit: [Interpreter.ts](backend/src/vm/Interpreter.ts) — add `MUL` case. +- edit: [GasMeter.ts](backend/src/vm/GasMeter.ts) — `MUL: 5n` in `GAS_COSTS`. + +**Implementation sketch** +- `case 'MUL': charge(GAS_COSTS.MUL); a = pop, b = pop; push(a * b);`. +- Type widening: cast both to `BigInt` so overflow doesn't lose precision; final push as Number if safe. + +**Acceptance** +- [ ] `[PUSH 3, PUSH 4, MUL]` → top of stack = 12. +- [ ] Out-of-gas at MUL → revert. + +**Verification** +- Add to VM unit fixtures (TASK-102). + +--- + +### TASK-062 — DIV opcode + +**Section:** vm +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Integer division. `b = pop; a = pop; push(a / b)`. Divide-by-zero → revert (not panic). + +**Files** +- edit: [Interpreter.ts](backend/src/vm/Interpreter.ts) — add `DIV` case. +- edit: [GasMeter.ts](backend/src/vm/GasMeter.ts) — `DIV: 5n`. + +**Implementation sketch** +- Charge gas before pop. +- If `b === 0`, return `{status:'revert', error: 'division by zero'}`. +- Floor-division semantics for negatives. + +**Acceptance** +- [ ] `[PUSH 10, PUSH 3, DIV]` → 3. +- [ ] `[PUSH 1, PUSH 0, DIV]` → revert with reason. + +**Verification** +- Unit fixture. + +--- + +### TASK-063 — MOD opcode + +**Section:** vm +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Modulo. Same divide-by-zero handling as DIV. + +**Files** +- edit: Interpreter.ts + GasMeter.ts. + +**Implementation sketch** +- `MOD: 5n`. `push(a % b)`. + +**Acceptance** +- [ ] `[PUSH 10, PUSH 3, MOD]` → 1. + +**Verification** +- Unit. + +--- + +### TASK-064 — EQ / LT / GT comparison ops + +**Section:** vm +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Three comparison opcodes. Push 1 for true, 0 for false. Standard for downstream JUMPI. + +**Files** +- edit: Interpreter.ts (3 cases) + GasMeter.ts (`EQ/LT/GT: 3n`). + +**Implementation sketch** +- For each: pop two, compare, push 0 or 1. +- LT and GT compare numerically (cast to bigint). + +**Acceptance** +- [ ] `[PUSH 5, PUSH 5, EQ]` → 1. +- [ ] `[PUSH 3, PUSH 5, LT]` → 1. + +**Verification** +- Unit. + +--- + +### TASK-065 — AND / OR / NOT bitwise ops + +**Section:** vm +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Bitwise booleans for flag manipulation. + +**Files** +- edit: Interpreter.ts + GasMeter.ts. + +**Implementation sketch** +- `AND: 3n`, `OR: 3n`, `NOT: 3n`. +- Operate on bigints via `&`, `|`, `~` semantics (NOT inverts low 256 bits). + +**Acceptance** +- [ ] Truth-table tests pass. + +**Verification** +- Unit. + +--- + +### TASK-066 — JUMP / JUMPI control flow + +**Section:** vm +**Effort:** M +**Depends on:** TASK-067 +**Type:** edit + +**Goal** +Unconditional and conditional jumps. Branch to op index N if condition is non-zero (JUMPI) or always (JUMP). + +**Files** +- edit: Interpreter.ts — add JUMP, JUMPI handlers; refactor execute() loop to use a `pc` (program counter) variable. + +**Reuses** +- JUMPDEST validation from TASK-067. + +**Implementation sketch** +- Replace the `for (i; ...)` with `let pc = 0; while (pc < ops.length)`. +- `JUMP { args: [target] }`: `pc = target` (must be a JUMPDEST per TASK-067). +- `JUMPI { args: [target] }`: pop condition; if non-zero, `pc = target`; else `pc++`. +- Out-of-bounds target → revert. +- Gas: `JUMP: 8n`, `JUMPI: 10n`. + +**Acceptance** +- [ ] Loop fixture: `[PUSH 0 PUSH 10 JUMPDEST DUP1 PUSH 1 ADD JUMPI 3 STOP]` (or similar) terminates correctly. + +**Verification** +- Unit. + +--- + +### TASK-067 — JUMPDEST validation pass + +**Section:** vm +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Compute the set of valid jump targets at execution start. JUMP/JUMPI to any other index reverts. + +**Files** +- edit: Interpreter.ts — at execute() start, scan for `op==='JUMPDEST'`, build `Set`. + +**Implementation sketch** +- Add `JUMPDEST` opcode (`1n` gas, no-op). +- Pre-scan: `const jumpdests = new Set(ops.map((o,i) => o.op==='JUMPDEST' ? i : -1).filter(i => i >= 0))`. +- Pass to JUMP/JUMPI handlers. + +**Acceptance** +- [ ] Jump to non-JUMPDEST → revert. +- [ ] Jump to JUMPDEST → succeeds. + +**Verification** +- Unit. + +--- + +### TASK-068 — SLOAD opcode (read storage) + +**Section:** vm +**Effort:** M +**Depends on:** TASK-314 (contract_storage table) +**Type:** edit + +**Goal** +Read contract storage by key. Pop key from stack, push value (or 0 if unset). + +**Files** +- edit: Interpreter.ts — add SLOAD; need DB read at execution time. + +**Implementation sketch** +- Async-ify `execute()` (becomes `Promise`). +- `SLOAD { args: [] }`: pop key. `SELECT storage_value FROM contract_storage WHERE contract_address=$1 AND storage_key=$2`. Push value or 0. +- Gas: `SLOAD_COLD: 800n`, `SLOAD_WARM: 100n`. Track warm set per execution. + +**Acceptance** +- [ ] Read after SSTORE returns the value. +- [ ] Read of unset key returns 0. +- [ ] Cold-then-warm pricing observed in gasUsed. + +**Verification** +- Integration: SSTORE then SLOAD in same program; receipt logs show expected value. + +--- + +### TASK-069 — Storage persistence to contract_storage table + +**Section:** vm +**Effort:** M +**Depends on:** TASK-314 +**Type:** edit + +**Goal** +Currently SSTORE writes to in-memory `storage` map and is discarded after execution. Persist to PG so SLOAD across blocks works. + +**Files** +- edit: Interpreter.ts — SSTORE handler upserts to `contract_storage`. +- edit: BlockProducer.ts — only commit storage writes if execution succeeded (revert → discard). + +**Implementation sketch** +- Buffer writes in execution result's `storage` object. +- After execute() returns, if status === 'success', `INSERT ... ON CONFLICT DO UPDATE` for each key. +- Updated_at_block = current block height. + +**Acceptance** +- [ ] After successful SSTORE block + restart, value still readable via SLOAD. +- [ ] Reverted execution doesn't persist. + +**Verification** +- Two-block test: write, restart, read. + +--- + +### TASK-070 — CALL opcode + +**Section:** vm +**Effort:** L +**Depends on:** TASK-068, TASK-082 +**Type:** edit + +**Goal** +Invoke another contract. Pop (gas, address, value, calldata) from stack; execute target's bytecode; push success flag. + +**Files** +- edit: Interpreter.ts — add CALL handler with nested execution. + +**Implementation sketch** +- `CALL { args: [] }`: pop `gas, addr, value, calldata`. +- Look up target contract code via `SELECT bytecode FROM contract_code WHERE address=$1`. +- If no code: treat as plain transfer (debit caller, credit target by `value`). +- Else: parse target program, recursively `execute(targetOps, gas, { ...ctx, contractAddress: addr })`. +- Track call depth (TASK-093 cap). +- Push 1 on success, 0 on revert. + +**Acceptance** +- [ ] CALL to contract that emits LOG → log appears in caller's receipt. +- [ ] CALL to non-existent contract → success-as-transfer. + +**Verification** +- Two-contract integration test. + +--- + +### TASK-071 — RETURN opcode + return-data buffer + +**Section:** vm +**Effort:** M +**Depends on:** TASK-070 +**Type:** edit + +**Goal** +Allow callee to return data to caller. RETURN pops a value and ends execution; CALL exposes it via RETURNDATA opcode (sub-task). + +**Files** +- edit: Interpreter.ts — add RETURN; track lastReturnData on execution context. + +**Implementation sketch** +- `RETURN { args: [] }`: pop value, set `result.returnData = value`, terminate. +- Caller's CALL handler reads `result.returnData` after sub-execution. +- Add `RETURNDATA` opcode that pushes the last returndata. + +**Acceptance** +- [ ] CALL → sub-RETURN with value → caller's RETURNDATA pushes that value. + +**Verification** +- Two-contract test. + +--- + +### TASK-072 — CALLDATA opcode + +**Section:** vm +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Push the calldata that this execution was invoked with. For top-level tx, calldata = whatever followed `vm:` in tx.data. + +**Files** +- edit: Interpreter.ts. + +**Implementation sketch** +- ExecutionContext gets `calldata: any`. +- BlockProducer passes the un-prefixed parsed calldata when present. +- `CALLDATA { args: [] }`: push ctx.calldata (as JSON-encoded if non-primitive). + +**Acceptance** +- [ ] Tx with `data: 'vm:[...ops]'` and program using CALLDATA gets the parsed program back. (More useful: define a calldata field separate from program — see TASK-079.) + +**Verification** +- Unit. + +--- + +### TASK-073 — CALLER / ORIGIN ops + +**Section:** vm +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Push the immediate caller's address (CALLER) or the originating tx's `from` address (ORIGIN). Distinct under CALL chains. + +**Files** +- edit: Interpreter.ts. + +**Implementation sketch** +- ExecutionContext gains `caller: string` and `origin: string`. +- For top-level tx: caller === origin === tx.from. +- For CALL: caller = the executing contract; origin unchanged. + +**Acceptance** +- [ ] Top-level: CALLER === ORIGIN === tx.from. +- [ ] Nested: CALLER differs from ORIGIN. + +**Verification** +- Unit. + +--- + +### TASK-074 — VALUE op (msg.value) + +**Section:** vm +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Push the value transferred to this execution. + +**Files** +- edit: Interpreter.ts. + +**Implementation sketch** +- ExecutionContext gains `value: bigint`. +- VALUE pushes `ctx.value`. + +**Acceptance** +- [ ] Tx with value 100 → VALUE returns 100. + +**Verification** +- Unit. + +--- + +### TASK-075 — BALANCE(addr) op + +**Section:** vm +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Pop address, push that account's balance. + +**Files** +- edit: Interpreter.ts — needs DB lookup via stateManager. + +**Reuses** +- `stateManager.getBalance(addr)`. + +**Implementation sketch** +- Gas: 700n. +- `BALANCE { args: [] }`: pop addr, `push(stateManager.getBalance(addr))`. + +**Acceptance** +- [ ] Returns correct balance. + +**Verification** +- Unit. + +--- + +### TASK-076 — BLOCKNUMBER / TIMESTAMP / DIFFICULTY ops + +**Section:** vm +**Effort:** S +**Depends on:** TASK-033 (beacon for randomness via DIFFICULTY) +**Type:** edit + +**Goal** +Standard block-context opcodes. + +**Files** +- edit: Interpreter.ts. + +**Implementation sketch** +- All 3 push from ctx (blockNumber, blockTimestamp, blockDifficulty). +- Provide values from BlockProducer's per-tx context construction. +- DIFFICULTY can be aliased to use the per-block beacon for randomness sources. + +**Acceptance** +- [ ] Each opcode returns the expected block field. + +**Verification** +- Unit + integration. + +--- + +### TASK-077 — SHA256 / KECCAK precompile + +**Section:** vm +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Hash precompiles. Pop input, push 32-byte hash. + +**Files** +- edit: Interpreter.ts — add SHA256, KECCAK opcodes. + +**Reuses** +- Node `crypto.createHash`. + +**Implementation sketch** +- Gas: 60n + 12n per word. +- KECCAK uses SHA3-256 (or actual keccak via `js-sha3` if Solidity-compat). + +**Acceptance** +- [ ] Hash of empty input matches expected. + +**Verification** +- Unit comparing to known hash values. + +--- + +### TASK-078 — ECRECOVER precompile + +**Section:** vm +**Effort:** M +**Depends on:** none +**Type:** edit + +**Goal** +Recover signer address from (msg, sig). Used by signature-based logic in contracts. + +**Files** +- edit: Interpreter.ts. + +**Reuses** +- Ed25519 verification from [Crypto.ts:119](backend/src/blockchain/Crypto.ts#L119) — note this is verify, not recover. For Ed25519, key recovery isn't direct; pop (msg, sig, pubkey) and push the verify result. Document the Ed25519-vs-secp256k1 deviation in spec. + +**Implementation sketch** +- `ECRECOVER` pops 3 args, verifies, pushes the pubkey if valid, 0 otherwise. + +**Acceptance** +- [ ] Valid sig → pubkey. +- [ ] Invalid → 0. + +**Verification** +- Unit. + +--- + +### TASK-079 — CREATE opcode (contract deployment) + +**Section:** vm +**Effort:** L +**Depends on:** TASK-313 (contract_code table) +**Type:** edit + +**Goal** +Deploy a new contract. Pop value + bytecode + salt; compute address; insert into contract_code; return new address. + +**Files** +- edit: Interpreter.ts. + +**Reuses** +- TASK-080's address derivation. + +**Implementation sketch** +- `CREATE { args: [] }`: pop value, bytecode, optional salt. +- Compute address from sender + nonce (or sender + salt for CREATE2-like determinism). +- INSERT into contract_code. +- Push new address. +- Gas: 32000n base + per-byte storage cost. + +**Acceptance** +- [ ] CREATE deploys a contract; subsequent CALL to that address executes its code. + +**Verification** +- Two-tx test: deploy then call. + +--- + +### TASK-080 — Contract address derivation keccak(sender + nonce) + +**Section:** vm +**Effort:** S +**Depends on:** none +**Type:** new-file + +**Goal** +Pure deterministic function used by CREATE. + +**Files** +- new: `backend/src/vm/contractAddress.ts` — exports `deriveContractAddress(sender: string, nonce: number): string`. + +**Implementation sketch** +- `sha256(`${sender}:${nonce}`).slice(0, 44)` (base58, address-shaped). +- Document: NOT EVM-compatible; documented as Hermes-specific. + +**Acceptance** +- [ ] Same inputs → same address. +- [ ] Different inputs → different addresses. + +**Verification** +- Unit. + +--- + +### TASK-081 — contract_code table storage helpers + +**Section:** vm +**Effort:** S +**Depends on:** TASK-313 +**Type:** new-file + +**Goal** +Wrap insert/lookup of contract bytecode in helpers. + +**Files** +- new: `backend/src/vm/codeStore.ts` — `storeCode(addr, hash, bytecode, deployedAtBlock, deployedBy)`, `loadCode(addr)`. + +**Reuses** +- `db.query`. + +**Implementation sketch** +- Compute code_hash via SHA256 of bytecode. +- Cache loadCode results in-memory (LRU, 1k entries). + +**Acceptance** +- [ ] Store + load round-trip. + +**Verification** +- Unit. + +--- + +### TASK-082 — Code-loaded execution path in BlockProducer + +**Section:** vm +**Effort:** M +**Depends on:** TASK-081 +**Type:** edit + +**Goal** +Currently BlockProducer dispatches to VM only when `tx.data.startsWith('vm:')`. Also need: if `tx.to` has deployed code, execute that code (calldata = tx.data). + +**Files** +- edit: BlockProducer.ts:115-160. + +**Reuses** +- `loadCode(addr)` from TASK-081. + +**Implementation sketch** +- After verifying signature: `const targetCode = await loadCode(tx.to);`. +- If targetCode exists: parse, execute with calldata = `tx.data`. +- Else if `tx.data.startsWith('vm:')`: existing path. +- Else: plain transfer (existing path). + +**Acceptance** +- [ ] Tx to contract address invokes contract code. + +**Verification** +- Deploy + call sequence. + +--- + +### TASK-083 — /api/tx/:hash/trace endpoint + +**Section:** vm +**Effort:** M +**Depends on:** none +**Type:** edit + +**Goal** +Re-execute a tx with full per-op tracing for debugging. Returns the op sequence with stack snapshots. + +**Files** +- new: `backend/src/api/vm-trace.ts`. + +**Reuses** +- TASK-055's tx simulation infrastructure. + +**API contract** +``` +GET /api/tx/:hash/trace +→ 200 { steps: [ { pc, op, gasBefore, gasAfter, stackBefore, stackAfter } ] } +``` + +**Implementation sketch** +- Look up tx + parent state. +- Run interpreter with `trace: true` flag that records each step. +- Return steps array. + +**Acceptance** +- [ ] Trace shape matches API contract. + +**Verification** +- Trace a known tx, walk steps. + +--- + +### TASK-084 — /api/tx/:hash/gas-profile per-op breakdown + +**Section:** vm +**Effort:** S +**Depends on:** TASK-083 +**Type:** edit + +**Goal** +Aggregate per-op gas usage from the trace. + +**Files** +- new endpoint or extend TASK-083 with `?profile=true`. + +**Implementation sketch** +- Walk trace, sum gas per opcode name. +- Return `{ MUL: 1500, SLOAD: 8000, ... }`. + +**Acceptance** +- [ ] Returns gas histogram by op. + +**Verification** +- Curl on a known tx. + +--- + +### TASK-085 — STATICCALL (read-only nested call) + +**Section:** vm +**Effort:** M +**Depends on:** TASK-070 +**Type:** edit + +**Goal** +Like CALL but disallows storage writes in the sub-execution. + +**Files** +- edit: Interpreter.ts. + +**Implementation sketch** +- Set `ctx.readOnly = true` during sub-execution. +- SSTORE / CREATE / SELFDESTRUCT in read-only ctx → revert. + +**Acceptance** +- [ ] STATICCALL into contract that tries SSTORE → revert. +- [ ] STATICCALL into pure-read contract → success. + +**Verification** +- Two-contract test. + +--- + +### TASK-086 — SELFDESTRUCT op + +**Section:** vm +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Mark contract for removal at end of block; refund balance to recipient address (popped). + +**Files** +- edit: Interpreter.ts. + +**Implementation sketch** +- Pop recipient address. +- `result.selfDestruct = { recipient }`. +- BlockProducer post-tx: if selfDestruct set, transfer balance to recipient and DELETE FROM contract_code. + +**Acceptance** +- [ ] After SELFDESTRUCT, contract code gone, balance moved. + +**Verification** +- Integration. + +--- + +### TASK-087 — SSTORE refund for zero-set + +**Section:** vm +**Effort:** S +**Depends on:** TASK-069 +**Type:** edit + +**Goal** +EVM-style: setting a non-zero slot to zero refunds gas (bounded). Encourages cleanup. + +**Files** +- edit: Interpreter.ts SSTORE handler. + +**Implementation sketch** +- If `currentValue !== 0` and `newValue === 0`: refund 15000n at end of execution (capped at 1/2 of gasUsed). + +**Acceptance** +- [ ] SSTORE 0 over a non-zero slot reduces final gasUsed. + +**Verification** +- Unit. + +--- + +### TASK-088 — Cold/warm SLOAD pricing + +**Section:** vm +**Effort:** S +**Depends on:** TASK-068 +**Type:** edit + +**Goal** +First read of a slot in a tx is "cold" (high gas); subsequent reads are "warm" (low). Already sketched in TASK-068; this task makes the pricing real. + +**Files** +- edit: Interpreter.ts. + +**Implementation sketch** +- Per-execution `Set warmSlots`. +- `cost = warmSlots.has(key) ? 100n : 800n; warmSlots.add(key);`. + +**Acceptance** +- [ ] First SLOAD cost 800; second cost 100. + +**Verification** +- Unit. + +--- + +### TASK-089 — Memory model: byte-addressable scratch + +**Section:** vm +**Effort:** M +**Depends on:** none +**Type:** edit + +**Goal** +Add a per-execution memory buffer (Uint8Array) addressable by byte offset. Used by hashing precompiles and contract internals. + +**Files** +- edit: Interpreter.ts. + +**Implementation sketch** +- `let memory = new Uint8Array(0)`. +- Grow on demand to nearest 32-byte boundary; track `memorySize` for gas accounting (TASK-091). + +**Acceptance** +- [ ] Memory accessible (after MLOAD/MSTORE in TASK-090). + +**Verification** +- Unit via TASK-090. + +--- + +### TASK-090 — MLOAD / MSTORE / MSTORE8 + +**Section:** vm +**Effort:** S +**Depends on:** TASK-089 +**Type:** edit + +**Goal** +Load/store memory. + +**Files** +- edit: Interpreter.ts. + +**Implementation sketch** +- MLOAD: pop offset, push 32-byte value as bigint. +- MSTORE: pop offset + value, write 32 bytes. +- MSTORE8: pop offset + value, write 1 byte (lowest byte of value). + +**Acceptance** +- [ ] Store + load round-trip preserves value. + +**Verification** +- Unit. + +--- + +### TASK-091 — Memory expansion gas + +**Section:** vm +**Effort:** S +**Depends on:** TASK-089 +**Type:** edit + +**Goal** +Charge for memory growth: cost = quadratic in size to discourage abuse. + +**Files** +- edit: Interpreter.ts memory grow path. + +**Implementation sketch** +- `cost(N) = 3n * N + N*N / 512n` where N = memory words. +- Charge difference between old and new cost on every grow. + +**Acceptance** +- [ ] Growing memory consumes gas. + +**Verification** +- Unit: write at high offset, observe gas spike. + +--- + +### TASK-092 — Stack depth limit 1024 + +**Section:** vm +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Prevent unbounded stack growth. + +**Files** +- edit: Interpreter.ts each push site. + +**Implementation sketch** +- Wrap stack push: `if (stack.length >= 1024) return revert('stack overflow')`. + +**Acceptance** +- [ ] Push 1025 items → revert at item 1025. + +**Verification** +- Unit. + +--- + +### TASK-093 — Call depth limit 1024 + +**Section:** vm +**Effort:** S +**Depends on:** TASK-070 +**Type:** edit + +**Goal** +Prevent infinite recursion via CALL. + +**Files** +- edit: Interpreter.ts CALL handler. + +**Implementation sketch** +- ExecutionContext tracks `depth: number`. +- CALL: `if (depth >= 1024) revert('call depth exceeded')`. +- Sub-execute with `depth + 1`. + +**Acceptance** +- [ ] 1025-deep CALL chain → revert at depth 1025. + +**Verification** +- Unit. + +--- + +### TASK-094 — REVERT with return data + +**Section:** vm +**Effort:** S +**Depends on:** TASK-071 +**Type:** edit + +**Goal** +Currently REVERT just sets an error string. Allow it to also pop return data so caller can read it. + +**Files** +- edit: Interpreter.ts REVERT handler. + +**Implementation sketch** +- `REVERT { args: [] }`: pop return value, set `result.returnData`, set status 'revert'. + +**Acceptance** +- [ ] CALL into reverting sub: caller's RETURNDATA returns the revert data. + +**Verification** +- Unit. + +--- + +### TASK-095 — Try-catch CALL semantics + +**Section:** vm +**Effort:** S +**Depends on:** TASK-070, TASK-094 +**Type:** edit + +**Goal** +CALL already returns success flag (TASK-070). Document try-catch as: caller checks the flag and branches; the language stub (TASK-104) emits the right code. + +**Files** +- new: `docs/vm/try-catch.md` (separate from /docs/vm spec) — convention doc. + +**Implementation sketch** +- No new opcode; convention only. + +**Acceptance** +- [ ] Doc explains pattern with example. + +**Verification** +- Doc renders. + +--- + +### TASK-096 — EVENT opcode separate from LOG + +**Section:** vm +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +LOG is EVM-style untyped. EVENT is named/typed (resolved against ABI registry from TASK-097). Cleaner DX. + +**Files** +- edit: Interpreter.ts. + +**Implementation sketch** +- `EVENT { args: { name: string, fields: Record } }`. +- Look up event signature in ABI registry; encode topics; emit as LOG under the hood. +- Gas: same as LOG. + +**Acceptance** +- [ ] EVENT 'Transfer' { from, to, value } → log row with correct topic0. + +**Verification** +- Unit + integration with TASK-097. + +--- + +### TASK-097 — Event ABI registry per contract + +**Section:** vm +**Effort:** M +**Depends on:** TASK-315 (contract_metadata table) +**Type:** new-file + +**Goal** +Per-contract registry of event names → topic0 hashes + field schemas. Used by EVENT (TASK-096) and decode endpoints. + +**Files** +- new: `backend/src/vm/abiRegistry.ts` — `registerAbi(contractAddress, abiJson)`, `lookupEvent(contractAddress, eventName)`. + +**Reuses** +- `contract_metadata.abi_json` from TASK-315. + +**Implementation sketch** +- ABI schema: `{ events: [ { name, fields: [{name, type}] } ] }`. +- topic0 = keccak(`${name}(${fields.map(f=>f.type).join(',')})`).slice(0, 32 bytes). +- Cache parsed ABI in-memory. + +**Acceptance** +- [ ] After register, lookup returns expected topic0. + +**Verification** +- Unit. + +--- + +### TASK-098 — /api/contract/:addr/source verifier + +**Section:** vm +**Effort:** M +**Depends on:** TASK-315 +**Type:** edit + +**Goal** +Allow uploading source for a deployed contract; recompile (via TASK-104 stub compiler), compare bytecode hash; mark verified. + +**Files** +- new: `backend/src/api/contract-verify.ts`. + +**API contract** +``` +POST /api/contract/:addr/source +body: { source: '...DSL source...' } +→ 200 { verified: true } +→ 409 { verified: false, reason: 'bytecode mismatch' } +``` + +**Implementation sketch** +- Compile source via TASK-104. +- Compare hash with `contract_code.code_hash`. +- On match: UPDATE contract_metadata SET source_verified = TRUE. + +**Acceptance** +- [ ] Matching source → verified. + +**Verification** +- Round-trip: deploy via known source, verify. + +--- + +### TASK-099 — /api/contract/:addr/disasm endpoint + +**Section:** vm +**Effort:** S +**Depends on:** TASK-313 +**Type:** edit + +**Goal** +Return the JSON-op program for a deployed contract. + +**Files** +- new endpoint. + +**API contract** +``` +GET /api/contract/:addr/disasm +→ 200 { ops: [ {op, args} ... ] } +``` + +**Acceptance** +- [ ] Returns parsed program. + +**Verification** +- Curl. + +--- + +### TASK-100 — /api/contract/:addr/storage browser + +**Section:** vm +**Effort:** S +**Depends on:** TASK-314 +**Type:** edit + +**Goal** +List a contract's stored slots with optional key prefix. + +**Files** +- new endpoint. + +**API contract** +``` +GET /api/contract/:addr/storage?prefix=&cursor=&limit=100 +→ 200 { items: [ {key, value, updated_at_block} ], next_cursor } +``` + +**Acceptance** +- [ ] Returns rows from `contract_storage`. + +**Verification** +- Curl. + +--- + +### TASK-101 — contract_metadata table is sufficient + +**Section:** vm +**Effort:** S +**Depends on:** TASK-315 +**Type:** docs + +**Goal** +This task overlaps with TASK-315; closed once that migration lands. Document the linkage. + +**Files** +- edit: queue.md index — annotate TASK-101 as covered by TASK-315. + +**Acceptance** +- [ ] Linkage documented. + +**Verification** +- N/A. + +--- + +### TASK-102 — VM unit test fixtures (30 sample programs) + +**Section:** vm +**Effort:** M +**Depends on:** none +**Type:** test + +**Goal** +30 small JSON-op programs exercising every opcode in golden-test fashion: input → expected (status, gasUsed, logs, storage). + +**Files** +- new: `backend/tests/vm-fixtures/*.json` (30 files). +- new: `backend/tests/vm-fixtures.test.ts` — runs interpreter against each fixture, asserts output. + +**Implementation sketch** +- Each fixture: `{ ops, gasLimit, ctx, expected: { status, gasUsed, logs, storage } }`. +- Cover: arithmetic (5), comparisons (5), control flow (5), storage (5), CALL/RETURN (5), edge cases (5). + +**Acceptance** +- [ ] All 30 pass. + +**Verification** +- `npm test`. + +--- + +### TASK-103 — /docs/vm spec page + +**Section:** vm +**Effort:** M +**Depends on:** TASK-061..102 +**Type:** docs + +**Goal** +Authoritative reference: every opcode, gas cost, semantics, edge cases. + +**Files** +- new: `docs/vm/spec.md`. + +**Implementation sketch** +- Table: opcode | gas | stack effect | description. +- Examples for non-trivial ops. +- Versioning note. + +**Acceptance** +- [ ] Doc covers every opcode in Interpreter.ts. + +**Verification** +- Manual review. + +--- + +### TASK-104 — Compiler stub: tiny DSL → JSON-op + +**Section:** vm +**Effort:** L +**Depends on:** TASK-061..094 +**Type:** new-file + +**Goal** +A toy higher-level language → JSON-op compiler so users can write `let x = 1 + 2; emit(x);` instead of raw stack manipulation. Just enough to demo: assignments, arithmetic, if/while, function call, emit. + +**Files** +- new: `backend/src/vm/compiler/` — lexer, parser, emitter. +- new: `backend/scripts/vm-compile.ts` — CLI wrapper. + +**Implementation sketch** +- Lexer: tokens (NUMBER, IDENT, OP, KEYWORD). +- Parser: recursive descent → AST. +- Emitter: AST → JSON-op array. +- Stack-machine register allocation (no SSA needed at this scale). + +**Acceptance** +- [ ] Counter contract compiles and runs. + +**Verification** +- `npm run vm:compile sample.hsm` → `sample.json` with valid program. + +--- + +### TASK-105 — Sample contracts: counter / erc20-like / multisig / vrf + +**Section:** vm +**Effort:** M +**Depends on:** TASK-104 +**Type:** docs + +**Goal** +Four worked examples in the new DSL with deployable JSON output and walkthrough docs. + +**Files** +- new: `examples/counter/{source.hsm,program.json,README.md}` +- new: `examples/erc20-like/...` +- new: `examples/multisig/...` +- new: `examples/vrf/...` + +**Implementation sketch** +- Counter: increment + read. +- ERC20-like: balances + transfer + approve. +- Multisig: 2-of-3 with on-chain proposals. +- VRF: consume per-block beacon from TASK-033. + +**Acceptance** +- [ ] All 4 deploy and execute on dev chain. + +**Verification** +- Walkthrough each example end-to-end. + +--- + +## Summary + +45 tasks: 24 small, 13 medium, 4 large, 1 docs-link. Heavy mid-cluster around storage + CALL semantics; lighter early (arithmetic) and late (sample contracts). From cb8ec38f70d592f47cfdcac3f48736f5db9d2d35 Mon Sep 17 00:00:00 2001 From: hermes agent Date: Tue, 28 Apr 2026 01:52:26 +0400 Subject: [PATCH 04/96] =?UTF-8?q?docs(backlog):=20detailed=20specs=20for?= =?UTF-8?q?=20section=2004=20=E2=80=94=20API=20&=20explorer=20(TASK-141..1?= =?UTF-8?q?80)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 40 task specs: OpenAPI 3.1 generation + Swagger UI, /api/v1/* versioning with deprecation headers, observability middleware (request-id, NDJSON access log, slow-request log, rate-limit headers), CORS allowlist via env, three-tier health checks (/live, /ready, /deep), build/flags/Prometheus metrics endpoints, search endpoints (blocks, txs, top-by-balance/activity, gas spenders, validator leaderboard), network dashboard bundle, block + tx detail with receipts/decoded logs, contract events feed, address tag system + suggestions, mempool snapshot/by-hash/cancel/bulk-submit/ idempotent-submit, WebSocket mirrors of every SSE channel, Socket.io per-address rooms, SSE event replay, GraphQL gateway, tRPC mirror, Ethereum JSON-RPC compat (eth_blockNumber/getBalance/call/sendRaw/ subscribe), Postman collection generator. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/backlog/queue/04-api-explorer.md | 1065 +++++++++++++++++++++++++ 1 file changed, 1065 insertions(+) create mode 100644 docs/backlog/queue/04-api-explorer.md diff --git a/docs/backlog/queue/04-api-explorer.md b/docs/backlog/queue/04-api-explorer.md new file mode 100644 index 00000000..19105796 --- /dev/null +++ b/docs/backlog/queue/04-api-explorer.md @@ -0,0 +1,1065 @@ +# Section 04 — API & Explorer Specs (TASK-141..180) + +40 tasks. OpenAPI/Swagger, versioning, observability middleware (rate-limit headers, request IDs, access log, slow-query log), health subroutes, build/flags/metrics endpoints, search + leaderboards, mempool/reorg/contract feeds, address tagging, WebSocket + Socket.io rooms, GraphQL/tRPC/JSON-RPC compatibility, Postman collection generator. + +**Preconditions used throughout:** +- Express server: [backend/src/api/server.ts](backend/src/api/server.ts) — current route mounts at lines 90-700. +- Auth: [backend/src/api/auth.ts](backend/src/api/auth.ts) — `requireApiKey('scope')` middleware, `ipRateLimit(perMin)`. +- Receipts: `loadReceipt()` from [TransactionReceipt.ts:255](backend/src/blockchain/TransactionReceipt.ts#L255). +- DB: `db.query`, `db.queryRead` (TASK-322). +- Mesh: [backend/src/network/api.ts](backend/src/network/api.ts). +- SSE pattern: [server.ts:971-1057](backend/src/api/server.ts#L971-L1057). + +--- + +### TASK-141 — /api/openapi.json generation + +**Section:** api +**Effort:** M +**Depends on:** none +**Type:** new-file + +**Goal** +Auto-generate OpenAPI 3.1 spec from route declarations so clients can codegen typed SDKs and we get free API docs. + +**Files** +- new: `backend/src/api/openapi.ts` — exports `buildOpenApiSpec(): object`. +- edit: `backend/src/api/server.ts` — register `GET /api/openapi.json`. + +**Reuses** +- Express's `app._router.stack` for route enumeration. + +**API contract** +``` +GET /api/openapi.json +→ 200 { openapi: "3.1.0", info, paths: {...}, components: {...} } +``` + +**Implementation sketch** +- Walk `app._router.stack` and registered routers; emit `paths[route][method]` entries. +- Per-route metadata supplied via a side annotation `routeDoc(route, { summary, params, responses })` collected at registration time. +- Components: shared schemas (Block, Transaction, Receipt) from a single source. + +**Acceptance** +- [ ] Returns valid OpenAPI 3.1 (passes `swagger-cli validate`). +- [ ] Every existing route appears. + +**Verification** +- `curl /api/openapi.json | swagger-cli validate -`. + +--- + +### TASK-142 — Swagger UI at /docs + +**Section:** api +**Effort:** S +**Depends on:** TASK-141 +**Type:** edit + +**Goal** +Mount swagger-ui-express at `/docs` reading from the openapi.json endpoint. + +**Files** +- edit: `backend/src/api/server.ts` — add `swaggerUi.serve` and `swaggerUi.setup`. +- add dep: `swagger-ui-express`. + +**Implementation sketch** +- `app.use('/docs', swaggerUi.serve, swaggerUi.setup(undefined, { swaggerOptions: { url: '/api/openapi.json' } }))`. + +**Acceptance** +- [ ] `/docs` renders the spec with try-it-out. + +**Verification** +- Open `/docs` in browser. + +--- + +### TASK-143 — /api/v1/* version prefix + deprecation headers + +**Section:** api +**Effort:** M +**Depends on:** none +**Type:** edit + +**Goal** +Stable version 1 prefix. Bare `/api/*` keeps working but returns `Deprecation: true` and `Sunset: ` headers. + +**Files** +- edit: `backend/src/api/server.ts` — wrap existing app.use calls in helper that mounts at both `/api/v1/...` and `/api/...`; the latter wraps with deprecation middleware. + +**Implementation sketch** +- `mountVersioned(app, '/auth', authRouter)` mounts twice; bare path adds the response header. + +**Acceptance** +- [ ] `/api/v1/status` works. +- [ ] `/api/status` works AND has `Deprecation: true` header. + +**Verification** +- `curl -I /api/status | grep Deprecation`. + +--- + +### TASK-144 — Rate-limit headers (X-RateLimit-*) + +**Section:** api +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +[ipRateLimit](backend/src/api/auth.ts) silently rejects with 429. Surface remaining quota in response headers so clients can self-pace. + +**Files** +- edit: `backend/src/api/auth.ts:ipRateLimit` — set `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset` on every response. + +**Acceptance** +- [ ] Headers present on every rate-limited route's response. +- [ ] Remaining decrements per call. + +**Verification** +- `curl -i /api/personality/hermes ...`. + +--- + +### TASK-145 — CORS allowlist via env + +**Section:** api +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +[server.ts:80](backend/src/api/server.ts#L80) uses `cors()` (open). Restrict to `CORS_ORIGINS` env (comma-separated) when set. + +**Files** +- edit: `backend/src/api/server.ts:80`. + +**Implementation sketch** +- If env set: `cors({ origin: (origin, cb) => cb(null, allowed.includes(origin)) })`. +- Default unchanged for dev. + +**Acceptance** +- [ ] With env, disallowed origin → CORS error. + +**Verification** +- Test with curl Origin header. + +--- + +### TASK-146 — Request-ID middleware + +**Section:** api +**Effort:** S +**Depends on:** none +**Type:** new-file + +**Goal** +Echo or assign `X-Request-ID` header on every request. Used by access log + slow-query log + error reporting for correlation. + +**Files** +- new: `backend/src/api/middleware/requestId.ts`. +- edit: `backend/src/api/server.ts` — register before any other middleware. + +**Implementation sketch** +- `req.id = req.headers['x-request-id'] || crypto.randomUUID()`. +- `res.setHeader('X-Request-ID', req.id)`. + +**Acceptance** +- [ ] Header present on all responses. + +**Verification** +- `curl -i /api/status | grep X-Request-ID`. + +--- + +### TASK-147 — Structured access log NDJSON + +**Section:** api +**Effort:** S +**Depends on:** TASK-146 +**Type:** new-file + +**Goal** +One JSON-line per request to stdout: `{ts, method, path, status, durationMs, bytes, requestId, ip}`. Easy to feed into Logflare/Datadog. + +**Files** +- new: `backend/src/api/middleware/accessLog.ts`. +- edit: server.ts — register after requestId. + +**Implementation sketch** +- On `res.on('finish')`, write the JSON line. + +**Acceptance** +- [ ] Every request logged exactly once. + +**Verification** +- `curl /api/status` → stdout shows the line. + +--- + +### TASK-148 — Slow-request log >1s + +**Section:** api +**Effort:** S +**Depends on:** TASK-147 +**Type:** edit + +**Goal** +Tag requests over a threshold (default 1s) in the access log with `slow: true` and elevate to console.warn. + +**Files** +- edit: accessLog.ts. + +**Implementation sketch** +- Threshold env `SLOW_REQUEST_MS` (default 1000). +- If durationMs > threshold: also `console.warn(...)`. + +**Acceptance** +- [ ] Slow request appears in console.warn. + +**Verification** +- Inject pg_sleep route or call /api/agent/stream briefly. + +--- + +### TASK-149 — /health/live + /health/ready + /health/deep + +**Section:** api +**Effort:** M +**Depends on:** none +**Type:** new-file + +**Goal** +Three-tier health checks per the design doc. live = process up; ready = can serve traffic; deep = exercises full read paths. + +**Files** +- new: `backend/src/api/health.ts`. +- edit: server.ts — mount. + +**API contract** +``` +GET /health/live → 200 { status: 'live' } +GET /health/ready + → 200 { ready: true, checks: {...} } + → 503 { ready: false, failures: [{check, reason}] } +GET /health/deep + → 200 { ready: true, latencyMs: { db: 12, redis: 3, chainHead: 1 } } + → 503 { ready: false, failures: [...] } +``` + +**Implementation sketch** +- `live`: always 200 unless shutting down (graceful-shutdown handler flips a flag). +- `ready`: check `db.poolStats()`, `cache.isConnected()`, agent worker heartbeat (TASK-332), chain bootstrap height. +- `deep`: actually run a SELECT, GET, and chain head fetch with timing. + +**Acceptance** +- [ ] All three endpoints return appropriate codes. +- [ ] During shutdown, ready returns 503. + +**Verification** +- Curl each. + +--- + +### TASK-150 — /api/build endpoint + +**Section:** api +**Effort:** S +**Depends on:** none +**Type:** new-file + +**Goal** +Returns build info for debugging deploys: commit SHA, build timestamp, version. + +**Files** +- new: `backend/src/api/build.ts`. +- edit: build script — write `backend/build-info.json` with `{ commit: $(git rev-parse HEAD), buildTime: now }`. + +**API contract** +``` +GET /api/build → 200 { commit, buildTime, version } +``` + +**Acceptance** +- [ ] Returns commit SHA matching deployed code. + +**Verification** +- Compare with `git log` after build. + +--- + +### TASK-151 — /api/flags feature-flag endpoint + +**Section:** api +**Effort:** S +**Depends on:** none +**Type:** new-file + +**Goal** +Surface the typed feature flag registry already in the codebase. Clients use it for progressive rollout UI. + +**Files** +- new: `backend/src/api/flags.ts`. +- edit: server.ts — mount. + +**Reuses** +- Existing flag registry (was committed in `feat(ops): typed feature flag registry`). + +**API contract** +``` +GET /api/flags → 200 { flags: { vmEnabled: true, beaconRandomness: false, ... } } +``` + +**Acceptance** +- [ ] Lists all flags + current values. + +**Verification** +- Curl. + +--- + +### TASK-152 — /api/metrics Prometheus text format + +**Section:** api +**Effort:** M +**Depends on:** TASK-319, TASK-320 +**Type:** new-file + +**Goal** +Standard Prometheus exposition format. Counters + gauges + histograms. + +**Files** +- new: `backend/src/api/metrics.ts`. + +**Reuses** +- `db.poolStats()` (TASK-319). +- Query histogram from TASK-320. +- Chain stats: `chain.getChainLength()`, mempool size, etc. + +**API contract** +``` +GET /api/metrics → 200 (text/plain) + # HELP hermes_chain_height ... + # TYPE hermes_chain_height gauge + hermes_chain_height 1234 + ... +``` + +**Acceptance** +- [ ] Output parses by `prom2json` without errors. +- [ ] Includes pool, query histogram, chain height, mempool size, peer count. + +**Verification** +- `curl /api/metrics | prom2json -`. + +--- + +### TASK-153 — Block search by height range with filters + +**Section:** api +**Effort:** S +**Depends on:** none +**Type:** new-file + +**Goal** +Paginated block listing with height range + producer filter. + +**Files** +- new endpoint in server.ts. + +**API contract** +``` +GET /api/blocks/search?from=&to=&producer=&limit=50&cursor= +→ 200 { items: [...], next_cursor } +``` + +**Acceptance** +- [ ] Filters apply. +- [ ] Cursor pagination correct. + +**Verification** +- Curl with combinations. + +--- + +### TASK-154 — Tx search by from/to/value range + +**Section:** api +**Effort:** S +**Depends on:** none +**Type:** new-file + +**Goal** +Find txs by sender, recipient, or value range. + +**Files** +- new endpoint in server.ts. + +**API contract** +``` +GET /api/tx/search?from=&to=&minValue=&maxValue=&limit=50&cursor= +→ 200 { items: [...], next_cursor } +``` + +**Acceptance** +- [ ] Filters apply. + +**Verification** +- Curl. + +--- + +### TASK-155 — Top accounts by balance + +**Section:** api +**Effort:** S +**Depends on:** TASK-308 +**Type:** new-file + +**Goal** +Leaderboard endpoint. + +**Files** +- new: `GET /api/accounts/top?limit=100`. + +**Reuses** +- `idx_accounts_balance_desc` from TASK-308. + +**API contract** +``` +GET /api/accounts/top?limit=100 +→ 200 { items: [{ address, balance, rank }] } +``` + +**Acceptance** +- [ ] Sorted by balance DESC. + +**Verification** +- Curl. + +--- + +### TASK-156 — Top accounts by tx count + +**Section:** api +**Effort:** S +**Depends on:** TASK-307 +**Type:** new-file + +**Goal** +Same shape as TASK-155 but ranked by `(SELECT COUNT(*) FROM transactions WHERE from_address = a.address OR to_address = a.address)`. + +**Files** +- new: `GET /api/accounts/top-by-activity?limit=100`. + +**Acceptance** +- [ ] Returns count-ranked list. + +**Verification** +- Curl. + +--- + +### TASK-157 — Validator leaderboard + +**Section:** api +**Effort:** S +**Depends on:** TASK-053 +**Type:** new-file + +**Goal** +Validators ranked by blocks_produced and uptime. + +**Files** +- new: `GET /api/validators/leaderboard`. + +**API contract** +``` +→ 200 { items: [{ address, name, blocks_produced, uptime, rank }] } +``` + +**Acceptance** +- [ ] Sorted correctly. + +**Verification** +- Curl. + +--- + +### TASK-158 — Network stats dashboard endpoint + +**Section:** api +**Effort:** S +**Depends on:** none +**Type:** new-file + +**Goal** +One endpoint that bundles peers/mempool/tps/finality-lag for the HUD. + +**Files** +- new: `GET /api/network/dashboard`. + +**API contract** +``` +→ 200 { + peers: { active: 5, total: 7 }, + mempool: { pending: 12 }, + tps: { window60: 3.4 }, + finality: { headHeight: 1234, finalizedHeight: 1222, lagBlocks: 12 } +} +``` + +**Acceptance** +- [ ] All fields populated. + +**Verification** +- Curl. + +--- + +### TASK-159 — Block detail with full receipts inline + +**Section:** api +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Existing `/api/blocks/:height` returns block JSON without receipts. Inline them. + +**Files** +- edit: `backend/src/api/server.ts:183`. + +**Reuses** +- `loadBlockReceipts(height)`. + +**API contract** +``` +GET /api/blocks/:height?include=receipts +→ 200 { ...block, receipts: [...] } +``` + +**Acceptance** +- [ ] `?include=receipts` returns receipts array. + +**Verification** +- Curl. + +--- + +### TASK-160 — Tx detail with decoded log events + +**Section:** api +**Effort:** M +**Depends on:** TASK-097 +**Type:** edit + +**Goal** +Existing `/api/tx/:hash` returns raw logs. Add ABI-decoded form when contract has registered ABI. + +**Files** +- edit: server.ts — extend tx handler. + +**Reuses** +- `lookupEvent` from TASK-097. + +**API contract** +``` +GET /api/tx/:hash?decodeLogs=true +→ 200 { ...tx, logs: [{ ...raw, decoded: { name, fields } | null }] } +``` + +**Acceptance** +- [ ] Logs from contracts with ABI get decoded form. + +**Verification** +- Curl. + +--- + +### TASK-161 — /api/contract/:addr/events feed + +**Section:** api +**Effort:** S +**Depends on:** TASK-310, TASK-097 +**Type:** new-file + +**Goal** +Per-contract event history. + +**Files** +- new: `GET /api/contract/:addr/events?limit=&cursor=`. + +**Acceptance** +- [ ] Returns logs filtered to that contract address. + +**Verification** +- Curl. + +--- + +### TASK-162 — Address tag system + +**Section:** api +**Effort:** M +**Depends on:** none +**Type:** new-file + +**Goal** +Operators can tag addresses ("exchange", "validator", "burn"). Tags surface alongside balances in account endpoints. + +**Files** +- new: `backend/src/database/migrations/0021_address_tags.sql` — `address_tags(address, tag, source, created_at, PK(address, tag))`. +- new: `backend/src/api/tags.ts` — CRUD endpoints. + +**API contract** +``` +GET /api/tags/:addr → 200 { address, tags: ['exchange', 'validator'] } +POST /api/tags (admin) body: {address, tag} → 200 { ok: true } +DELETE /api/tags/:addr/:tag (admin) → 200 { ok: true } +``` + +**Acceptance** +- [ ] Tags persist + appear in `/api/account/:addr`. + +**Verification** +- Add + read. + +--- + +### TASK-163 — Tag suggestion endpoint + +**Section:** api +**Effort:** S +**Depends on:** TASK-162 +**Type:** new-file + +**Goal** +Heuristic-based suggestions for untagged addresses (high tx count, large balance, validator). + +**Files** +- new: `GET /api/tags/suggest/:addr`. + +**Implementation sketch** +- If validator → suggest 'validator'. +- If receives many small tx daily → suggest 'exchange'. +- If sends many txs but receives ~0 → suggest 'distributor'. + +**Acceptance** +- [ ] Returns list of suggestions with confidence. + +**Verification** +- Curl on a known active address. + +--- + +### TASK-164 — Top gas spenders last 24h + +**Section:** api +**Effort:** S +**Depends on:** none +**Type:** new-file + +**Goal** +Leaderboard of who's burning the most gas. + +**Files** +- new: `GET /api/accounts/top-gas?windowHours=24`. + +**Acceptance** +- [ ] Returns sorted list. + +**Verification** +- Curl. + +--- + +### TASK-165 — /api/reorg/:id detail page + +**Section:** api +**Effort:** S +**Depends on:** TASK-060 +**Type:** new-file + +**Goal** +Drill into a specific reorg event from `reorg_log`. + +**Files** +- new: `GET /api/reorg/:id`. + +**Acceptance** +- [ ] Returns row + linked orphaned blocks. + +**Verification** +- Curl. + +--- + +### TASK-166 — /api/mempool snapshot + +**Section:** api +**Effort:** S +**Depends on:** none +**Type:** new-file + +**Goal** +List pending txs (capped). + +**Files** +- new: `GET /api/mempool?limit=200`. + +**Acceptance** +- [ ] Returns array of pending tx JSON. + +**Verification** +- Curl. + +--- + +### TASK-167 — /api/mempool/:hash pending tx by hash + +**Section:** api +**Effort:** S +**Depends on:** none +**Type:** new-file + +**Goal** +Lookup a specific pending tx (404 if mined or unknown). + +**Files** +- new: `GET /api/mempool/:hash`. + +**Acceptance** +- [ ] 200 with tx if pending; 404 if mined. + +**Verification** +- Curl. + +--- + +### TASK-168 — Cancel pending tx endpoint + +**Section:** api +**Effort:** M +**Depends on:** TASK-019 +**Type:** new-file + +**Goal** +Sender-signed cancel: submit a no-op tx with same nonce + 11% higher gas (RBF). + +**Files** +- new: `POST /api/mempool/:hash/cancel`. + +**Implementation sketch** +- Body: `{ signature }` over message `cancel:${hash}:${timestamp}`. +- Verify signature against tx.from. +- Construct a self-transfer (from→from, value=0) with same nonce + bumped gasPrice. +- Submit via TransactionPool. + +**Acceptance** +- [ ] Pending tx cancelled (replaced by no-op). + +**Verification** +- Submit, cancel, observe replacement. + +--- + +### TASK-169 — Bulk tx submit endpoint + +**Section:** api +**Effort:** S +**Depends on:** none +**Type:** new-file + +**Goal** +Single POST with multiple txs. + +**Files** +- new: `POST /api/transactions/bulk` body: `{ transactions: [...] }` → `{ accepted: [hashes], rejected: [{tx, reason}] }`. + +**Acceptance** +- [ ] Accepts up to 100 per call. + +**Verification** +- Bulk curl. + +--- + +### TASK-170 — Idempotent tx submit (dedup on hash) + +**Section:** api +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Re-submitting the same hash should return the existing acceptance, not error. + +**Files** +- edit: `backend/src/api/server.ts:319` — check for existing pending or confirmed tx with same hash, short-circuit. + +**Acceptance** +- [ ] Same tx submitted twice → 200 both times. + +**Verification** +- Submit twice. + +--- + +### TASK-171 — WebSocket equivalents of all SSE channels + +**Section:** api +**Effort:** L +**Depends on:** TASK-047, TASK-048, TASK-049 +**Type:** new-file + +**Goal** +Every SSE endpoint also exposed via WebSocket (for clients that prefer ws). + +**Files** +- new: `backend/src/api/ws.ts` — uses `ws` package. +- edit: server.ts — attach ws server to httpServer at `/ws`. + +**API contract** +``` +ws://host/ws/agent +ws://host/ws/logs +ws://host/ws/mempool +ws://host/ws/forks +``` + +**Acceptance** +- [ ] Each ws path delivers same events as the SSE counterpart. + +**Verification** +- `wscat -c ws://...`. + +--- + +### TASK-172 — Socket.io rooms per address + +**Section:** api +**Effort:** M +**Depends on:** none +**Type:** edit + +**Goal** +Allow clients to subscribe to `/socket.io` and join a room per address; receive only events touching that address. + +**Files** +- edit: server.ts socket.io setup. + +**Implementation sketch** +- On `connection`: `socket.on('subscribe', addr => socket.join(`addr:${addr}`))`. +- When a tx is mined or balance changes for addr, emit to `addr:${addr}` room. + +**Acceptance** +- [ ] Subscriber gets events only for their address. + +**Verification** +- Two browsers, one subscribed; receive only own. + +--- + +### TASK-173 — SSE event replay since cursor + +**Section:** api +**Effort:** M +**Depends on:** none +**Type:** edit + +**Goal** +SSE clients that reconnect with `Last-Event-ID` should receive missed events. + +**Files** +- edit: SSE handlers. + +**Implementation sketch** +- Buffer last 1000 events per channel. +- On reconnect, replay events with id > Last-Event-ID, then resume live. + +**Acceptance** +- [ ] Reconnect after disconnect → no missed events (within buffer). + +**Verification** +- Drop SSE, produce events, reconnect, observe. + +--- + +### TASK-174 — GraphQL gateway over REST + +**Section:** api +**Effort:** L +**Depends on:** none +**Type:** new-file + +**Goal** +Single GraphQL endpoint at `/graphql` exposing typed queries over the existing REST surface. + +**Files** +- new: `backend/src/api/graphql/{schema,resolvers,server}.ts`. +- add deps: `graphql`, `graphql-yoga`. + +**Implementation sketch** +- Schema covers: block, tx, account, validator, mempool, logs. +- Resolvers call existing REST handlers internally. + +**Acceptance** +- [ ] Sample query returns expected shape. + +**Verification** +- GraphiQL. + +--- + +### TASK-175 — tRPC endpoint mirror + +**Section:** api +**Effort:** M +**Depends on:** none +**Type:** new-file + +**Goal** +tRPC router mirroring REST so TS clients get end-to-end types. + +**Files** +- new: `backend/src/api/trpc/router.ts`. +- add dep: `@trpc/server`. + +**Implementation sketch** +- Define procedures matching key REST handlers. +- Mount at `/trpc`. + +**Acceptance** +- [ ] TypeScript client gets typed responses. + +**Verification** +- Sample client script. + +--- + +### TASK-176 — JSON-RPC eth_blockNumber + eth_getBalance + +**Section:** api +**Effort:** M +**Depends on:** none +**Type:** new-file + +**Goal** +Minimal Ethereum JSON-RPC compat layer so MetaMask + other tools can connect. + +**Files** +- new: `backend/src/api/jsonrpc.ts` — POST `/rpc` handler dispatching by `method` field. + +**Implementation sketch** +- `eth_blockNumber` → hex of `chain.getChainLength()`. +- `eth_getBalance` → hex of `stateManager.getBalance(addr)`. +- Other methods → `{ error: { code: -32601, message: 'method not found' } }`. + +**Acceptance** +- [ ] MetaMask can connect and read balance. + +**Verification** +- Configure MetaMask custom RPC. + +--- + +### TASK-177 — JSON-RPC eth_call via VM + +**Section:** api +**Effort:** M +**Depends on:** TASK-176, TASK-055 +**Type:** edit + +**Goal** +Implement `eth_call` for read-only contract execution. + +**Files** +- edit: jsonrpc.ts. + +**Implementation sketch** +- Dispatch eth_call → run interpreter against current state with read-only flag. +- Return hex of returndata. + +**Acceptance** +- [ ] eth_call against deployed contract returns expected result. + +**Verification** +- Sample contract call. + +--- + +### TASK-178 — JSON-RPC eth_sendRawTransaction + +**Section:** api +**Effort:** M +**Depends on:** TASK-176 +**Type:** edit + +**Goal** +Accept a signed raw tx and submit to mempool. + +**Files** +- edit: jsonrpc.ts. + +**Implementation sketch** +- Decode raw tx (RLP if EVM-compat, else our format). +- Insert via `txPool.addTransaction`. +- Return tx hash hex. + +**Acceptance** +- [ ] MetaMask can send tx. + +**Verification** +- MetaMask sample tx. + +--- + +### TASK-179 — JSON-RPC subscriptions + +**Section:** api +**Effort:** L +**Depends on:** TASK-176 +**Type:** edit + +**Goal** +`eth_subscribe` and `eth_unsubscribe` over WebSocket for newHeads, logs, newPendingTransactions. + +**Files** +- edit: jsonrpc.ts + ws.ts. + +**Acceptance** +- [ ] Subscribe to newHeads → receives block on each produced. + +**Verification** +- wscat with eth_subscribe. + +--- + +### TASK-180 — Postman/Bruno collection generator + +**Section:** api +**Effort:** S +**Depends on:** TASK-141 +**Type:** script + +**Goal** +Generate a Postman / Bruno collection from openapi.json so non-dev users can poke the API. + +**Files** +- new: `backend/scripts/gen-postman.ts`. + +**Implementation sketch** +- Read openapi.json, walk paths, emit Postman v2.1 collection format. + +**Acceptance** +- [ ] Generated collection imports cleanly into Postman. + +**Verification** +- Import. + +--- + +## Summary + +40 tasks: 23 small, 13 medium, 4 large. Heavy-cluster on JSON-RPC compat (176-179) and observability middleware (146-152). From f127afeecb835cbf672dc4c213179df7c47706d2 Mon Sep 17 00:00:00 2001 From: hermes agent Date: Tue, 28 Apr 2026 01:55:49 +0400 Subject: [PATCH 05/96] =?UTF-8?q?docs(backlog):=20detailed=20specs=20for?= =?UTF-8?q?=20section=2003=20=E2=80=94=20wallet=20&=20accounts=20(TASK-106?= =?UTF-8?q?..140)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 35 task specs: HD wallet derivation (BIP32-style), mnemonic export/import + recovery flows, watch-only mode, multi-sig primitive (m-of-n contract), ENS-like name registry + reverse lookup, wallet alias + contact book, token balance aggregation + approve/transferFrom + allowance + transfer history, faucet hardening (IP rate-limit, hCaptcha hook, dynamic drip, auto-refill), batch send + scheduled send, hardware-key signing protocol stub, session-key delegation, account-abstraction paymaster stub, social recovery (3-of-5 guardians), per-account gas budget cap, password-encrypted wallet export/import, address validity checker, vanity + bulk address generator scripts, wallet metrics endpoint. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/backlog/queue/03-wallet.md | 877 ++++++++++++++++++++++++++++++++ 1 file changed, 877 insertions(+) create mode 100644 docs/backlog/queue/03-wallet.md diff --git a/docs/backlog/queue/03-wallet.md b/docs/backlog/queue/03-wallet.md new file mode 100644 index 00000000..e44ed025 --- /dev/null +++ b/docs/backlog/queue/03-wallet.md @@ -0,0 +1,877 @@ +# Section 03 — Wallet & Accounts Specs (TASK-106..140) + +35 tasks. HD derivation, mnemonic flows, multi-sig, ENS-like names, contact book, token primitives, faucet hardening, batch send, scheduling, hardware-key + session-key delegation, account-abstraction stub, social recovery, encrypted export/import, vanity gen, metrics. + +**Preconditions used throughout:** +- Wallet API: [backend/src/api/wallet.ts](backend/src/api/wallet.ts) — current send/balance/faucet handlers. +- Crypto: [backend/src/blockchain/Crypto.ts](backend/src/blockchain/Crypto.ts) — `generateKeypair`, `derivePublicKey`, `sign`, `verify`, `verifyTransactionSignature`. +- State: [StateManager.ts](backend/src/blockchain/StateManager.ts) — `getBalance`, `getNonce`. +- Tx pool: [TransactionPool.ts](backend/src/blockchain/TransactionPool.ts) — `addTransaction`. +- DB: standard `db.query`. + +--- + +### TASK-106 — HD wallet derivation BIP32-style + +**Section:** wallet +**Effort:** L +**Depends on:** none +**Type:** new-file + +**Goal** +One mnemonic → many addresses via deterministic derivation. Reduces user key-management burden; matches industry conventions. + +**Files** +- new: `backend/src/wallet/hd.ts` — `seedFromMnemonic(mnemonic): Buffer`, `deriveKeypair(seed, path): {pub, priv}`. +- add deps: `bip39`, `ed25519-hd-key`. + +**Implementation sketch** +- `bip39.mnemonicToSeedSync(mnemonic)` → 64-byte seed. +- Path format `m/44'/9999'/0'/0/N` (9999 = our coin type). +- `ed25519-hd-key.derivePath(path, seed.toString('hex')).key` → 32-byte priv. +- Public key via existing `derivePublicKey`. + +**Acceptance** +- [ ] Same mnemonic → same address sequence. +- [ ] Different paths → different addresses. + +**Verification** +- Unit: known mnemonic → known addresses. + +--- + +### TASK-107 — Mnemonic export endpoint + +**Section:** wallet +**Effort:** S +**Depends on:** TASK-106 +**Type:** new-file + +**Goal** +Authenticated wallets can retrieve their mnemonic for backup. Heavily rate-limited. + +**Files** +- new: `POST /api/wallet/:addr/mnemonic/export` (in wallet.ts). + +**Reuses** +- Mnemonic store side (assumes mnemonic was stored encrypted at create-time). +- TASK-351 rate limit (1/min). + +**API contract** +``` +POST /api/wallet/:addr/mnemonic/export +body: { signature: '' } +→ 200 { mnemonic: '...', warning: 'never share' } +→ 401 { error: 'invalid signature' } +``` + +**Acceptance** +- [ ] Valid sig → mnemonic returned. +- [ ] Invalid → 401. + +**Verification** +- Sign + curl. + +--- + +### TASK-108 — Mnemonic import + recovery + +**Section:** wallet +**Effort:** M +**Depends on:** TASK-106 +**Type:** new-file + +**Goal** +User pastes a mnemonic; we derive their addresses + scan for any with on-chain history. + +**Files** +- new: `POST /api/wallet/import` body: `{ mnemonic, scanCount: 20 }`. + +**Implementation sketch** +- Derive `scanCount` addresses. +- For each: query `/api/account/:addr` to get balance + nonce. +- Return list with non-zero or non-zero-nonce addresses flagged as active. + +**Acceptance** +- [ ] Existing mnemonic returns its known addresses. + +**Verification** +- Import a known mnemonic. + +--- + +### TASK-109 — Watch-only address mode + +**Section:** wallet +**Effort:** S +**Depends on:** none +**Type:** new-file + +**Goal** +Users can add an address to track without holding the key. UI shows balance + activity, no send button. + +**Files** +- new: `POST /api/wallet/watch` body: `{ address }`. +- new: `GET /api/wallet/watched/:userKey`. + +**Implementation sketch** +- Per-session list of watched addresses (cookie-keyed) or per-API-key. +- Just metadata; no key storage. + +**Acceptance** +- [ ] Watched address appears in list. + +**Verification** +- Add + read. + +--- + +### TASK-110 — Multi-sig wallet primitive + +**Section:** wallet +**Effort:** L +**Depends on:** TASK-070, TASK-079 +**Type:** new-file + +**Goal** +Deploy an m-of-n multi-sig contract; all signers must approve to send. + +**Files** +- new: `examples/multisig/{source.hsm,program.json,README.md}` (overlaps with TASK-105). +- new: `backend/src/wallet/multisig.ts` — helper to construct & deploy. + +**Implementation sketch** +- Multi-sig stores list of owners + threshold M. +- Tx submission via `propose(target, value, data)` → returns proposal id. +- Approve via `confirm(proposalId, signature)`. +- Execute when M confirmations collected. + +**Acceptance** +- [ ] 2-of-3 multisig: 1 approval insufficient, 2 sufficient. + +**Verification** +- E2E test. + +--- + +### TASK-111 — Wallet name aliases + +**Section:** wallet +**Effort:** S +**Depends on:** none +**Type:** new-file + +**Goal** +Per-user nickname for any address (off-chain, scoped to API key). + +**Files** +- new: `backend/src/database/migrations/0022_wallet_aliases.sql` — `wallet_aliases(api_key_hash, address, alias, PK(api_key_hash, address))`. +- new: CRUD endpoints `GET/POST/DELETE /api/wallet/aliases`. + +**Acceptance** +- [ ] Aliases scoped to user. + +**Verification** +- Add + read. + +--- + +### TASK-112 — ENS-like /api/names/:name resolver + +**Section:** wallet +**Effort:** M +**Depends on:** none +**Type:** new-file + +**Goal** +Global on-chain name → address registry. First-claim wins. + +**Files** +- new: `backend/src/database/migrations/0023_names.sql` — `names(name TEXT PK, address, owner, registered_at, expires_at)`. +- new: `backend/src/api/names.ts` — register, transfer, resolve. + +**API contract** +``` +GET /api/names/:name → 200 { address } +POST /api/names body: { name, signature } → 200 { ok: true } +``` + +**Acceptance** +- [ ] Names resolvable. + +**Verification** +- Register + resolve. + +--- + +### TASK-113 — Reverse name lookup + +**Section:** wallet +**Effort:** S +**Depends on:** TASK-112 +**Type:** new-file + +**Goal** +Address → registered name(s). + +**Files** +- new: `GET /api/names/reverse/:addr` → `{ names: [...] }`. + +**Acceptance** +- [ ] Returns names where address matches. + +**Verification** +- Curl. + +--- + +### TASK-114 — Wallet activity feed + +**Section:** wallet +**Effort:** M +**Depends on:** TASK-058 +**Type:** edit + +**Goal** +Combined send/receive/contract-event timeline for an address. + +**Files** +- edit: `backend/src/api/wallet.ts` — `GET /api/wallet/:addr/activity?cursor=&limit=`. + +**Reuses** +- TASK-058 history; TASK-161 events. + +**API contract** +``` +→ 200 { items: [{ type: 'send'|'receive'|'event', ts, ...details }], next_cursor } +``` + +**Acceptance** +- [ ] Mixed feed returned in time order. + +**Verification** +- Curl. + +--- + +### TASK-115 — CSV export of wallet history + +**Section:** wallet +**Effort:** S +**Depends on:** TASK-114 +**Type:** edit + +**Goal** +Same data as TASK-114 but CSV for spreadsheets/tax tools. + +**Files** +- new: `GET /api/wallet/:addr/activity.csv?from=&to=`. + +**Implementation sketch** +- Stream CSV rows with header. + +**Acceptance** +- [ ] Returns valid CSV. + +**Verification** +- `curl > out.csv && head out.csv`. + +--- + +### TASK-116 — /api/wallet/:addr/qr.png + +**Section:** wallet +**Effort:** S +**Depends on:** none +**Type:** new-file + +**Goal** +QR code PNG for an address (for receive screens). + +**Files** +- new: `GET /api/wallet/:addr/qr.png?size=256`. +- add dep: `qrcode`. + +**Implementation sketch** +- `QRCode.toBuffer(addr, { width: size })` → res. + +**Acceptance** +- [ ] PNG returned with correct content-type. + +**Verification** +- Curl, view image. + +--- + +### TASK-117 — Wallet contact book + +**Section:** wallet +**Effort:** S +**Depends on:** TASK-111 +**Type:** new-file + +**Goal** +Per-user contact list (alias + address + notes). + +**Files** +- new: migration `wallet_contacts` table. +- new: CRUD endpoints. + +**Acceptance** +- [ ] Add + list contacts. + +**Verification** +- Curl. + +--- + +### TASK-118 — Token balance aggregation + +**Section:** wallet +**Effort:** M +**Depends on:** TASK-105 (erc20-like example) +**Type:** new-file + +**Goal** +Across all deployed token contracts, return user's holdings. + +**Files** +- new: `GET /api/wallet/:addr/tokens` → `{ tokens: [{ contractAddress, symbol, balance }] }`. + +**Implementation sketch** +- Query `contract_metadata` for contracts with `symbol` field. +- For each, run a balance read against `contract_storage`. + +**Acceptance** +- [ ] Lists all balances > 0. + +**Verification** +- Curl. + +--- + +### TASK-119 — Approve / transferFrom flow for tokens + +**Section:** wallet +**Effort:** S +**Depends on:** TASK-105 +**Type:** docs + helper + +**Goal** +Standard ERC20-like allowance flow. Document in /docs and provide helper endpoints that build the txs. + +**Files** +- new: `POST /api/wallet/token/:contract/approve` body: `{spender, amount, signature}` → constructed tx. + +**Acceptance** +- [ ] Approve + transferFrom round-trip works. + +**Verification** +- Two-tx test. + +--- + +### TASK-120 — Allowance lookup endpoint + +**Section:** wallet +**Effort:** S +**Depends on:** TASK-119 +**Type:** new-file + +**Goal** +Read `allowance(owner, spender)` from a token contract. + +**Files** +- new: `GET /api/wallet/token/:contract/allowance/:owner/:spender`. + +**Acceptance** +- [ ] Returns numeric allowance. + +**Verification** +- Curl. + +--- + +### TASK-121 — Token transfer history per-account + +**Section:** wallet +**Effort:** S +**Depends on:** TASK-114 +**Type:** new-file + +**Goal** +Filter activity feed to just token transfers (Transfer event topic). + +**Files** +- new: `GET /api/wallet/:addr/token-history?contract=`. + +**Acceptance** +- [ ] Returns Transfer events involving this address. + +**Verification** +- Curl. + +--- + +### TASK-122 — Faucet rate-limit by IP not just address + +**Section:** wallet +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Current faucet limits per address (24h cooldown). Also limit by IP to prevent address-rotation abuse. + +**Files** +- edit: `backend/src/api/wallet.ts` faucet handler. + +**Implementation sketch** +- Track `faucet_ip_drips(ip, last_drip_at, count_24h)` in Redis with TTL. +- Cap: 5 drips per IP per 24h. + +**Acceptance** +- [ ] Same IP across 6 different addresses → 6th rejected. + +**Verification** +- Loop curl from same IP. + +--- + +### TASK-123 — Faucet captcha hook + +**Section:** wallet +**Effort:** M +**Depends on:** TASK-122 +**Type:** edit + +**Goal** +Optional hCaptcha verification before faucet drip. + +**Files** +- edit: faucet handler. +- add dep: `hcaptcha` (or fetch directly). + +**Implementation sketch** +- If `HCAPTCHA_SECRET` env set: require `captchaToken` field, verify against hCaptcha. +- Else: skip (dev/staging). + +**Acceptance** +- [ ] Without token (when configured): rejected. +- [ ] Valid token: passes. + +**Verification** +- Test in staging. + +--- + +### TASK-124 — Faucet drip dynamic amount + +**Section:** wallet +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Adjust drip size based on demand (queue depth) and pool reserves. + +**Files** +- edit: faucet handler. + +**Implementation sketch** +- Base = 100 OPEN. +- If pool reserves < 1000 OPEN: drip /= 4. +- If queue length > 100/min: drip /= 2. + +**Acceptance** +- [ ] Drip amount reduces under low reserves. + +**Verification** +- Drain pool, check next drip. + +--- + +### TASK-125 — Faucet pool refill schedule + +**Section:** wallet +**Effort:** S +**Depends on:** none +**Type:** new-file + +**Goal** +Auto-refill faucet pool from a treasury address on a schedule. + +**Files** +- new: `backend/src/wallet/faucetRefiller.ts` — interval that monitors pool, sends from treasury when below threshold. + +**Implementation sketch** +- Threshold via env (default 5000 OPEN). +- Refill amount: 50000 OPEN at a time. +- Treasury private key from env (sealed). + +**Acceptance** +- [ ] Pool refills on schedule. + +**Verification** +- Drop pool below threshold, observe. + +--- + +### TASK-126 — Wallet send batch (many recipients) + +**Section:** wallet +**Effort:** M +**Depends on:** TASK-169 +**Type:** new-file + +**Goal** +Single signed payload sends to N recipients (one tx per recipient). + +**Files** +- new: `POST /api/wallet/send-batch` body: `{ from, recipients: [{to, amount}], signature }`. + +**Implementation sketch** +- Verify signature over canonical message. +- Construct N txs with sequential nonces. +- Use bulk submit (TASK-169). + +**Acceptance** +- [ ] All txs accepted with correct nonces. + +**Verification** +- Send to 3 recipients in one call. + +--- + +### TASK-127 — Tx scheduling (broadcast at future height) + +**Section:** wallet +**Effort:** M +**Depends on:** none +**Type:** new-file + +**Goal** +Submit a signed tx now to be broadcast when chain reaches height H. + +**Files** +- new: migration `scheduled_txs(hash PK, target_height, payload_json, status, scheduled_by)`. +- new: `POST /api/wallet/schedule` body: `{ tx, height }`. +- new: scheduler interval that submits when height matches. + +**Acceptance** +- [ ] Tx scheduled at height N executes at or after N. + +**Verification** +- Schedule + wait. + +--- + +### TASK-128 — Tx replacement UI flow (cancel-by-replace) + +**Section:** wallet +**Effort:** S +**Depends on:** TASK-168 +**Type:** docs + +**Goal** +Document the cancel flow + provide a one-call helper. + +**Files** +- new: `docs/wallet/cancel-tx.md`. + +**Acceptance** +- [ ] Doc walks through with curl examples. + +**Verification** +- Manual. + +--- + +### TASK-129 — Hardware-key signing protocol stub + +**Section:** wallet +**Effort:** M +**Depends on:** none +**Type:** new-file + +**Goal** +Stub protocol for Ledger-style flow: backend constructs unsigned tx, sends to client, client signs on hw device, returns sig, backend submits. + +**Files** +- new: `backend/src/wallet/hardwareSigning.ts`. +- new: docs `/docs/wallet/hardware-keys.md`. + +**Implementation sketch** +- `POST /api/wallet/sign-request` body: `{ from, to, value }` → `{ unsignedTx, message }`. +- Client side responsibility: sign on device. +- `POST /api/wallet/sign-submit` body: `{ unsignedTx, signature }`. + +**Acceptance** +- [ ] Stub round-trip works with software signer simulating hw. + +**Verification** +- Dev test. + +--- + +### TASK-130 — Session key delegation + +**Section:** wallet +**Effort:** M +**Depends on:** none +**Type:** new-file + +**Goal** +Sign once with master key to authorize a session key with scoped permissions (max value, expiry). + +**Files** +- new: migration `session_keys(id, master_address, session_pubkey, max_value, expires_at, signature)`. +- new: middleware that accepts session-key signatures for txs within scope. + +**Acceptance** +- [ ] Tx signed by session key within scope: accepted. +- [ ] Out-of-scope: rejected. + +**Verification** +- Dev test. + +--- + +### TASK-131 — Account abstraction stub: paymaster + +**Section:** wallet +**Effort:** M +**Depends on:** TASK-079 +**Type:** new-file + +**Goal** +Allow gasless tx: paymaster contract pays the gas on behalf of the sender. + +**Files** +- new: `examples/paymaster/{source.hsm,program.json}`. +- edit: BlockProducer to accept paymaster-stamped txs (charge fee to paymaster, not sender). + +**Acceptance** +- [ ] Tx with valid paymaster authorization: sender pays nothing. + +**Verification** +- E2E test. + +--- + +### TASK-132 — Wallet-side mempool view + +**Section:** wallet +**Effort:** S +**Depends on:** TASK-166 +**Type:** new-file + +**Goal** +Filter `/api/mempool` to txs from/to a specific address. + +**Files** +- new: `GET /api/wallet/:addr/pending`. + +**Acceptance** +- [ ] Returns only txs touching this address. + +**Verification** +- Curl after submit. + +--- + +### TASK-133 — Wallet recovery via social guardians + +**Section:** wallet +**Effort:** L +**Depends on:** TASK-110 +**Type:** new-file + +**Goal** +3-of-5 friends approve a recovery to swap the master key on an account. + +**Files** +- new: `examples/social-recovery/...`. +- new: `POST /api/wallet/recovery/setup` body: `{guardians: [...]}`. +- new: `POST /api/wallet/recovery/initiate` body: `{newKey, signatures}`. + +**Implementation sketch** +- Recovery contract holds the guardian set. +- M-of-N approve a `setKey(newPubKey)` call. +- Time-lock 24h before swap takes effect. + +**Acceptance** +- [ ] Recovery flow works. + +**Verification** +- E2E test. + +--- + +### TASK-134 — Per-account gas budget cap + +**Section:** wallet +**Effort:** S +**Depends on:** none +**Type:** new-file + +**Goal** +Account-level cap on gas burn per 24h. Prevents runaway scripts from draining. + +**Files** +- new: migration `account_gas_caps(address PK, max_24h, used_24h, period_started)`. +- edit: TransactionPool — reject if cap would exceed. + +**Acceptance** +- [ ] Tx beyond cap: rejected. + +**Verification** +- Set low cap, attempt tx. + +--- + +### TASK-135 — Wallet password-encrypted export + +**Section:** wallet +**Effort:** M +**Depends on:** TASK-107 +**Type:** new-file + +**Goal** +Export wallet (mnemonic + addresses) as JSON encrypted with PBKDF2 + AES-256-GCM keyed by user password. + +**Files** +- new: `backend/src/wallet/encryptedExport.ts` — `exportEncrypted(walletData, password)`, `decryptImport(blob, password)`. + +**Implementation sketch** +- PBKDF2-SHA256, 100k iterations, 32-byte key. +- AES-256-GCM with random 12-byte IV. +- JSON envelope: `{ kdf: 'pbkdf2', iterations, salt, iv, ciphertext, tag }`. + +**Acceptance** +- [ ] Round-trip with correct password. +- [ ] Wrong password → decrypt failure. + +**Verification** +- Unit. + +--- + +### TASK-136 — Wallet import from JSON + +**Section:** wallet +**Effort:** S +**Depends on:** TASK-135 +**Type:** edit + +**Goal** +Accept the encrypted JSON, decrypt, derive addresses. + +**Files** +- new: `POST /api/wallet/import-encrypted` body: `{ blob, password }`. + +**Acceptance** +- [ ] Successful decrypt restores wallet. + +**Verification** +- Round-trip. + +--- + +### TASK-137 — Address validity checker endpoint + +**Section:** wallet +**Effort:** S +**Depends on:** none +**Type:** new-file + +**Goal** +Pre-flight check: is this string a valid Hermes address? (base58 + length + checksum). + +**Files** +- new: `GET /api/wallet/validate/:input`. + +**API contract** +``` +→ 200 { valid: true|false, reason?: 'bad-base58'|'wrong-length'|'bad-checksum' } +``` + +**Acceptance** +- [ ] Valid address: true. +- [ ] Garbage: false with reason. + +**Verification** +- Curl. + +--- + +### TASK-138 — Vanity address generator script + +**Section:** wallet +**Effort:** S +**Depends on:** none +**Type:** script + +**Goal** +Given a prefix, brute-force keypairs until address starts with prefix. CLI. + +**Files** +- new: `backend/scripts/vanity-address.ts`. + +**Implementation sketch** +- Loop generateKeypair, check prefix, count attempts, print on match. +- `--prefix Hermes --max 1000000`. + +**Acceptance** +- [ ] Returns address with the prefix. + +**Verification** +- `npm run vanity -- --prefix abc`. + +--- + +### TASK-139 — Bulk address generator for testing + +**Section:** wallet +**Effort:** S +**Depends on:** none +**Type:** script + +**Goal** +Generate N keypairs at once, output JSON for test fixtures. + +**Files** +- new: `backend/scripts/bulk-keys.ts`. + +**Acceptance** +- [ ] Outputs N pairs. + +**Verification** +- Run with N=100. + +--- + +### TASK-140 — Wallet metrics endpoint + +**Section:** wallet +**Effort:** S +**Depends on:** none +**Type:** new-file + +**Goal** +Aggregate counts for ops dashboard. + +**Files** +- new: `GET /api/wallet/metrics` → `{ total, active_24h, new_24h, send_volume_24h }`. + +**Acceptance** +- [ ] Returns sensible numbers. + +**Verification** +- Curl. + +--- + +## Summary + +35 tasks: 21 small, 11 medium, 3 large. Heavier-cluster around HD/multi-sig/social recovery (106, 110, 133); rest mostly endpoints + helpers. From d80eecef2d0c95ed08f21c85a41de3d9ddce2aee Mon Sep 17 00:00:00 2001 From: hermes agent Date: Tue, 28 Apr 2026 01:57:30 +0400 Subject: [PATCH 06/96] =?UTF-8?q?docs(backlog):=20detailed=20specs=20for?= =?UTF-8?q?=20section=2009=20=E2=80=94=20security=20(TASK-336..370)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 35 task specs: CSRF + Helmet + body-size + per-endpoint rate limit overrides, SQL-injection audit script, input length caps, API-key scoping (chain:read/ write etc) + 90d expiry + rotation + audit log, admin-token rotation flow, failed-auth IP lockout (5/min → 15min block) + suspicious feed, tx replay protection (chainId enforcement + nonce window tightening), wallet-export rate limit + obscured mnemonic display, log redaction wrapper for keys/sigs/mnemonics, CI hooks (gitleaks, npm audit, CodeQL, Snyk), CSP nonces + SRI on CDN scripts, HTTPS-only redirect, SameSite=Strict cookies, session-fixation defense, password-strength meter, encryption-at-rest for mnemonics + KMS integration stub, Tor/VPN flag, ip2geo on auth log, threat-feed (AbuseIPDB/Spamhaus) IP blocking, cert-pinning notes, disclosure-response template. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/backlog/queue/09-security.md | 791 ++++++++++++++++++++++++++++++ 1 file changed, 791 insertions(+) create mode 100644 docs/backlog/queue/09-security.md diff --git a/docs/backlog/queue/09-security.md b/docs/backlog/queue/09-security.md new file mode 100644 index 00000000..7ce1e69e --- /dev/null +++ b/docs/backlog/queue/09-security.md @@ -0,0 +1,791 @@ +# Section 09 — Security Specs (TASK-336..370) + +35 tasks. CSRF + Helmet + body-size + rate-limit overrides, API-key scoping/expiry/rotation/audit, admin-token rotation, lockout + suspicious feed, replay protection (chainId + nonce), wallet-export rate limit + obscured display, log redaction, secrets scan + dep audit + CodeQL + Snyk, CSP nonces + SRI, HTTPS-only + cookie flags, mnemonic encryption at rest, KMS stub, Tor/VPN flag, ip2geo, threat-feed, cert-pinning notes, disclosure response template. + +**Preconditions used throughout:** +- Auth: [backend/src/api/auth.ts](backend/src/api/auth.ts) — `requireApiKey`, `ipRateLimit`. +- Crypto helpers in [Crypto.ts](backend/src/blockchain/Crypto.ts). +- Server middleware stack in [server.ts](backend/src/api/server.ts). + +--- + +### TASK-336 — CSRF protection on POST endpoints + +**Section:** security +**Effort:** M +**Depends on:** none +**Type:** new-file + +**Goal** +State-changing routes need CSRF protection. Issue per-session token; reject mismatched. + +**Files** +- new: `backend/src/api/middleware/csrf.ts`. +- edit: server.ts — apply to all non-API-key POST routes. + +**Implementation sketch** +- Double-submit cookie pattern: server sets `csrf-token` cookie + expects `X-CSRF-Token` header to match. +- API-key authenticated routes exempt (token-based already). + +**Acceptance** +- [ ] POST without token: 403. +- [ ] With matching token: passes. + +**Verification** +- Curl with/without header. + +--- + +### TASK-337 — Helmet middleware (CSP, HSTS, frameguard) + +**Section:** security +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Standard security headers via `helmet`. + +**Files** +- edit: server.ts — `app.use(helmet({...}))`. +- add dep: `helmet`. + +**Implementation sketch** +- Default config + custom CSP allowing self + Sentry + analytics. +- HSTS max-age 1 year, includeSubDomains. + +**Acceptance** +- [ ] `curl -I /` shows X-Frame-Options, Strict-Transport-Security, Content-Security-Policy. + +**Verification** +- Inspect headers. + +--- + +### TASK-338 — SQL-injection audit pass + +**Section:** security +**Effort:** M +**Depends on:** none +**Type:** edit + +**Goal** +Audit every `db.query` callsite for string concatenation; convert any to parameterized. + +**Files** +- new: `backend/scripts/audit-sql-injection.ts` — grep `db.query` callsites for `+ req.` or template string with `${req.`. +- edit: any flagged callsites. + +**Acceptance** +- [ ] Audit script returns 0 hits. + +**Verification** +- Run script. + +--- + +### TASK-339 — Input length caps on every endpoint + +**Section:** security +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Per-field max length to block oversized inputs. + +**Files** +- new: `backend/src/api/middleware/inputCaps.ts`. + +**Implementation sketch** +- Generic middleware checking `req.body` field by field against schema (use zod or hand-rolled). +- Reject 413 with field name on overflow. + +**Acceptance** +- [ ] 10MB string in any field → 413. + +**Verification** +- Curl with huge field. + +--- + +### TASK-340 — JSON body size limit + +**Section:** security +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Cap total JSON body to 1MB via express.json options. + +**Files** +- edit: server.ts:81 — `express.json({ limit: '1mb' })`. + +**Acceptance** +- [ ] 2MB body: 413. + +**Verification** +- Curl. + +--- + +### TASK-341 — Per-endpoint rate limit overrides + +**Section:** security +**Effort:** S +**Depends on:** TASK-144 +**Type:** edit + +**Goal** +Different routes need different rates. `ipRateLimit(60)` for chat vs `ipRateLimit(600)` for /api/blocks. + +**Files** +- edit: auth.ts — already supports per-call rate; document + apply selectively in server.ts. + +**Acceptance** +- [ ] Chat limited to 60/min; reads to 600/min. + +**Verification** +- Curl loops. + +--- + +### TASK-342 — API-key scope chain:write vs chain:read + +**Section:** security +**Effort:** M +**Depends on:** none +**Type:** edit + +**Goal** +Currently API keys have boolean permissions. Add granular scopes: `chain:read`, `chain:write`, `wallet:send`, `keys:create`, `jobs:write`, `admin`. + +**Files** +- new: migration adding `permissions` JSONB column to `api_keys`. +- edit: `requireApiKey(scope)` middleware to check scope inclusion. + +**Acceptance** +- [ ] Read-scope key: blocked from write routes. + +**Verification** +- Mint key with read-only, attempt write. + +--- + +### TASK-343 — API-key expiry default 90d + +**Section:** security +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +New keys expire 90 days from creation unless overridden. + +**Files** +- edit: `POST /auth/keys` to set `expires_at`. + +**Acceptance** +- [ ] Expired keys: rejected with 401. + +**Verification** +- Backdate, attempt use. + +--- + +### TASK-344 — API-key rotation endpoint + +**Section:** security +**Effort:** S +**Depends on:** TASK-343 +**Type:** new-file + +**Goal** +Rotate a key in place: same permissions, new secret, old gracefully expires in 24h. + +**Files** +- new: `POST /auth/keys/:id/rotate` → returns new secret. + +**Acceptance** +- [ ] Old key still valid for 24h; new key works immediately. + +**Verification** +- Rotate + use both for 24h. + +--- + +### TASK-345 — API-key audit log + +**Section:** security +**Effort:** S +**Depends on:** none +**Type:** new-file + +**Goal** +Log every key creation, rotation, deletion to a tamper-evident audit table. + +**Files** +- new: migration `api_key_audit(id, key_id, action, actor, metadata, occurred_at)`. +- edit: auth router to write audit row on every key change. + +**Acceptance** +- [ ] All key changes appear in audit log. + +**Verification** +- Mint + delete key, query audit. + +--- + +### TASK-346 — Admin-token rotation flow + +**Section:** security +**Effort:** S +**Depends on:** none +**Type:** docs + +**Goal** +Document procedure for rotating `ADMIN_TOKEN` env without downtime. + +**Files** +- new: `docs/security/admin-token-rotation.md`. + +**Implementation sketch** +- Set `ADMIN_TOKEN_SECONDARY` to new value; both work. +- Restart with new as primary; remove old after deploy. + +**Acceptance** +- [ ] Doc exists with step-by-step. + +**Verification** +- Manual. + +--- + +### TASK-347 — Failed-auth lockout (5 in 1min → 15min block) + +**Section:** security +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Brute-force defense: 5 failures in 1 min from same IP → 15 min block. + +**Files** +- edit: auth.ts. + +**Implementation sketch** +- Redis counter `auth:fail:${ip}` with 60s TTL. +- On count > 5: set `auth:block:${ip}` with 900s TTL; reject with 429. + +**Acceptance** +- [ ] 6th failure within minute: blocked for 15 min. + +**Verification** +- Loop bad creds. + +--- + +### TASK-348 — Suspicious-activity feed + +**Section:** security +**Effort:** S +**Depends on:** TASK-347 +**Type:** new-file + +**Goal** +Stream auth failures, rate-limit hits, blocked IPs to ops channel. + +**Files** +- new: `GET /api/security/suspicious` (admin-gated). + +**API contract** +``` +→ 200 { items: [{ type, ip, reason, ts }] } +``` + +**Acceptance** +- [ ] Returns recent suspicious events. + +**Verification** +- Curl after triggering events. + +--- + +### TASK-349 — Tx replay protection: chainId + +**Section:** security +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Tx must include chainId; reject mismatched. Prevents cross-chain replay. + +**Files** +- edit: TransactionPool.validateTransaction — check `tx.chainId === HERMES_CHAIN_ID`. + +**Acceptance** +- [ ] Wrong chainId: rejected. + +**Verification** +- Submit with wrong id. + +--- + +### TASK-350 — Tx replay protection: per-key nonce window tightening + +**Section:** security +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +Currently nonce window is +10 from current. Tighten to +5; reject txs outside. + +**Files** +- edit: TransactionPool.validateTransaction. + +**Acceptance** +- [ ] Nonce > current+5: rejected. + +**Verification** +- Submit far-future nonce. + +--- + +### TASK-351 — Wallet-export rate limit 1/min + +**Section:** security +**Effort:** S +**Depends on:** TASK-107 +**Type:** edit + +**Goal** +Mnemonic export at most once per minute per address. + +**Files** +- edit: TASK-107 endpoint. + +**Acceptance** +- [ ] Second export within 60s: 429. + +**Verification** +- Two exports in row. + +--- + +### TASK-352 — Mnemonic display obscured + reveal button + +**Section:** security +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +HUD shows mnemonic as `••••••• ••••••• ...`; click-to-reveal. + +**Files** +- edit: relevant frontend component (TASK-231 area). + +**Acceptance** +- [ ] Mnemonic hidden by default in UI. + +**Verification** +- Visual. + +--- + +### TASK-353 — Server log redaction + +**Section:** security +**Effort:** M +**Depends on:** none +**Type:** new-file + +**Goal** +Wrap `console.log/warn/error` to scrub: API keys, signatures, mnemonics, private keys, JWT. + +**Files** +- new: `backend/src/utils/safeLog.ts` — exports `safeLog.{info,warn,error}` that scrubs. + +**Implementation sketch** +- Regex set: `/sk_[a-zA-Z0-9]{32,}/`, `/[A-Za-z0-9+/]{86}=/` (base58 sig length), `/(?:\b\w+\s){11}\w+/` (potential mnemonic). +- Replace matches with `[REDACTED]`. + +**Acceptance** +- [ ] Log line containing API key emerges with `[REDACTED]`. + +**Verification** +- Unit. + +--- + +### TASK-354 — Secrets scan in CI (gitleaks) + +**Section:** security +**Effort:** S +**Depends on:** none +**Type:** new-file + +**Goal** +Add gitleaks to CI workflow; block PRs that introduce secrets. + +**Files** +- new: `.github/workflows/secrets-scan.yml`. + +**Acceptance** +- [ ] PR with synthetic secret: scan fails. + +**Verification** +- Test PR. + +--- + +### TASK-355 — Dependency audit on PR (npm audit) + +**Section:** security +**Effort:** S +**Depends on:** none +**Type:** new-file + +**Goal** +CI step `npm audit --audit-level=high` blocking on high+ vulnerabilities. + +**Files** +- new: `.github/workflows/npm-audit.yml`. + +**Acceptance** +- [ ] PR introducing vulnerable dep: fails. + +**Verification** +- Manual. + +--- + +### TASK-356 — CodeQL workflow on push + +**Section:** security +**Effort:** S +**Depends on:** none +**Type:** new-file + +**Goal** +GitHub CodeQL static analysis weekly + on push. + +**Files** +- new: `.github/workflows/codeql.yml`. + +**Acceptance** +- [ ] CodeQL runs on push. + +**Verification** +- Push, view Actions. + +--- + +### TASK-357 — Snyk integration + +**Section:** security +**Effort:** S +**Depends on:** none +**Type:** new-file + +**Goal** +Snyk dependency + container scan on PR. + +**Files** +- new: `.github/workflows/snyk.yml`. + +**Acceptance** +- [ ] Workflow runs. + +**Verification** +- View Actions. + +--- + +### TASK-358 — CSP nonce per request + +**Section:** security +**Effort:** M +**Depends on:** TASK-337 +**Type:** edit + +**Goal** +Inline scripts must carry per-request nonce matching CSP header. + +**Files** +- edit: server.ts middleware to set `res.locals.cspNonce` and pass into HTML responses. +- edit: helmet config to use the nonce. + +**Acceptance** +- [ ] No unsafe-inline in CSP. + +**Verification** +- Inspect headers + source. + +--- + +### TASK-359 — Subresource integrity on CDN + +**Section:** security +**Effort:** S +**Depends on:** none +**Type:** edit + +**Goal** +External script tags include `integrity=` SRI hashes. + +**Files** +- edit: HTML templates. + +**Acceptance** +- [ ] All external `