You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
- Nicolas Rinaudo, “Effects as Capabilities”: https://nrinaudo.github.io/articles/capabilities.html
42
+
- Nicolas Rinaudo, “Hands‑on Capture Checking”: https://nrinaudo.github.io/articles/capture_checking.html
43
+
44
+
---
45
+
46
+
## Related Work & Alignment (Post‑Reading Notes)
47
+
48
+
- Capabilities for control flow (Nicolas Rinaudo): https://nrinaudo.github.io/articles/capabilities_flow.html
49
+
- Takeaway: model special powers (labels/boundaries) as capabilities that must not escape their scope.
50
+
- Relevance: identical non‑escape discipline applies to data/IO resources (connectors, sessions, handles). Our goal is to ensure such capabilities cannot leak into pure UDFs or lazy structures.
51
+
52
+
- Effects as Capabilities (N. Rinaudo): https://nrinaudo.github.io/articles/capabilities.html
53
+
- Takeaway: express required effects/resources as context functions (direct‑style) rather than monads, keeping execution requirements explicit and composable.
54
+
- Relevance: informs an optional Scala 3 facade to declare engine/connectors as required capabilities without changing core runtime.
55
+
56
+
- Direct‑style Effects (N. Welsh): https://noelwelsh.com/posts/direct-style/
57
+
- Takeaway: direct‑style APIs with effect handlers/context functions can improve ergonomics while preserving separation of description vs. action.
58
+
- Relevance: we can prototype a facade using context functions alongside our existing tagless‑final API; this remains optional and non‑blocking.
59
+
60
+
Conclusion: the article validates our direction — use Scala 3 capture checking to enforce non‑escaping capabilities and pure arrows (`A -> B`) for UDFs.
61
+
62
+
## ADR Amendments (No deletions; clarifications and additions)
63
+
64
+
### 1) Terminology & Surface Types
65
+
66
+
- Adopt Scala 3 terminology explicitly:
67
+
- Pure functions use the pure arrow: `A -> B` (non‑capturing).
68
+
- Tracked capabilities use the caret type: `C^`.
69
+
- Keep existing aliases (for readability) but document their mapping:
70
+
-`type PureFn[-A,+B] = A -> B` (already planned — reaffirmed).
71
+
72
+
### 2) API Shapes Informed by Capabilities Article
73
+
74
+
Add the following opt‑in APIs (names stable; semantics experimental):
75
+
76
+
```scala
77
+
// Scope a capability so it cannot escape; pure by construction
// Pipeline builder addition: enforce purity at the type level for internal transforms
84
+
defpureTransform[A, B](name: String)(f: A->B):PipelineBuilder[WithTransform, F, In, B]
85
+
```
86
+
87
+
Design notes:
88
+
-`withCapability` mirrors the boundary/break model: callers get a scoped power (`C^`), but cannot store or return it.
89
+
-`pureTransform` makes “pure inside, effects at the edges” the default for UDFs.
90
+
91
+
### 3) Laziness & Non‑Escape (Caveats)
92
+
93
+
Pitfalls to prevent (mirroring the article’s Iterator example):
94
+
- Returning closures that capture `C^` from a `withCapability` region.
95
+
- Storing `C^` in a field of an object that outlives the region.
96
+
- Building lazy collections/streams that reference `C^` (evaluation may occur after the region closes).
97
+
98
+
Mitigations (compile‑time):
99
+
- Let capture checking reject escaping `C^` in the cases above.
100
+
- Provide guidance to prefer eager, total transformations in `A -> B`; if laziness is required, ensure all use happens inside the capability scope.
101
+
102
+
### 4) Error Message Guidance (DX)
103
+
104
+
When capture checking rejects a program, aim for messages of the form:
105
+
106
+
```
107
+
Caprese: capability C^ escapes its scope
108
+
• captured in closure returned from `withCapability` at Foo.scala:42
109
+
• referenced by lazy value `it` evaluated outside scope
110
+
Hint: compute eagerly inside `withCapability { (c: C^) => ... }` and return plain values (A -> B).
111
+
```
112
+
113
+
### 5) Interop & Incremental Adoption
114
+
115
+
- Interop: allow existing `A => B` transforms to coexist; `pureTransform` is opt‑in.
116
+
- Escape hatch (temporary): an explicit, scoped suppression annotation (e.g., `@capreseUnsafeEscape`) for code that cannot be rewritten immediately. Not for production paths; tracked in CI.
117
+
118
+
### 6) Open Questions to Validate in POC
119
+
120
+
- False positives/negatives around laziness (Iterators, Streams, fs2/ZIO streams).
121
+
- Ergonomics: can we keep ceremony low for common UDFs (does `A -> B` feel natural for teams)?
122
+
- Tooling: scalafix lints to recommend `pureTransform` for obvious pure lambdas.
123
+
124
+
### 6.1) POC Hardening Plan (Realistic, Value‑Add)
125
+
126
+
- Tests that must fail compilation (ensure capture checking works in practice):
127
+
1. Returning `C^` from `withCapability`.
128
+
2. Storing `C^` in an object that outlives the scope (val/field).
129
+
3. Creating lazy collections/streams (Iterator/Stream/fs2/ZIO) that capture `C^` then evaluate outside scope.
130
+
4. Writing a `pureTransform` that closes over an IO handle (should be rejected).
131
+
132
+
- Tests that must pass:
133
+
1.`A -> B` transforms with no captures; composition of multiple pure transforms.
134
+
2.`withCapability` used to compute a plain `A` result that does not leak `C^`.
135
+
3. Interop: `A => B` transforms continue to work (without purity guarantees).
136
+
137
+
- DX checks:
138
+
- Error message clarity: include escape site and hint (see Error Message Guidance above).
139
+
- Scalafix lint (advisory): suggest `pureTransform` when lambda is syntactically pure.
140
+
141
+
- Deliverables (time‑boxed):
142
+
- Experimental Scala 3 module (exists: experimental‑caprese) with CI task to compile both “good” and “bad” examples.
143
+
- Short migration note for authors (how to move `A => B` to `A -> B`).
- No regressions in performance or operator ergonomics.
189
+
190
+
### 9) Traceability
191
+
192
+
- This amendment was informed by: “Capabilities and Control Flow in Scala”, Nicolas Rinaudo (link above). The non‑escape discipline and laziness caveats map directly to our goals for pure UDFs and scoped resources.
- A design approach for data pipelines in Scala where schema contracts are enforced at compile time and orchestration respects effect/fiber safety at runtime.
5
+
- Business logic stays pure and testable; IO is explicit and resource‑safe.
6
+
7
+
- Why it matters (outcomes)
8
+
- Fewer incidents: Contract drift blocked before deployment (compile gate), reducing data quality outages.
- Higher developer velocity: Pure transformations unit‑test in milliseconds; fewer flaky E2E tests.
11
+
- Portability: Same pipeline logic runs on multiple engines (e.g., Spark/Flink) via a trait‑based runner.
12
+
- Compliance & governance: Typed contracts + policy variants encode intent and enforce via CI.
13
+
14
+
- How it works (high level)
15
+
- Compile‑time: Case classes → Magnolia Shape → Schema AST → policy compare → compile success or fail.
16
+
- Runtime: Pipelines are Kleisli graphs executed with a fiber‑aware effect system (Cats‑Effect/ZIO), with explicit resource safety.
17
+
18
+
- ROI levers (example targets over 6–12 months)
19
+
- 50–80% reduction in schema‑related incidents in batch/stream pipelines.
20
+
- 30–50% reduction in E2E test runtime by shifting to pure unit tests for inner transforms.
21
+
- 20–40% faster onboarding due to templates and policy‑driven guardrails.
22
+
- 25–40% fewer ad‑hoc hotfixes caused by unplanned contract changes.
23
+
24
+
- Costs and risks
25
+
- Upfront learning: Team needs to learn the idioms (phantom types, type classes, Kleisli, effect systems).
26
+
- Template/CI adoption: Requires build and CI wiring to enforce compile gates.
27
+
- Integration work: Engine adapters (Kafka/Spark/Flink) and DQ preferences (native vs Deequ) must be chosen per team.
28
+
29
+
- Risk mitigations
30
+
- Start with one golden path template; demonstrate red→green contract fixes in CI.
31
+
- Pick a single effect system per service (IO or ZIO) to limit cognitive load.
32
+
- Phase policies: begin with Exact for critical interfaces, use Backward/Forward during migrations.
33
+
34
+
- KPIs to track
35
+
- Contract drift incidents per quarter; MTTR for data breakages; test runtime; percentage of pipelines on the template; change failure rate for schema‑touching PRs.
36
+
37
+
- Adoption plan (90 days)
38
+
- Weeks 1–2: Pilot one pipeline; wire compile‑fail tests and CI policy gates.
0 commit comments