Skip to content

Commit cdc5c92

Browse files
anconguiancongui
committed
Merge feature/starter-data-evolution: bug fixes, new features, GenAI bridge, and documentation
Bug Fixes: - Fix DataSizeCalculator locale bug (Locale.US instead of es_ES) - Fix cache eviction to use prefix-based eviction instead of full clear - Fix DataEnricherRegistry priority tie-breaking (deterministic by provider name) - Add ExceptionHandler for EnrichmentValidationException (HTTP 400) New Features: - Add configurable per-stage job timeout enforcement - Add per-provider resilience configuration (ProviderResiliencyRegistry) - Add DataQualityEngine with rule-based validation (FAIL_FAST/COLLECT_ALL) - Add data lineage tracking with InMemoryLineageTracker - Add enrichment fallback chains via @EnricherFallback annotation - Add composable data transformation pipeline (TransformationChain) - Add enrichment preview/dry-run endpoint - Add per-provider cost tracking and reporting - Add SSE streaming endpoint for batch enrichments - Wire all new features into Spring Boot auto-configuration GenAI Bridge: - Create fireflyframework-genai-data Python package - Tools: DataEnrichmentTool, DataJobTool, DataOperationsTool, DataToolKit - Pipeline steps: EnrichmentStep, QualityGateStep - Middleware: DataLineageMiddleware - Agent template: create_data_analyst_agent() Documentation: - Update README, architecture, configuration, resiliency, enricher, and API docs - Create new guides: data-quality, data-lineage, data-transformation, genai-bridge Tests: 383 passing (376 existing + 7 new integration tests)
2 parents 9cf1914 + f5ec1e9 commit cdc5c92

88 files changed

Lines changed: 9360 additions & 91 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/README.md‎

Lines changed: 73 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,26 @@ For fetching and integrating data from external third-party providers (credit bu
2828

2929
**Learn More:** [Data Enrichers Documentation →](data-enrichers/README.md)
3030

31+
### 3. **Data Quality** - Validation & Quality Gates
32+
Rule-based data validation with configurable strategies (fail-fast or collect-all).
33+
34+
**Learn More:** [Data Quality Framework →](common/data-quality.md)
35+
36+
### 4. **Data Lineage** - Provenance Tracking
37+
Track data transformations and enrichments across your pipeline.
38+
39+
**Learn More:** [Data Lineage Tracking →](common/data-lineage.md)
40+
41+
### 5. **Data Transformation** - Post-Processing Pipelines
42+
Composable transformation chains for field mapping and computed fields.
43+
44+
**Learn More:** [Data Transformation →](common/data-transformation.md)
45+
46+
### 6. **GenAI Bridge** - Native AI Integration
47+
Python bridge package for `fireflyframework-genai` with tools, pipeline steps, and agent templates.
48+
49+
**Learn More:** [GenAI Integration →](common/genai-bridge.md)
50+
3151
---
3252

3353
## 🚀 Quick Start
@@ -70,6 +90,13 @@ Shared concepts, architecture, and utilities:
7090
- **[API Reference](common/api-reference.md)** - Complete API documentation
7191
- **[Examples](common/examples.md)** - Real-world usage patterns
7292

93+
### [New Features](common/)
94+
Advanced capabilities added in this release:
95+
- **[Data Quality Framework](common/data-quality.md)** - Rule-based validation and quality gates
96+
- **[Data Lineage Tracking](common/data-lineage.md)** - Provenance and audit trail
97+
- **[Data Transformation](common/data-transformation.md)** - Post-enrichment transformation pipelines
98+
- **[GenAI Bridge](common/genai-bridge.md)** - Native integration with fireflyframework-genai
99+
73100
---
74101

75102
## 🎯 Common Tasks
@@ -144,6 +171,24 @@ Add the following to your `pom.xml`:
144171
- ✅ **CQRS Integration** - Command/Query separation
145172
- ✅ **SAGA Support** - Distributed transaction patterns
146173

174+
### Data Quality & Lineage
175+
- ✅ **Quality Gates** - Rule-based validation with fail-fast and collect-all strategies
176+
- ✅ **Data Lineage** - Track provenance across enrichments and transformations
177+
- ✅ **Transformation Pipelines** - Composable field mapping and computed fields
178+
179+
### Enrichment Enhancements
180+
- ✅ **Fallback Chains** - Automatic provider failover with `@EnricherFallback`
181+
- ✅ **Per-Provider Resilience** - Independent circuit breaker, retry, rate limiter per provider
182+
- ✅ **Cost Tracking** - Per-provider call counting and cost reports
183+
- ✅ **Preview/Dry-Run** - Preview enrichment routing without execution
184+
- ✅ **SSE Streaming** - Real-time batch enrichment results via Server-Sent Events
185+
- ✅ **Job Timeouts** - Configurable per-stage timeout enforcement
186+
187+
### GenAI Integration
188+
- ✅ **Native Bridge** - Python package for `fireflyframework-genai` integration
189+
- ✅ **AI Agent Tools** - Data enrichment and job management as agent tools
190+
- ✅ **Pipeline Steps** - Enrichment and quality gate steps for GenAI pipelines
191+
147192
### Developer Experience
148193
- ✅ **Abstract Base Classes** - Minimal boilerplate code
149194
- ✅ **Type-Safe APIs** - Compile-time safety
@@ -157,33 +202,33 @@ Add the following to your `pom.xml`:
157202
The library follows **Hexagonal Architecture** (Ports and Adapters):
158203

159204
```
160-
┌────────────────────────────────────────────────────────┐
161-
│ Your Application │
162-
│ │
163-
│ ┌──────────────┐ ┌──────────────┐ │
164-
│ │ Data Jobs │ │ Enrichers │ │
165-
│ │ │ │ │ │
166-
│ │ - Async │ │ - Credit │ │
167-
│ │ - Sync │ │ - Company │ │
168-
│ └──────────────┘ └──────────────┘ │
169-
│ ↓ ↓ │
170-
│ ┌──────────────────────────────────────────────┐ │
171-
│ │ fireflyframework-starter-data (Core) │ │
172-
│ │ │ │
173-
│ │ - Abstract base classes │ │
174-
│ │ - Observability (automatic) │ │
175-
│ │ - Resiliency (automatic) │ │
176-
│ │ - Event publishing (automatic) │ │
177-
│ └──────────────────────────────────────────────┘ │
178-
│ ↓ ↓ │
179-
│ ┌──────────────┐ ┌──────────────┐ │
180-
│ │ Orchestrators│ │ Providers │ │
181-
│ │ │ │ │ │
182-
│ │ - Airflow │ │ - REST APIs │ │
183-
│ │ - AWS SF │ │ - SOAP APIs │ │
184-
│ │ - Mock │ │ - gRPC APIs │ │
185-
│ └──────────────┘ └──────────────┘ │
186-
└────────────────────────────────────────────────────────┘
205+
┌────────────────────────────────────────────────────────────────┐
206+
│ Your Application │
207+
│ │
208+
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
209+
│ │ Data Jobs │ │ Enrichers │ │ Quality & │ │
210+
│ │ │ │ │ │ Lineage │ │
211+
│ │ - Async │ │ - Credit │ │ - Rules │ │
212+
│ │ - Sync │ │ - Company │ │ - Tracking │ │
213+
│ └──────────────┘ └──────────────┘ └──────────────┘ │
214+
│ ↓ ↓ ↓ │
215+
│ ┌──────────────────────────────────────────────────────┐ │
216+
│ │ fireflyframework-starter-data (Core) │ │
217+
│ │ │ │
218+
│ │ - Abstract base classes - Fallback chains │ │
219+
│ │ - Observability (auto) - Cost tracking │ │
220+
│ │ - Resiliency (auto/prov) - Transformation │ │
221+
│ │ - Event publishing (auto) - Preview & SSE │ │
222+
│ └──────────────────────────────────────────────────────┘ │
223+
│ ↓ ↓ ↓ │
224+
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
225+
│ │ Orchestrators│ │ Providers │ │ GenAI │ │
226+
│ │ │ │ │ │ Bridge │ │
227+
│ │ - Airflow │ │ - REST APIs │ │ - Tools │ │
228+
│ │ - AWS SF │ │ - SOAP APIs │ │ - Steps │ │
229+
│ │ - Mock │ │ - gRPC APIs │ │ - Agents │ │
230+
│ └──────────────┘ └──────────────┘ └──────────────┘ │
231+
└────────────────────────────────────────────────────────────────┘
187232
```
188233

189234
**Learn More:** [Architecture Overview](common/architecture.md)
@@ -217,5 +262,5 @@ For questions, issues, or contributions:
217262

218263
## 📝 License
219264

220-
Copyright © 2024 Firefly. All rights reserved.
265+
Copyright © 2024-2026 Firefly Software Solutions Inc. All rights reserved.
221266

‎docs/common/api-reference.md‎

Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1661,6 +1661,170 @@ if (tooLarge) {
16611661

16621662
---
16631663

1664+
## Enrichment API Extensions
1665+
1666+
### Preview Endpoint
1667+
1668+
```java
1669+
POST /api/v1/enrichment/smart/preview
1670+
```
1671+
1672+
Preview which enricher would handle a request without executing it.
1673+
1674+
**Request Body:** Same as `POST /api/v1/enrichment/smart`
1675+
1676+
**Response:**
1677+
```json
1678+
{
1679+
"providerName": "Equifax",
1680+
"enrichmentType": "credit-report",
1681+
"cached": false,
1682+
"priority": 100,
1683+
"supportedStrategies": ["ENHANCE", "MERGE", "REPLACE", "RAW"],
1684+
"estimatedCost": 0.50,
1685+
"costCurrency": "USD"
1686+
}
1687+
```
1688+
1689+
### SSE Streaming Endpoint
1690+
1691+
```java
1692+
POST /api/v1/enrichment/smart/stream
1693+
Content-Type: application/json
1694+
Accept: text/event-stream
1695+
```
1696+
1697+
Stream batch enrichment results via Server-Sent Events.
1698+
1699+
**Request Body:** Array of enrichment requests
1700+
```json
1701+
[
1702+
{"type": "credit-report", "params": {"companyId": "123"}},
1703+
{"type": "company-profile", "params": {"companyId": "123"}}
1704+
]
1705+
```
1706+
1707+
**Response:** SSE stream
1708+
```
1709+
id: 0
1710+
event: enrichment-result
1711+
data: {"success": true, "enrichedData": {...}}
1712+
1713+
id: 1
1714+
event: enrichment-result
1715+
data: {"success": true, "enrichedData": {...}}
1716+
1717+
event: complete
1718+
```
1719+
1720+
### Cost Tracking Endpoint
1721+
1722+
```java
1723+
GET /api/v1/enrichment/costs
1724+
GET /api/v1/enrichment/costs?tenantId={tenantId}
1725+
```
1726+
1727+
Returns per-provider enrichment cost report.
1728+
1729+
**Response:**
1730+
```json
1731+
{
1732+
"totalCalls": 1500,
1733+
"totalCost": 750.00,
1734+
"currency": "USD",
1735+
"providers": {
1736+
"equifax": {"calls": 1000, "costPerCall": 0.50, "totalCost": 500.00},
1737+
"moodys": {"calls": 500, "costPerCall": 0.50, "totalCost": 250.00}
1738+
}
1739+
}
1740+
```
1741+
1742+
### Error Handling
1743+
1744+
The `DataExceptionHandler` (`@RestControllerAdvice`) provides structured error responses:
1745+
1746+
**EnrichmentValidationException → HTTP 400:**
1747+
```json
1748+
{
1749+
"status": 400,
1750+
"error": "Validation Failed",
1751+
"message": "Enrichment request validation failed",
1752+
"errors": ["Required parameter 'companyId' is missing"],
1753+
"timestamp": "2026-02-22T20:00:00Z"
1754+
}
1755+
```
1756+
1757+
---
1758+
1759+
## Data Quality API
1760+
1761+
### DataQualityEngine
1762+
1763+
```java
1764+
// Evaluate with default strategy (COLLECT_ALL)
1765+
Mono<QualityReport> report = qualityEngine.evaluate(data);
1766+
1767+
// Evaluate with specific strategy
1768+
Mono<QualityReport> report = qualityEngine.evaluate(data, QualityStrategy.FAIL_FAST);
1769+
```
1770+
1771+
### QualityReport
1772+
1773+
```java
1774+
QualityReport report = ...;
1775+
report.isPassed(); // true if no CRITICAL failures
1776+
report.getTotalRules(); // number of rules evaluated
1777+
report.getFailedRules(); // number of failures
1778+
report.getFailures(); // list of failed QualityResults
1779+
report.getBySeverity(CRITICAL); // filter by severity
1780+
```
1781+
1782+
---
1783+
1784+
## Data Lineage API
1785+
1786+
### LineageTracker
1787+
1788+
```java
1789+
// Record lineage
1790+
lineageTracker.record(LineageRecord.builder()
1791+
.recordId(UUID.randomUUID().toString())
1792+
.entityId("customer-123")
1793+
.sourceSystem("equifax")
1794+
.operation("ENRICHMENT")
1795+
.operatorId("EquifaxCreditEnricher")
1796+
.timestamp(Instant.now())
1797+
.build()
1798+
).subscribe();
1799+
1800+
// Query lineage
1801+
Flux<LineageRecord> history = lineageTracker.getLineage("customer-123");
1802+
Flux<LineageRecord> byOp = lineageTracker.getLineageByOperator("EquifaxCreditEnricher");
1803+
```
1804+
1805+
---
1806+
1807+
## Data Transformation API
1808+
1809+
### TransformationChain
1810+
1811+
```java
1812+
// Build a transformation chain
1813+
TransformationChain<Map<String, Object>, Map<String, Object>> chain =
1814+
TransformationChain.of(
1815+
new FieldMappingTransformer(Map.of("first_name", "firstName", "last_name", "lastName"))
1816+
).then(
1817+
new ComputedFieldTransformer(Map.of(
1818+
"fullName", data -> data.get("firstName") + " " + data.get("lastName")
1819+
))
1820+
);
1821+
1822+
// Execute
1823+
Mono<Map<String, Object>> result = chain.execute(inputData);
1824+
```
1825+
1826+
---
1827+
16641828
## See Also
16651829

16661830
- [Getting Started](getting-started.md) - Setup guide

0 commit comments

Comments
 (0)