Skip to content

Commit 99fe013

Browse files
FrancescoL96FrancescoL96
committed
docs: added readme
1 parent 0f4785b commit 99fe013

1 file changed

Lines changed: 107 additions & 84 deletions

File tree

‎README.md‎

Lines changed: 107 additions & 84 deletions
Original file line numberDiff line numberDiff line change
@@ -1,41 +1,38 @@
1-
# ingester
2-
3-
4-
5-
1+
# Resources Ingester
62

73
```txt
84
Kubernetes → ingester → PostgreSQL → presenter → Clients
95
```
106

11-
The ingester focuses on **collection, normalization, and storage** of objects at scale.
7+
The ingester focuses on **collection, normalization, and storage** of Kubernetes resources at scale.
128

139
## What Does It Do?
1410

15-
The service continuously watches Kubernetes events across configured namespaces and transforms them into structured records stored in PostgreSQL.
11+
The service continuously watches Kubernetes resources across configured namespaces and transforms them into structured records stored in PostgreSQL.
1612

1713
Core responsibilities:
1814

19-
- Watch cluster events using the in-cluster Kubernetes client
20-
- Resolve related objects (for example composition IDs)
21-
- Enrich events with cluster metadata
15+
- Watch any Kubernetes resource kind using the in-cluster client
16+
- Dynamically discover and watch CRDs belonging to managed API groups
17+
- Resolve composition relationships for Krateo resources
18+
- Enrich records with cluster metadata for multi-tenancy
2219
- Batch inserts for high performance
2320
- Push records into PostgreSQL
24-
- Enable downstream streaming via LISTEN/NOTIFY
2521
- Provide health probes for Kubernetes
2622
- Handle graceful shutdown
2723

28-
2924
## Architecture Overview
3025

3126
```
3227
Kubernetes API
3328
↓
34-
Event Router (watch + resync)
29+
Static Routers (fixed resource kinds)
30+
+
31+
Dynamic Routers (CRD-driven, managed groups)
3532
↓
36-
Ingester (enrichment)
33+
Worker Pool (shared queue)
3734
↓
38-
Worker Queue
35+
Ingester (enrichment + routing)
3936
↓
4037
Batch Writer
4138
↓
@@ -44,77 +41,46 @@ PostgreSQL
4441

4542
### Key Components
4643

47-
**Event Router**
48-
49-
- Watches namespaces
50-
- Periodically resyncs
51-
- Applies throttling to prevent event storms
52-
53-
**Ingester**
54-
55-
- Extracts useful metadata
56-
- Resolves object relationships
57-
- Generates globally unique identifiers
58-
- Serializes events as JSON
59-
60-
**Queue + Workers**
61-
62-
- Buffers ingestion jobs
63-
- Enables concurrent processing
44+
**Static Routers**
6445

65-
**Batch Writer**
46+
- Defined at compile time via `router/assets/static`
47+
- Watch fixed resource kinds (Pods, Deployments, Namespaces, CRDs, etc.)
48+
- Start immediately on service boot
6649

67-
- Groups inserts for better database performance
50+
**Dynamic Routers (Manager)**
6851

69-
**Cache Cleaner**
52+
- Watch `CustomResourceDefinition` events
53+
- Dynamically start or stop routers when CRDs in managed API groups are created, updated, or deleted
54+
- Managed groups are defined in `manager/assets/managed_groups`
7055

71-
- Periodically clears internal resolver caches.
56+
## Stored Resource Model
7257

73-
74-
## Features
75-
76-
- Kubernetes-native (uses `InClusterConfig`)
77-
- High-throughput batching
78-
- Structured logging
79-
- Automatic cluster name detection
80-
- Event enrichment pipeline
81-
- Backpressure-safe queue
82-
- Health probes
83-
- Graceful shutdown
84-
- Cloud-ready design
85-
86-
## Stored Event Model
87-
88-
Each Kubernetes event is transformed into a structured record containing:
58+
Each Kubernetes resource is transformed into a structured record containing:
8959

9060
- Cluster name
9161
- Namespace
92-
- Resource kind and name
93-
- Event type and reason
94-
- Human-readable message
62+
- API group, version, kind, and plural resource name
63+
- Resource name
9564
- Composition ID (when available)
96-
- Creation timestamp
97-
- Resource version
98-
- Raw JSON payload
99-
- Globally unique ID (`cluster:eventUID`)
100-
101-
Events are **minified** before storage to reduce payload size.
65+
- Creation, update, and deletion timestamps
66+
- Raw JSON payload (full object)
67+
- UID and globally unique ID (`cluster:uid`)
10268

69+
Resources are stored with their full unstructured representation.
10370

10471
## Requirements
10572

10673
- Kubernetes cluster
10774
- PostgreSQL
10875
- Network connectivity between the service and the database
109-
- RBAC permissions to watch events
110-
76+
- RBAC permissions to watch the configured resource kinds
11177

11278
## Configuration
11379

11480
The application is configured via environment variables.
11581

11682
| Variable | Description | Default |
117-
|------------|----------------|------------|
83+
|---|---|---|
11884
| `PORT` | Health probe server port | `8080` (implementation-dependent) |
11985
| `DB_USER` | Database username | — |
12086
| `DB_PASS` | Database password | — |
@@ -127,43 +93,98 @@ The application is configured via environment variables.
12793

12894
> The service builds the PostgreSQL connection string from these values.
12995
96+
## Configuring Watched Resources
13097

131-
## How It Works
98+
The ingester has two separate mechanisms for deciding which resources to watch.
13299

133-
### 1. Startup Sequence
100+
### Static Routers
134101

135-
- Waits for PostgreSQL to become available
136-
- Detects the cluster name
137-
- Starts the health probe server
138-
- Initializes the Kubernetes client
139-
- Launches workers and batch processor
140-
- Begins watching events
102+
Static routers are defined in `router/assets/static`. Each line declares a resource kind to watch from the moment the service starts, regardless of CRDs.
141103

104+
**Format:** `group/version/resource/Kind`
142105

143-
### Event Flow
106+
For cluster-scoped resources and core API group resources, leave the group segment empty.
144107

145-
1. Kubernetes emits an event.
146-
2. The router forwards it to the ingester.
147-
3. The ingester:
108+
```txt
109+
apiextensions.k8s.io/v1/customresourcedefinitions/CustomResourceDefinition
110+
apps/v1/deployments/Deployment
111+
/v1/pods/Pod
112+
/v1/namespaces/Namespace
113+
```
114+
115+
Rules:
116+
117+
- One entry per line, blank lines are ignored
118+
- Exactly four `/`-separated segments are required; lines with a different count are skipped
119+
- The group segment may be empty (core API group resources such as Pods and Namespaces)
120+
- CRDs **must** be listed here for the dynamic manager to function, since it subscribes to CRD events
121+
122+
### Managed API Groups (Dynamic CRD Watching)
123+
124+
Dynamic routers are driven by CRD discovery. When a `CustomResourceDefinition` is created, updated, or deleted, the manager checks whether its API group appears in `manager/assets/managed_groups`. If it does, a router is started (or stopped) automatically.
125+
126+
**Format:** one API group per line.
127+
128+
```txt
129+
composition.krateo.io
130+
widgets.templates.krateo.io
131+
core.krateo.io
132+
```
148133

149-
- Resolves related objects
150-
- Extracts metadata
151-
- Attaches composition ID
152-
- Builds a structured record
134+
Rules:
153135

154-
4. The record becomes a queued job.
155-
5. The batch worker writes it to PostgreSQL.
136+
- One group per line, blank lines are ignored
137+
- Groups are matched **exactly** against the CRD's `spec.group`
138+
- Adding a group here causes the service to begin watching all CRDs installed under that group
139+
- Removing a group and redeploying stops watching those resources
140+
- The CRD kind itself must be listed in the static routers file above for this mechanism to receive events
141+
142+
## How It Works
143+
144+
### 1. Startup Sequence
145+
146+
1. Waits for PostgreSQL to become available
147+
2. Detects the cluster name
148+
3. Starts the health probe server
149+
4. Initializes the Kubernetes dynamic client
150+
5. Launches the batch writer and worker pool
151+
6. Starts static routers for fixed resource kinds
152+
7. Starts the dynamic manager to respond to CRD changes
153+
8. Begins processing events
154+
155+
### Event Flow
156+
157+
1. Kubernetes emits a resource change (add, update, or delete).
158+
2. The router enqueues a `QueueItem` containing the object key and GVR.
159+
3. The worker pool dequeues the item and calls `reconcile`:
160+
- For CRDs (`apiextensions.k8s.io`): routes to the Manager via event bus to start or stop a dynamic router.
161+
- For all other resources: looks up the current object from the informer store and routes to Storage.
162+
4. The ingester:
163+
- Resolves the composition ID from labels
164+
- Extracts metadata (group, version, kind, plural, cluster name)
165+
- Sets `deleted_at` for delete events
166+
- Builds a structured record
167+
5. The record becomes a queued batch job.
168+
6. The batch writer flushes records to PostgreSQL.
156169

157170
## Health Probes
158171

159172
The service exposes probe endpoints suitable for Kubernetes deployments.
160173

161174
| Endpoint | Purpose |
162-
|------------|------------|
175+
|---|---|
163176
| `/livez` | Process is alive |
164177
| `/readyz` | Database reachable and service ready |
165178

179+
## Metrics
180+
181+
Prometheus metrics are exposed at `:9090/metrics`.
166182

183+
| Metric | Description |
184+
|---|---|
185+
| Worker processed count | Total successfully processed queue items |
186+
| Worker error count | Total failed queue items |
187+
| Worker processing duration | Histogram of per-item processing time |
167188

168189
## Deployment Notes
169190

@@ -173,6 +194,8 @@ Best practices:
173194

174195
- Store DB credentials in Secrets
175196
- Use resource limits
176-
- Monitor queue depth
197+
- Monitor queue depth via the pipeline status log line (emitted every 30 seconds)
198+
- Monitor active informer count and goroutine count in logs
177199
- Enable PostgreSQL connection pooling
178200
- Co-locate with the database when possible to reduce latency
201+
- Grant RBAC `watch`, `list`, and `get` permissions for all resource kinds listed in `router/assets/static` and all CRDs in managed groups

0 commit comments

Comments
 (0)