Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ on:

jobs:
build:
name: Unit build (no server required)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
Expand All @@ -17,8 +18,12 @@ jobs:
java-version: '17'
cache: maven

- name: Build and test
run: mvn verify
# `verify` compiles, runs the unit tests, packages and runs SpotBugs.
# Integration tests (*IT) are skipped unless -Pintegration is requested,
# so this job never needs a broker.
- name: Build, unit test and static analysis
run: mvn --batch-mode verify

- name: Static analysis
run: mvn spotbugs:check -DfailOnError=true
- name: Build testcontainers module
run: mvn --batch-mode verify
working-directory: testcontainers
43 changes: 39 additions & 4 deletions .github/workflows/integration.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,18 +8,29 @@ on:
schedule:
- cron: '0 4 * * 1' # Weekly Monday 4am UTC
workflow_dispatch:
inputs:
image:
description: Streamline image to test against
required: false
default: ghcr.io/streamlinelabs/streamline:latest

env:
STREAMLINE_IMAGE: ${{ inputs.image || 'ghcr.io/streamlinelabs/streamline:latest' }}

jobs:
integration:
name: Integration tests (live server)
runs-on: ubuntu-latest
services:
streamline:
image: ghcr.io/streamlinelabs/streamline:latest
# GitHub does not expand `env` in service definitions, so the image is
# taken straight from the workflow input (defaulted to :latest).
image: ${{ inputs.image || 'ghcr.io/streamlinelabs/streamline:latest' }}
ports:
- 9092:9092
- 9094:9094
options: >-
--health-cmd "curl -f http://localhost:9094/health || exit 1"
--health-cmd "curl -f http://localhost:9094/health/live || exit 1"
--health-interval 5s
--health-timeout 5s
--health-retries 10
Expand All @@ -30,15 +41,39 @@ jobs:
java-version: '17'
distribution: 'temurin'
cache: 'maven'

- name: Wait for Streamline
run: |
for i in $(seq 1 30); do
curl -sf http://localhost:9094/health && break
curl -sf http://localhost:9094/health/live && exit 0
echo "Waiting for Streamline... ($i/30)"
sleep 2
done
echo "Streamline did not become healthy" >&2
exit 1

# STREAMLINE_INTEGRATION=1 is the explicit opt-in: the *IT suites fail fast
# (instead of silently skipping) when the endpoints below are unreachable.
- name: Run integration tests
run: mvn --batch-mode -Pintegration verify
run: mvn --batch-mode verify -Pintegration
env:
STREAMLINE_INTEGRATION: '1'
STREAMLINE_BOOTSTRAP_SERVERS: localhost:9092
STREAMLINE_HTTP_URL: http://localhost:9094

testcontainers:
name: Testcontainers module
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-java@v4
with:
java-version: '17'
distribution: 'temurin'
cache: 'maven'

- name: Run container integration tests
run: mvn --batch-mode verify -Pintegration
working-directory: testcontainers
env:
STREAMLINE_INTEGRATION: '1'
46 changes: 46 additions & 0 deletions AUDIT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# Clean Code and SRP Audit

## Summary

- **Highest-leverage future split:** separate HTTP request mechanics from the
500-line `SchemaRegistryClient`, preserving its public synchronous API.
- Producer, consumer, and main client are stateful lifecycle actors; splitting
them without stronger concurrency coverage risks hidden ordering changes.
- `AdminClient` spans topic/group/cluster operations but shares one Kafka admin
backend and has bounded methods; another facade would add indirection.
- Spring listener discovery and listener execution share one processor because
lifecycle state connects them; a split is only useful after startup/shutdown
behavior is more deeply characterized.
- The baseline repair now compiles examples and cleanly separates 336 unit
tests from 58 opt-in integration tests.

## Findings

| ID | Location | Category | Severity | Actors in conflict | Cost | Size | Behavior risk |
|---|---|---|---|---|---|---|---|
| JAVA-SRP-1 | `schema/SchemaRegistryClient.java` | SRP, HTTP client | P2 | Schema Registry endpoint policy; HTTP transport/auth/cache | Repeated request/status/decoding mechanics obscure endpoint-specific rules. | L | Medium |
| JAVA-SRP-2 | `testcontainers/StreamlineContainer.java` | SRP, test product | P2 | container lifecycle; endpoint helpers; topic/test utility methods | Docker lifecycle and SDK-specific convenience operations change for different test actors. | L | Medium |
| JAVA-CC-1 | `StreamlineAutoConfiguration`/listener processor | Spring lifecycle warning | P2 | auto-configuration; listener discovery | BeanPostProcessor construction eagerly creates configuration/client beans, producing Spring eligibility warnings. | M | Medium |

## Ordered Refactor Sequence

1. Characterize Schema Registry request paths, bodies, auth, statuses, and
decoding with an in-process HTTP server.
2. Extract a private request executor; keep endpoint policy in public methods.
3. Add listener-processor tests for lazy client resolution, bean discovery,
start, stop, and shutdown.
4. Only then remove eager BeanPostProcessor dependencies.
5. Keep producer/consumer state intact until race/lifecycle coverage improves.

## Deferred

- Schema Registry transport extraction needs broader endpoint tests.
- Listener processor cleanup needs Spring lifecycle characterization.
- Live integration remains blocked by registry access.

## Out of Scope

- `StreamlineConfig`/`StreamlineProperties`: public configuration contracts.
- Producer and consumer: cohesive stateful actors.
- `AdminClient`: one Kafka administration backend.
- Examples: compiled documentation product, intentionally separate module.
37 changes: 37 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,43 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Changed
- **Test layout** — unit tests (`*Test`, Surefire) are now hermetic and never contact a
broker or HTTP endpoint, so `mvn verify` is self-contained and bounded. Tests that need
a live server moved to Failsafe integration tests (`*IT`, tagged `integration`):
`ProducerIT`, `ConsumerIT`, `StreamlineIT`, `ConformanceIT` (formerly `ConformanceTest`)
and `StreamlineContainerIT` (formerly `StreamlineContainerTest`).
- Integration tests are opt-in: they run only under `-Pintegration` (auto-activated by
`STREAMLINE_INTEGRATION=1`) and require `STREAMLINE_INTEGRATION=1`. When enabled, an
unreachable endpoint now fails fast instead of being silently skipped. Endpoints are
configurable with `STREAMLINE_BOOTSTRAP_SERVERS`, `STREAMLINE_HTTP_URL` and
`STREAMLINE_SCHEMA_REGISTRY_URL`.
- `docker-compose.test.yml` no longer pins an unpublished image tag; it uses
`STREAMLINE_IMAGE` (default `ghcr.io/streamlinelabs/streamline:latest`) and the
`/health/live` probe.
- Examples moved from the non-existent `com.streamline.*` API to the real
`dev.streamline.*` API and are now compiled by every build as the `examples` module
(never installed or deployed).
- Compiler now uses `release` instead of `source`/`target`, so building on a newer JDK
cannot link against post-Java-17 APIs.

### Fixed
- `streamline-client` and `streamline-spring-boot-starter` declared parent version
`0.2.0` while the parent POM was `0.3.0`, so the build silently resolved a stale
installed parent (and failed outright on a clean machine).
- Spring Boot starter no longer fails to start in non-web applications: the
`StreamlineTemplate` bean falls back to its own `ObjectMapper` when the application
does not define one, and `StreamlineMetrics` is only created when a `MeterRegistry`
bean exists.
- `AdminClient` branch operations and `QueryClient.explain` now carry per-request
timeouts; `AdminClient` honours the configured connect/request timeouts.
- `StreamlineVerifier` checks for missing attestation fields instead of catching
`NullPointerException`, and `CircuitBreaker` switches have explicit default branches.
- Static analysis runs again on modern JDKs (SpotBugs 4.9.x); documented exclusions live
in `spotbugs-exclude.xml`.
- Mockito and Byte Buddy are managed explicitly so mocking works on current JDKs; the
Spring Boot BOM previously pinned an unusable Byte Buddy version.


## [0.3.0] - 2026-04-20

Expand Down
23 changes: 18 additions & 5 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,13 @@ Java 17 SDK for [Streamline](https://github.com/streamlinelabs/streamline) with
## Build & Test
```bash
mvn compile # Build
mvn verify # Build + test + SpotBugs
mvn test # Run tests only
mvn verify # Build + unit tests + package + SpotBugs (no server needed)
mvn test # Run unit tests only
mvn javadoc:javadoc # Generate Javadoc

# Integration tests (*IT) — opt-in, needs a live server
docker compose -f docker-compose.test.yml up -d
STREAMLINE_INTEGRATION=1 mvn verify -Pintegration
```

## Architecture
Expand All @@ -32,6 +36,8 @@ streamline-java-sdk/
│ ├── StreamlineProperties.java
│ ├── StreamlineTemplate.java
│ └── @StreamlineListener annotation
├── examples/ # Runnable examples, compiled but never published
└── testcontainers/ # Standalone Testcontainers module (own coordinates)
```

## Coding Conventions
Expand Down Expand Up @@ -64,8 +70,15 @@ public class EventConsumer {
```

## Testing
- JUnit 5.10 + Mockito 5.7 for unit tests
- Testcontainers 1.19 for integration tests
- JUnit 5.10 + Mockito for unit tests; Testcontainers 1.19 for the container module
- **Unit tests (`*Test`, Surefire) must be hermetic** — never depend on a service running
on the machine. Use `UnitTestEndpoints.BOOTSTRAP_SERVERS` (TEST-NET-1, unroutable) when
a real client object is required; in-process stubs on ephemeral loopback ports are fine.
- **Integration tests (`*IT`, tagged `integration`, Failsafe)** need a live server. They
are skipped unless `-Pintegration` is active, and require `STREAMLINE_INTEGRATION=1`;
once enabled, an unreachable endpoint fails fast instead of skipping. Endpoints come
from `dev.streamline.testsupport.IntegrationEnvironment`
(`STREAMLINE_BOOTSTRAP_SERVERS`, `STREAMLINE_HTTP_URL`, `STREAMLINE_SCHEMA_REGISTRY_URL`).
- JaCoCo for coverage (runs on `verify`)
- SpotBugs for static analysis (runs on `verify`)
- SpotBugs for static analysis (runs on `verify`); exclusions live in `spotbugs-exclude.xml`

53 changes: 44 additions & 9 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,33 +37,68 @@ This is a multi-module Maven project:

- `streamline-client/` — Core Java client library
- `streamline-spring-boot-starter/` — Spring Boot auto-configuration starter
- `examples/` — Runnable examples, compiled by every build but never published
- `testcontainers/` — Standalone Testcontainers module (own coordinates, built separately)

## Running Tests

Tests are split into two phases:

| Phase | Naming | Runner | Needs a server |
|---|---|---|---|
| Unit | `*Test` | Surefire | No |
| Integration | `*IT` (tagged `integration`) | Failsafe | Yes |

Unit tests are hermetic: they must never depend on a service running on the machine.
Where a real client object is needed, point it at
`UnitTestEndpoints.BOOTSTRAP_SERVERS` (TEST-NET-1, guaranteed unroutable) so the
result cannot change depending on whether a Streamline server happens to be running
locally. Binding an in-process stub to an ephemeral loopback port is fine — that is
still self-contained.

```bash
# Unit tests
# Unit tests only — no server required
mvn test

# Full verification (unit + integration tests)
# Compile, unit test, package and run SpotBugs — still no server required
mvn verify

# Run a specific test class
mvn test -pl streamline-client -Dtest=StreamlineProducerTest
mvn test -pl streamline-client -Dtest=ProducerTest
```

### Integration Tests

Integration tests require a running Streamline server:
Integration tests are opt-in and require a running Streamline server:

```bash
# Start the server
# Start the server (STREAMLINE_IMAGE overrides the image)
docker compose -f docker-compose.test.yml up -d

# Run integration tests
mvn verify -Pintegration
# Run everything, including *IT
STREAMLINE_INTEGRATION=1 mvn verify -Pintegration

# Stop the server
docker compose -f docker-compose.test.yml down
docker compose -f docker-compose.test.yml down -v
```

Selection rules:

- Without `-Pintegration` (and without `STREAMLINE_INTEGRATION=1`), Failsafe is
skipped entirely, so `mvn verify` is self-contained and bounded.
- With the profile but without `STREAMLINE_INTEGRATION=1`, the `*IT` suites are
reported as skipped.
- With `STREAMLINE_INTEGRATION=1`, an unreachable endpoint fails the build within
seconds — integration tests never pass silently because a server was missing.

Endpoints are configurable through `STREAMLINE_BOOTSTRAP_SERVERS`,
`STREAMLINE_HTTP_URL` and `STREAMLINE_SCHEMA_REGISTRY_URL`; see
`dev.streamline.testsupport.IntegrationEnvironment`.

The `testcontainers/` module is built separately and follows the same rules:

```bash
cd testcontainers && STREAMLINE_INTEGRATION=1 mvn verify -Pintegration
```

## Code Style
Expand All @@ -78,7 +113,7 @@ docker compose -f docker-compose.test.yml down
- Write clear commit messages
- Add tests for new functionality
- Update documentation if needed
- Ensure `mvn verify` passes before submitting
- Ensure `mvn verify` passes before submitting (it must not need a server)

## Reporting Issues

Expand Down
45 changes: 28 additions & 17 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,40 +1,51 @@
.PHONY: integration-test build test lint fmt clean help
.PHONY: help build test unit-test integration-test verify lint fmt clean package release

MVN ?= $(shell [ -x ./mvnw ] && echo ./mvnw || echo mvn)
COMPOSE ?= docker compose -f docker-compose.test.yml
# Override to test a specific build, e.g. STREAMLINE_IMAGE=ghcr.io/streamlinelabs/streamline:0.3.0
export STREAMLINE_IMAGE ?= ghcr.io/streamlinelabs/streamline:latest

help: ## Show this help
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-15s\033[0m %s\n", $$1, $$2}'
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-16s\033[0m %s\n", $$1, $$2}'

build: ## Compile the SDK
mvn compile -q
$(MVN) compile -q

test: unit-test ## Alias for unit-test

unit-test: ## Run unit tests only (no server required)
$(MVN) test

test: ## Run all tests
mvn verify -q
verify: ## Build, unit-test, package and run static analysis (no server required)
$(MVN) verify

lint: ## Run linting and style checks
mvn checkstyle:check -q 2>/dev/null || echo "checkstyle not configured, skipping"
lint: ## Run static analysis (SpotBugs)
$(MVN) compile spotbugs:check

fmt: ## Format code (check only)
@echo "Use IDE formatting or google-java-format"

clean: ## Clean build artifacts
mvn clean -q
$(MVN) clean -q
-$(COMPOSE) down -v >/dev/null 2>&1

package: ## Build JAR package
mvn package -q -DskipTests
$(MVN) package -q -DskipTests

release: ## Deploy to Maven Central
mvn deploy -P release -DskipTests
# update parent POM dependency versions
# configure Maven Central publishing plugin
$(MVN) deploy -P release -DskipTests

integration-test: ## Run integration tests (requires Docker)
docker compose -f docker-compose.test.yml up -d
@echo "Waiting for Streamline server..."
integration-test: ## Run integration tests against a live server (requires Docker)
$(COMPOSE) up -d
@echo "Waiting for Streamline ($(STREAMLINE_IMAGE))..."
@for i in $$(seq 1 30); do \
if curl -sf http://localhost:9094/health/live > /dev/null 2>&1; then \
echo "Server ready"; \
break; \
fi; \
sleep 2; \
done
MVN=./mvnw && [ -f "$$MVN" ] && $$MVN verify -Pintegration || mvn verify -Pintegration
docker compose -f docker-compose.test.yml down -v
STREAMLINE_INTEGRATION=1 $(MVN) verify -Pintegration; \
status=$$?; \
$(COMPOSE) down -v; \
exit $$status
Loading
Loading