Skip to content

TESTING.md

Latest commit

 

History

History
360 lines (265 loc) · 17.1 KB

TESTING.md

File metadata and controls

360 lines (265 loc) · 17.1 KB
Raw

MapReduce Platform: Testing Specifications & Documentation

This document provides comprehensive documentation on the automated testing suite for the MapReduce platform. It outlines the architecture, tools used, module-specific test coverage, and execution instructions.

1. Objectives & Scope

The test suite ensures the reliability, scalability, and resilience of the distributed platform as outlined in Section 8 (Testing Specifications) of the system design. It is designed to run completely isolated from external infrastructure (like physical Kubernetes clusters or external cloud providers) while maintaining full end-to-end realism using containerization.

2. Technology Stack

The testing layer leverages the following tools:

  • JUnit 5 (Jupiter): The core testing framework for all modules.
  • Mockito & AssertJ: For robust object mocking and fluent assertions.
  • Spring Boot Test (@SpringBootTest, MockMvc): For testing REST controllers and full Spring application contexts.
  • Spring Security Test: For mocking JWT tokens and bypassing real Keycloak instances in test contexts.
  • Testcontainers: To programmatically spin up real PostgreSQL and MinIO Docker containers.
  • Maven Surefire Plugin: Orchestrates the execution of tests during the build lifecycle.

3. Test Suite Breakdown

3.1. Unit & Controller Tests

Unit tests are designed to be extremely fast and validate core business logic without loading the full Spring Application Context. Controller tests use MockMvc to validate HTTP routing, security boundaries, and payload validation.

Manager Service (manager-service)

  • FileServiceTest: Validates metadata generation and database persistence without interacting with real MinIO.
  • HeartbeatWatchdogTest: Simulates time progression to ensure dead workers are detected and their tasks are automatically reassigned.
  • InternalTaskControllerTest / InternalJobControllerTest: Validates internal routing used by Kubernetes workers.
  • JobOrchestrationCancelTest: Ensures that in-flight jobs can be aborted and their running worker pods are terminated via the Kubernetes API.

UI Service (ui-service)

  • AuthControllerTest: Verifies Keycloak redirect logic and session management.
  • JobControllerTest, DataControllerTest, CodeControllerTest: Verifies that UI interactions correctly format and relay REST requests to the downstream manager-service.

3.2. Integration Tests

Integration tests run with a real Spring Context, a real database, and real object storage.

  • MinioStorageServiceIT: Validates actual file uploads, directory listing algorithms, bucket creation, and presigned URL generation against a real MinIO container.
  • JobLifecycleIT: Orchestrates a full end-to-end job submission. It ensures the system correctly transitions from MAP_PHASE -> REDUCE_PHASE -> COMPLETED.
  • WorkerFailureRecoveryIT: The most critical integration test. It simulates worker failures during a job execution and ensures the Orchestration Service correctly retries the task 3 times. On the 4th failure, it guarantees the entire job is marked as FAILED.
  • TaskRepositoryIT: Validates custom JPA queries and database constraints.

3.3. Chaos Engineering Tests (ChaosEngineeringIT)

These tests simulate catastrophic failures in the distributed system to ensure the orchestrator handles them gracefully.

  • Random Pod Killing: Simulates 30% of worker pods randomly crashing during the map phase. Validates that the system eventually recovers via the retry mechanism.
  • Database Connection Pool Exhaustion: Submits numerous jobs simultaneously on separate threads to ensure HikariCP connections, JPA locks, and transactions do not result in deadlocks.
  • Concurrent Task Updates: Validates that concurrent HTTP requests from different workers finishing at the same exact millisecond do not result in lost updates or race conditions when transitioning the job phase.
  • (Disabled) MinIO Service Disruption: Simulating network partitions is best handled via Toxiproxy. Directly stopping the singleton MinIO container is disabled as it corrupts the port mappings for subsequent tests in the JVM lifecycle.

3.4 Representative Code Examples

Below are concise, copyable excerpts from the actual tests under manager-service/src/test/ that illustrate the patterns used throughout the suite. These examples are intentionally short — see the referenced test files for full context and additional assertions.

  • Singleton Testcontainers setup — shows how PostgreSQL and MinIO are started once per JVM and wired into Spring properties (TestContainersBase.java):
public static final PostgreSQLContainer<?> POSTGRES =
	new PostgreSQLContainer<>(DockerImageName.parse("postgres:16-alpine"))
		.withDatabaseName("mapreduce_test")
		.withUsername("test")
		.withPassword("test");

public static final GenericContainer<?> MINIO =
	new GenericContainer<>(DockerImageName.parse("quay.io/minio/minio:latest"))
		.withCommand("server /data")
		.withEnv("MINIO_ROOT_USER", "minioadmin")
		.withEnv("MINIO_ROOT_PASSWORD", "minioadmin")
		.withExposedPorts(9000);

static {
	POSTGRES.start();
	MINIO.start();
}

@DynamicPropertySource
static void overrideProperties(DynamicPropertyRegistry registry) {
	registry.add("spring.datasource.url", POSTGRES::getJdbcUrl);
	registry.add("minio.endpoint", () -> "http://localhost:" + MINIO.getMappedPort(9000));
	registry.add("kubernetes.disable.autoConfig", () -> "true");
}
  • MinIO integration test — upload + download validation (MinioStorageServiceIT.java):
String key = "test/integration/" + UUID.randomUUID() + "/data.txt";
String content = "Hello, MinIO integration test!";

minioService.upload(key,
		new ByteArrayInputStream(content.getBytes(StandardCharsets.UTF_8)),
		content.length(), "text/plain");

try (InputStream is = minioService.download(key)) {
	String downloaded = new String(is.readAllBytes(), StandardCharsets.UTF_8);
	assertThat(downloaded).isEqualTo(content);
}
  • Full job lifecycle (submit → map → reduce → complete) — shows job submission and simulating task completions (JobLifecycleIT.java / CompleteJobLifecycleIT.java):
var request = new gr.tuc.distributed.common.dto.JobSubmitRequest();
request.setDataId(dataFile.getFileId().toString());
request.setCodeId(codeFile.getFileId().toString());
request.setNumReducers(2);

UUID jobId = orchestrationService.submitJob(request, userId);

// simulate completing all map tasks
taskRepository.findByJobJobIdAndTaskType(jobId, TaskType.MAP)
	.forEach(t -> {
		TaskStatusUpdate u = new TaskStatusUpdate();
		u.setStatus(TaskStatus.COMPLETED);
		u.setOutputLocation("users/temp/" + jobId + "/map-" + t.getTaskId() + "/part-0.txt");
		orchestrationService.handleTaskUpdate(t.getTaskId(), u);
	});

// then complete reduce tasks similarly and assert final state
  • Worker failure & retry behavior — failing a task multiple times and asserting retries (WorkerFailureRecoveryIT.java):
TaskStatusUpdate fail = new TaskStatusUpdate();
fail.setStatus(TaskStatus.FAILED);
fail.setErrorMessage("simulated crash #" + i);
orchestrationService.handleTaskUpdate(firstTask.getTaskId(), fail);

Task updated = taskRepository.findById(firstTask.getTaskId()).orElseThrow();
assertThat(updated.getStatus()).isEqualTo(TaskStatus.IN_PROGRESS);
assertThat(updated.getRetryCount()).isEqualTo(i + 1);
  • Unit test: file upload service — mocking MinioStorageService and asserting path construction (FileServiceTest.java):
when(minioService.upload(anyString(), any(), anyLong(), anyString()))
		.thenAnswer(inv -> inv.getArgument(0));

fileService.uploadData(sampleFile, userId);

ArgumentCaptor<String> keyCaptor = ArgumentCaptor.forClass(String.class);
verify(minioService).upload(keyCaptor.capture(), any(), eq(11L), eq("text/plain"));
assertThat(keyCaptor.getValue()).startsWith("users/" + userId + "/raw/");
  • Controller test: MockMvc worker callback — how internal endpoints are exercised without JWT complexity (InternalTaskControllerTest.java):
TaskStatusUpdate update = new TaskStatusUpdate();
update.setStatus(TaskStatus.COMPLETED);
update.setOutputLocation("some/output/path");

mockMvc.perform(post("/internal/tasks/{taskId}/status", taskId)
				.contentType(MediaType.APPLICATION_JSON)
				.content(objectMapper.writeValueAsString(update)))
		.andExpect(status().isNoContent());

verify(orchestrationService).handleTaskUpdate(eq(taskId), any(TaskStatusUpdate.class));
  • Watchdog test: detect stale heartbeat and fail task (HeartbeatWatchdogTest.java):
Task dead = buildInProgressTask(Instant.now().minus(60, ChronoUnit.SECONDS));
when(taskRepository.findByStatusAndLastHeartbeatBefore(eq(TaskStatus.IN_PROGRESS), any()))
		.thenReturn(List.of(dead));

watchdog.checkDeadWorkers();

ArgumentCaptor<TaskStatusUpdate> captor = ArgumentCaptor.forClass(TaskStatusUpdate.class);
verify(orchestrationService).handleTaskUpdate(eq(dead.getTaskId()), captor.capture());
assertThat(captor.getValue().getStatus()).isEqualTo(TaskStatus.FAILED);

For full tests and more patterns (concurrent submissions, chaos scenarios, repository-level checks), inspect the files under manager-service/src/test/ mentioned earlier.

3.5 Additional Examples

Below are more focused snippets showing common test patterns found across the suite.

  • Mocking the Kubernetes launcher (used in many integration tests to avoid creating real pods):
when(k8sLauncher.launchWorker(any(), any(), anyString(), anyString(), anyString()))
		.thenAnswer(inv -> "mock-pod-" + inv.getArgument(0));
  • Concurrent job submissions (submit multiple jobs in parallel to validate DB concurrency handling):
int concurrency = 5;
ExecutorService executor = Executors.newFixedThreadPool(concurrency);
List<Future<UUID>> futures = new ArrayList<>();

for (int i = 0; i < concurrency; i++) {
	futures.add(executor.submit(() -> {
		JobSubmitRequest req = new JobSubmitRequest();
		req.setDataId(df.getFileId().toString());
		req.setCodeId(cf.getFileId().toString());
		req.setNumReducers(1);
		return orchestrationService.submitJob(req, uid);
	}));
}

// collect results and assert no submission errors
  • Concurrent task updates (synchronize threads with a CountDownLatch to create simultaneous updates):
ExecutorService executor = Executors.newFixedThreadPool(mapTasks.size());
CountDownLatch latch = new CountDownLatch(1);
for (Task t : mapTasks) {
	executor.submit(() -> {
		latch.await();
		TaskStatusUpdate update = new TaskStatusUpdate();
		update.setStatus(TaskStatus.COMPLETED);
		orchestrationService.handleTaskUpdate(t.getTaskId(), update);
	});
}
latch.countDown(); // fire all threads simultaneously
  • Repository queries — examples of JPA-backed test assertions (TaskRepositoryIT.java):
Instant threshold = Instant.now().minus(30, ChronoUnit.SECONDS);
List<Task> deadTasks = taskRepository.findByStatusAndLastHeartbeatBefore(
		TaskStatus.IN_PROGRESS, threshold);

long completed = taskRepository.countByJobJobIdAndStatus(job.getJobId(), TaskStatus.COMPLETED);
assertThat(completed).isEqualTo(2);
  • Cancel job behaviour — verify Kubernetes job deletion is invoked and errors are handled gracefully (JobOrchestrationCancelTest.java):
service.cancelJob(job.getJobId(), userId);
verify(k8sLauncher).deleteJob("worker-reduce-1");

// Simulate K8s delete failure — should not throw
doThrow(new RuntimeException("K8s unreachable")).when(k8sLauncher).deleteJob("worker-map-broken");
service.cancelJob(job.getJobId(), userId); // does not propagate
  • Controller: submit job with test JWT (InternalJobControllerTest.java):
mockMvc.perform(post("/internal/jobs")
		.with(SecurityMockMvcRequestPostProcessors.jwt()
				.jwt(jwt -> jwt.subject("user-1")))
		.contentType(MediaType.APPLICATION_JSON)
		.content(objectMapper.writeValueAsString(request)))
	.andExpect(status().isAccepted())
	.andExpect(jsonPath("$.jobId").value(jobId.toString()));

These snippets represent recurring patterns: Testcontainers orchestration, mocking external subsystems (Kubernetes, MinIO), using MockMvc for controller tests, and stress-testing concurrency with Java's ExecutorService and synchronization primitives.

4. Testcontainers Architecture (The Singleton Pattern)

To ensure the tests are realistic but run quickly, we use Testcontainers to spin up actual Docker containers during the test run.

Instead of starting and stopping a MinIO and PostgreSQL container for every test class (which causes massive delays, JVM overhead, and port exhaustion), we implemented the Singleton Container Pattern.

In TestContainersBase.java, the containers are started in a static block (static { ... }). They are started exactly once per test session. All integration tests extend this base class and share the running containers, making the entire suite extremely fast while retaining data isolation (tests use unique UUIDs for all database entities and MinIO bucket prefixes).

5. Kubernetes Decoupling Strategy

One of the most powerful features of this test suite is that Minikube (or any real Kubernetes cluster) does NOT need to be running.

We bypass Kubernetes entirely so that tests can run anywhere (like GitHub Actions) without needing a full cluster. Here is how we achieved that:

  1. Disabling K8s Auto-Config: In TestContainersBase.java, we inject kubernetes.disable.autoConfig=true. This tells the Kubernetes Fabric8 Client to skip searching for a .kube/config file when Spring Boot boots up.
  2. Mocking the Launcher: In integration tests (like JobLifecycleIT), we use Spring's @MockBean on the KubernetesJobLauncher. Instead of actually talking to the K8s API to spawn a pod, the mock intercepts the request and instantly returns a mock pod name.
  3. Simulating Worker Callbacks: Since there are no real pods running, the test itself acts as the "worker". It manually pushes TaskStatusUpdate events (like IN_PROGRESS, FAILED, or COMPLETED) directly to the JobOrchestrationService, simulating exactly what a real pod would do via HTTP.

6. Development & Execution Requirements

Prerequisites

  • Docker Daemon: Must be running locally (Testcontainers requires Docker API version 1.40+).
  • Java 25: The system compiles using JDK 25.

Known Caveats

  • Mockito Inline Mocking on Java 25: To support advanced mocking on JDK 25, the Maven pom.xml explicitly overrides the byte-buddy dependency to version 1.18.8. The maven-surefire-plugin is also configured to export internal JVM modules (--add-opens java.base/java.lang=ALL-UNNAMED).
  • Docker Client API Versioning: If you experience "client version too old" errors from Testcontainers, it is due to older daemon compatibility. We force the API version in manager-service/src/test/resources/docker-java.properties (api.version=1.44).

Execution Commands

You can execute the test suite using standard Maven commands.

Note

By default, standard Maven Surefire configuration only runs classes ending in *Test (Unit and Controller tests). To run Integration and Chaos tests (which end in *IT), you must explicitly use the -Dtest="*IT" filter.

Run Unit and Controller Tests:

mvn test

Run Only Integration and Chaos Tests:

mvn test -pl manager-service -Dtest="*IT"

Run All Tests (Unit, Controller, Integration, Chaos):

mvn test -Dtest="*Test,*IT" -Dsurefire.failIfNoSpecifiedTests=false

Run a Specific Test Class:

mvn test -pl manager-service -Dtest="JobLifecycleIT"

7. Performance & Load Testing (Apache Bench)

To validate the platform's stability, response times, and resilience under high concurrent loads, you can use the Apache Bench (ab) load testing suite.

Prerequisites

  • Apache Bench: Ensure ab is installed locally (via apache2-utils on Ubuntu/Debian or httpd-tools on CentOS/RHEL).

Execution Wrapper Script

The script run-ab.sh is a simple wrapper to automate running load tests against the UI/API service.

Basic Usage:

./scripts/run-ab.sh [options]

Available Options:

  • -h host Host address (default: 192.168.49.2)
  • -p port Service port (default: 30080)
  • -s scheme Protocol scheme (http or https, default: http)
  • -u path Request endpoint path (default: /api/v1/jobs)
  • -t token Authorization Bearer Token (optional, for authenticated routes)
  • -n requests Total number of requests to perform (default: 10000)
  • -c concurrency Number of multiple concurrent requests to make at a time (default: 500)
  • -o out Output log file (default: ab-<timestamp>.log)

Example (Anonymous load test against default host/port):

./scripts/run-ab.sh -n 5000 -c 100

Example (Authenticated load test with JWT Bearer Token):

./scripts/run-ab.sh -h localhost -p 30080 -u /api/v1/jobs -t "YOUR_JWT_ACCESS_TOKEN" -n 1000 -c 50

The script prints a summary (first 40 lines of the Apache Bench output) directly to the console upon completion and stores the full execution log in the designated output file (e.g., ab-1779879616.log).