CRDT Benchmarking + Refactor - #11
Open
RubenvanGemeren wants to merge 166 commits into
Open
Conversation
first refactor for QueryProcFun, remove fatal warnings from sbt build
* Refactor process function and snapshot method interface * Add outputFunction callback for process function
This reverts commit 5dd461d.
CRDTs are almost in sync
Window aggregates does line up perfectly yet
Attempting to fix logical issues
* Refactor process function and snapshot method interface * Add outputFunction callback for process function * Add basic snapshotting & failure recovery * Basic centralized failure recovery with partition ownership * Fixes issue with serialization/deserialization of crdts before the fix a CRDT may be deserialized to a different value. The issue was due to the protocol and system names differing before and after serialization, as internally this information is not serialized by the ReplicatedDataSerializer. * Containerize application * Refactor FailureDetector * Refactor checkpointing logic into CheckpointManager * Request ownership after recovery * Refactor checkpoint manager --------- Co-authored-by: Jonas Spenger <6832643+jspenger@users.noreply.github.com>
Based on feedback, a restructure of the entire project with the focus on functional separation of core components and specific query implementations.
- Introduced `run-experiment.sh` to automate the end-to-end experiment process on AWS. - Added supporting scripts for setup, template selection, EC2 management, and result collection. - Refactored `deploy.sh` for improved clarity and functionality, including dynamic query handling and Docker Compose integration. - Removed outdated SQL query files to streamline the project structure. - Updated README files for better guidance on remote deployment and file management.
- Added `.holon.env.example` for environment variable setup, ensuring sensitive data is kept out of version control. - Updated `.gitignore` to include new files and directories for experiment results and environment configuration. - Modified `HOLON-WIKI.md` and `README.md` to reflect changes in build scripts and provide clearer instructions for running experiments. - Introduced `SETUP.md` to outline prerequisites and one-time setup steps for running experiments on AWS.
- Updated environment variable definitions to support default values. - Changed volume paths to relative paths for better portability. - Increased heartbeat timeout for improved stability. - Switched Kafka image to the official Apache Kafka image for reliability. - Added a new producer start gate service for managing producer instances. - Adjusted taskmanager and jobmanager memory settings for better resource management.
- Introduced a new Python service (`gate.py`) to manage producer readiness via HTTP endpoints. - Added `requirements.txt` for Flask dependency management. - Refactored Nexmark producer classes to utilize the new start gate mechanism for improved synchronization. - Updated `DecentralizedCheckpointManager` to read local files with UTF-8 encoding. - Enhanced logging and waiting mechanisms across various components for better clarity and functionality.
…ionality - Updated base images in Dockerfiles from `openjdk:11-jre-slim` to `eclipse-temurin:11-jre-jammy` for better performance and support. - Enhanced script execution by ensuring proper line endings with `sed` in multiple scripts. - Modified entry points and class references in scripts to align with new package structures. - Introduced a new Dockerfile for the producer start gate service, facilitating producer readiness management via HTTP endpoints.
- Introduced `q0-passthrough.env` for a pass-through experiment with 10k events/sec. - Added `q4-interval.env` for a single-job interval join with a constant bound. - Created `q4-join-test.env` for testing inner join functionality in Q4. - Implemented `q4-min.env` and `q4-min2.env` for minimal configurations without joins. - Added `q4-min3.env` and `q4-min4.env` for testing interval joins and outer aggregation. - Included `q4-nexmark.env` and various split templates for detailed diagnostics on auction and bid views. - Each template is configured for 500 events/sec and optimized for Flink performance.
- Introduced `q4_interval.sql` for a performant single-job interval join, optimizing state management. - Added `q4_join_test.sql` to validate join functionality without outer aggregation. - Created `q4_min.sql`, `q4_min2.sql`, `q4_min3.sql`, and `q4_min4.sql` for incremental testing of the Q4 pipeline, focusing on minimal configurations and nested aggregations. - Implemented `q4_nexmark.sql` to align with the official structure using a single source table for auctions and bids. - Added `q4_nx_auctiononly.sql` and `q4_nx_bidonly.sql` for passthrough tests of auction and bid views. - Included `q4_nx_join.sql` to isolate and test the join functionality. - Each query is designed to facilitate debugging and performance evaluation in the Flink environment.
- Added `--platform` option to specify the platform (flink|holon) without interactive prompts. - Introduced `VOLUME_RAISED` variable to track EBS volume adjustments during experiments. - Implemented cleanup logic to automatically ramp down EBS volumes after experiments, with warnings for manual intervention if necessary. - Updated documentation to reflect changes in template locations.
- Updated `run-experiment.sh` to source new modular scripts for improved clarity and maintainability. - Introduced `02-environment.sh` for platform selection and `04-volume.sh` for EBS volume management. - Adjusted sourcing order to reflect the new structure, ensuring a streamlined execution flow.
- Introduced multiple new scripts: `02-environment.sh`, `03-template.sh`, `04-volume.sh`, `05-ec2-setup.sh`, `06-run.sh`, `07-monitor.sh`, and `08-export.sh` to structure the experiment process into distinct phases. - Implemented platform selection, template management, and EC2 instance setup within the new scripts. - Enhanced user interaction for template selection and configuration, including runtime monitoring and result collection. - Added new experiment templates for various Q4 configurations to facilitate testing and performance evaluation in Flink.
…eadiness checks - Updated `run-experiment.sh` to include a new phase for prompting users to ramp down the EBS volume after experiments. - Modified `01-setup.sh` to track whether the user has been prompted about the EBS volume, preventing unnecessary warnings. - Improved `06-run.sh` to wait for all producers to announce readiness before releasing the start gate, ensuring data flows only when all producers are ready. - Introduced `09-volume-down.sh` to handle user confirmation for ramping down the EBS volume, emphasizing the importance of manual intervention due to AWS limitations.
- Introduced a new Dockerfile for the producer start gate service, setting up a Python environment with Flask. - Added `gate.py` to manage producer readiness through HTTP endpoints, allowing for synchronization of distributed runs. - Created `requirements.txt` to specify Flask dependency for the service. - Implemented endpoints for readiness checks and producer announcements, enhancing the experiment workflow.
…l script - Renamed the Docker image for the producer start gate service to align with shared deployment practices. - Enhanced comments for clarity regarding the shared nature of the start gate with the Holon deployment. - Updated the pull script to reflect the new image name and provide context on its shared usage.
- Introduced `deploy.sh` for orchestrating Holon remote deployments, managing container lifecycle and Kafka setup. - Added `docker-compose.holon.yml` to define services and configurations for the Holon experiment environment. - Created `holon-remote-pull.sh` to streamline the pulling of necessary Docker images for the Holon deployment. - Updated Dockerfiles for Holon nodes, Nexmark producers, and output consumer to use `eclipse-temurin` base images and ensure compatibility with Linux line endings. - Removed GCP credential dependencies from Dockerfiles, as the new architecture utilizes an HTTP start gate for coordination.
…emote deployment - Refactored Docker image references for the producer start gate to a shared naming convention across Flink and Holon deployments. - Enhanced comments for clarity regarding the shared nature of the producer start gate. - Introduced a new script `sync-holon-remote.sh` to facilitate syncing core deployment files for Holon to an external instance, including usage instructions and options for SSH connection.
…ments - Enhanced `03-template.sh` to define platform-specific variables for Holon and Flink, including deployment directories, sync scripts, and logging configurations. - Updated `05-ec2-setup.sh` to utilize the new platform variables, ensuring correct setup for both environments. - Modified `06-run.sh` to adjust producer settings based on the selected platform, including Kafka bootstrap server configurations and deployment commands. - Refined `07-monitor.sh` and `08-export.sh` to reflect platform-specific logging and result collection paths, improving clarity and functionality across deployments.
- Introduced `holon-q4-nexmark.env` for the Holon Q4 Nexmark interval join setup, specifying parameters for high throughput with 50 nodes. - Added `holon-q7-baseline.env` for the Holon Q7 Baseline tumbling window configuration, detailing settings for 10 nodes and HTTP start gate. - Both templates include essential parameters such as event rates, producer settings, and runtime configurations to streamline deployment and experimentation.
- Deleted unused Docker Compose files (`docker-compose.remote.yml` and `docker-compose.yml`) and associated build scripts (`local-build.sh`, `remote-pull.sh`, `start-consumer.sh`, `start-nodes.sh`, `start-producers.sh`) to streamline the project structure. - Removed Dockerfiles for Holon nodes, Nexmark producers, and output consumer, as they are no longer needed in the current deployment architecture. - This cleanup enhances maintainability and reduces confusion regarding outdated configurations.
- Created a new `.gitattributes` file to ensure that all shell scripts maintain LF line endings. This change is crucial for compatibility with the SBT "on-the-fly" Holon deployment, preventing issues with CRLF line endings in Linux containers.
- Introduced `deploy.sh` for orchestrating Holon remote deployments, managing container lifecycle and Kafka setup. - Added `docker-compose.holon-sbt.yml` to define services and configurations for the Holon experiment environment, enabling source compilation at runtime. - Created `holon-sbt-pull.sh` to pre-pull necessary Docker images for the Holon deployment, ensuring faster bring-up times. - Implemented `holon-build` and service scripts for launching Holon nodes, Nexmark producers, and output consumers, utilizing a shared classpath from compiled sources. - Updated `README.md` to reflect the new deployment approach and usage instructions for the Holon experiment runner. - Enhanced `init-kafka.sh` to utilize the Kafka CLI for topic creation, improving the setup process for Kafka services.
- Updated `03-template.sh` to support two Holon deployment profiles: `nexmark-remote-java` and `nexmark-remote`, allowing for on-the-fly compilation of Scala sources. - Added disk space management in `05-ec2-setup.sh` to prune unused Docker images when disk usage exceeds a specified threshold, preventing "no space left on device" errors. - Improved `08-export.sh` to parse raw logs into CSV metrics, providing a summary of experiment results and ensuring metrics are saved for further analysis. - Introduced `parse_output_log.py` for automated log parsing, generating CSVs for latency, throughput, state size, and message sizes from the experiment logs. - Created `build-holon-sbt-image.sh` to build and push a Docker image for the SBT "on-the-fly" Holon flow, optimizing dependency management. - Added `sync-holon-sbt-remote.sh` to facilitate syncing of Holon deployment files, including source code, to remote instances for real-time compilation.
- Created a new environment template file `holon-q7-sbt.env` for the Holon Q7 experiment, enabling on-the-fly Scala compilation with a remote deployment setup. - Defined key parameters such as workload, event rates, and node configurations to facilitate streamlined experimentation and deployment.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Major System Enhancements
Added large-scale benchmarking capabilities, multiple deployment methods, delta CRDT implementation, and NYC taxi dataset support (WIP).
Key Changes
Large-Scale Benchmarking
Multiple Deployment Methods
Delta CRDT Implementation
WindowedRecordProcFunTechnical Improvements