This repository provides a comprehensive benchmark comparison of Variational Autoencoder (VAE) implementations for time series anomaly detection. The benchmark evaluates performance across multiple dimensions:
- Hardware Acceleration: CPU vs GPU (Apple Silicon Metal Performance Shaders)
- TensorFlow/Keras: 2.13.0+ with Metal GPU acceleration
- PyTorch: 2.0.0+ with MPS backend support
- pandas: DataFrame operations and time series handling
- numpy: Numerical computing and array operations
- scikit-learn: RobustScaler preprocessing and metrics
- Plotly: Interactive plotting and anomaly visualization
- uv: Fast Python package manager and virtual environment management
- Python: 3.9 and 3.11 support
- Makefile: Build automation
- Ruff: Fast Python linter
- LSTM-based Variational Autoencoder designed for time series analysis
- Encoder-decoder architecture with configurable latent dimensions
- Binary cross-entropy reconstruction loss with KL divergence regularization
- Automated performance testing across framework and hardware configurations
- Multiple warmup iterations to ensure stable measurements
- Statistical analysis of timing results (mean, standard deviation, min, max)
- JSON export of benchmark results for further analysis
- TensorFlow: Metal GPU acceleration via tensorflow-metal
- PyTorch: Metal Performance Shaders (MPS) backend support
- Automatic device detection and fallback to CPU when GPU unavailable
- Configurable synthetic time series generator with realistic patterns
- Support for seasonal components, trends, and anomaly injection
- RobustScaler preprocessing for outlier resilience
- Efficient sequence generation for time series windowing
- Fast and reliable environment setup using
uvpackage manager - Isolated virtual environments per framework and Python version
- Reproducible builds with pinned dependency versions
Variational Autoencoders are a versatile building block, and these benchmarks are targeted at a few concrete workflows, especially in regulated manufacturing settings:
- Process anomaly detection (e.g., pharmaceutical manufacturing): VAEs learn the "normal" sensor signatures for each zone, flagging subtle drift before it becomes a deviation.
- Operational escalation planning: Once a deviation is confirmed, the same latent features feed quick triage dashboards so teams know when to escalate from laptop experimentation (outside change control) to validated retraining workflows.
- Synthetic data generation & what-if analysis: The synthetic pipeline mirrors multi-zone temperature systems so teams can prototype mitigation strategies or stress scenarios offline without exposing regulated production data.
- Dimensionality reduction for telemetry dashboards: Compressing hundreds of correlated signals into latent channels keeps downstream monitoring lightweight while preserving the ability to reconstruct raw traces when needed.
- Apple Silicon prototyping before qualified deployment: Many data scientists iterate on M-series laptops, then migrate workloads to NVIDIA-based, validated clusters. Knowing that PyTorch MPS is ~14% faster than TensorFlow Metal on GPU (and ~20% faster than TensorFlow on CPU) helps teams choose the framework that keeps local loops tight while matching the stack used in qualified compute environments.
- Enterprise Python compatibility: Because the Python 3.9 vs 3.11 delta stays below 3%, regulated environments can standardize on whichever interpreter their MLOps platform supports without losing iteration speed.
Beyond individual workflows, the benchmark design targets a few recurring MLOps pain points, particularly for validated (GxP/CSV) systems:
- Consistent dev → prod handoffs: Matching frameworks and Python versions between MacBook development and qualified training infrastructure reduces "works on my machine" drift; the benchmark table doubles as a go/no-go checklist when promoting experiments through change control.
- Faster CI signal: The repo ships with uv + Ruff + pytest automation, so platform teams can drop these jobs into GitHub Actions (or any pipeline) to validate preprocessing, model code, and benchmark scripts before artifacts hit the registry.
- Hardware-aware scheduling for controlled environments: Knowing that PyTorch/MPS is ~14% faster than TensorFlow/Metal on GPU and ~20% faster on CPU helps pipeline owners decide which framework to run on Apple Silicon CI runners (or qualified build agents), reserving pricier NVIDIA gear for validation suites that must run on fixed CUDA stacks or air-gapped networks.
- Regulatory traceability: Each run emits
results/benchmark_comparison.jsonand Plotly exports. Versioning these artifacts alongside model releases provides auditors with evidence that framework X / Python Y combinations produce consistent latent representations across dev/QA/prod, turning the benchmark into part of the validation package.
- macOS (Apple Silicon recommended for GPU benchmarks)
- uv package manager
- Install uv package manager:
curl -LsSf https://astral.sh/uv/install.sh | sh- Clone the repository:
git clone https://github.com/jvachier/Keras-vs-Pytorch-MacOs-CPU-vs-GPU.git
cd Keras-vs-Pytorch-MacOs-CPU-vs-GPU- Create virtual environments:
./setup_envs.shThis script creates four isolated environments:
.venv-tf-py39: TensorFlow with Python 3.9.venv-tf-py311: TensorFlow with Python 3.11.venv-torch-py39: PyTorch with Python 3.9.venv-torch-py311: PyTorch with Python 3.11
Alternatively, use the Makefile:
make setupInstall the optional development tooling (linters, Plotly export helpers, etc.):
uv pip install -e .[dev]If you prefer plain pip, make sure kaleido is also installed so Plotly can write static figures:
pip install -e .[dev] kaleidoExecute all benchmarks sequentially:
./run_benchmarks.shOr use the Makefile:
make runRun individual framework benchmarks:
# TensorFlow/Keras with Python 3.11
source .venv-tf-py311/bin/activate
python benchmarks/keras_vae_benchmark.py
deactivate
# PyTorch with Python 3.11
source .venv-torch-py311/bin/activate
python benchmarks/pytorch_vae_benchmark.py
deactivateEdit config.json to customize benchmark parameters:
{
"synthetic_data": {
"num_samples": 500000,
"num_features": 5,
"seasonal_amplitude": 3.0,
"seasonal_period": 1440
},
"vae_model": {
"latent_dim": 8,
"learning_rate": 0.0001
},
"data_processor": {
"sequence_length": 100,
"batch_size": 2048
},
"training": {
"epochs": 3
},
"benchmark": {
"warmup_epochs": 2,
"num_runs": 5
}
}Benchmark results are saved in the results/ directory:
keras_benchmark_results_py39.json/keras_benchmark_results_py311.json: TensorFlow performance metrics per Python versionpytorch_benchmark_results_py39.json/pytorch_benchmark_results_py311.json: PyTorch performance metrics per Python version*.log: Detailed execution logs for each benchmark run
Generate Plotly comparison charts (HTML + PNG) directly from benchmark JSON files.
uv run python scripts/plot_benchmark_results.pyOutputs:
results/benchmark_comparison_plot.html- Interactive comparisonassets/benchmark_comparison.png- Framework comparison chartassets/benchmark_speedup.png- GPU speedup chart
uv run python scripts/plot_benchmark_results.py --compare-python-versionsOutputs:
results/benchmark_python_comparison_plot.html- Interactive comparisonassets/benchmark_python_comparison.png- CPU vs GPU across Python versionsassets/benchmark_python_speedup_comparison.png- Speedup comparison chart
The static PNGs can be embedded in presentations/README, while HTML files provide interactive hover cards for deeper exploration.
| Framework | Device | Seconds / Epoch | Speedup vs CPU |
|---|---|---|---|
| TensorFlow/Keras | CPU | 213.63 s | – |
| TensorFlow/Keras | Metal GPU | 56.93 s | 3.75× |
| PyTorch | CPU | 205.02 s | – |
| PyTorch | MPS (Metal) | 53.75 s | 3.81× |
Results captured on an Apple M2 (10‑core CPU / 16‑core GPU). Expect slightly longer epochs on entry-level M1/M2 Air devices and larger gains on higher-bin chips (M1 Pro/Max, M2 Pro/Max, etc.) thanks to wider memory bandwidth and more GPU compute units.
- M1 vs M2 vs M-series Pro/Max: CPU epoch times generally scale with base clock + thermal headroom, so passive-cooled MacBook Air models may throttle sooner than Pro/Max machines under sustained training loads.
- GPU core counts: Higher-tier chips expose more GPU cores and memory bandwidth, translating to >4× speedups in practice; the ~3.8× figure reported here is representative of a mid-bin M2.
- Memory capacity: 8 GB unified memory can become a bottleneck for larger batches; prefer 16 GB+ configs to reproduce these numbers without swapping.
- Benchmarks were collected primarily on Python 3.11, but reruns on Python 3.9 stayed within ±1% of the reported epoch times. The variance is smaller than the natural run-to-run jitter from GPU warmup, so you can pick whichever interpreter version matches your dependency stack or hardware tooling.
- Both interpreter versions are still provided (
.venv-*for 3.9 and 3.11) to make it easy to validate compatibility or reproduce edge cases.
.
├── benchmarks/ # Benchmark execution scripts
│ ├── keras_vae_benchmark.py # TensorFlow/Keras benchmark
│ ├── pytorch_vae_benchmark.py # PyTorch benchmark
│ └── utils.py # Benchmarking utilities
├── src/ # Core source code
│ ├── data/ # Data processing pipeline
│ │ ├── data_processor.py # Preprocessing and sequence generation
│ │ └── synthetic_data.py # Synthetic time series generator
│ ├── models/ # VAE implementations
│ │ ├── keras_models.py # TensorFlow/Keras VAE model
│ │ └── pytorch_models.py # PyTorch VAE model
│ └── utils/ # Utility modules
│ └── logging_config.py # Logging configuration
├── requirements/ # Framework-specific dependencies
│ ├── tf-py39.txt # TensorFlow for Python 3.9
│ ├── tf-py311.txt # TensorFlow for Python 3.11
│ ├── torch-py39.txt # PyTorch for Python 3.9
│ └── torch-py311.txt # PyTorch for Python 3.11
├── config.json # Benchmark configuration
├── pyproject.toml # Project metadata and dependencies
├── setup_envs.sh # Automated environment setup
├── run_benchmarks.sh # Benchmark execution script
└── Makefile # Build automation
The VAE implementation consists of:
Encoder:
- LSTM layers: 64 → 32 → 16 units
- Latent space projection with mean and log-variance outputs
- Reparameterization trick for backpropagation
Decoder:
- LSTM layers: 16 → 32 → 64 units
- Time-distributed dense layer with GELU activation
- Reconstruction of original sequence dimensions
Loss Function:
- Reconstruction loss: Binary cross-entropy
- Regularization: KL divergence between latent distribution and unit Gaussian
- Environment Isolation: Each configuration runs in a dedicated virtual environment
- Warmup Phase: Initial iterations to stabilize GPU/CPU state
- Measurement Phase: Multiple timed runs for statistical reliability
- Metrics Collection: Timing statistics, speedup ratios, and hardware utilization
Minimum:
- Any modern x86_64 or ARM64 CPU
- 8 GB RAM
- 5 GB disk space
Recommended for GPU Benchmarks:
- Apple Silicon Mac (M1/M2/M3/M4)
- 16 GB RAM
- macOS 12.0 or later
# Create development environment
uv venv
source .venv/bin/activate
# Install with development dependencies
uv pip install -e ".[dev]"Install development dependencies:
uv pip install -e ".[dev]"Set up pre-commit hooks:
pre-commit installRun quality checks manually:
# Format code with Black
uv run black .
# Lint code with Ruff
uv run ruff check .
# Auto-fix linting issues
uv run ruff check --fix .
# Type checking with mypy
uv run mypy src/
# Security checks with Bandit
uv run bandit -r src/
# Run all pre-commit hooks
pre-commit run --all-filesThis project is licensed under the Apache License 2.0. See LICENSE for full terms.
Copyright 2025 Jeremy Vachier
If you use this benchmark in your research, please cite:
@software{vachier2025keras_pytorch_benchmark,
author = {Vachier, Jeremy},
title = {Keras vs PyTorch: macOS CPU vs GPU Benchmark},
year = {2025},
url = {https://github.com/jvachier/Keras-vs-Pytorch-MacOs-CPU-vs-GPU}
}Contributions are welcome. Please follow these guidelines:
- Fork the repository
- Create a feature branch (
git checkout -b feature/improvement) - Commit your changes (
git commit -am 'Add new feature') - Push to the branch (
git push origin feature/improvement) - Open a Pull Request
For major changes, please open an issue first to discuss proposed modifications.
Author: Jeremy Vachier
- GitHub: @jvachier
For questions or issues, please use the GitHub issue tracker.