Cardiac MLP Benchmark

A controlled benchmark of a multi-layer perceptron, MLP, against my earlier class-weighted support vector machine, SVM, for predicting atrial fibrillation after myocardial infarction.

The purpose is not to prove that a neural network is better. The purpose is to test it under a closely matched protocol and report the result honestly, including a loss.

Research question

On the same 1,700-patient UCI myocardial infarction dataset, with the same target, held-out split, and headline metrics, does an MLP improve on the previous class-weighted SVM?

Previous project: Machine-Learning-and-Data-Mining

Serving project: cardiac-inference-api - FastAPI REST inference service for the selected MLP, with request validation, imputation disclosure, hermetic pytest tests and GitHub Actions CI.

Dataset: UCI Myocardial Infarction Complications, dataset 579, 1,700 patients and 111 input features. Target: FIBR_PREDS.

Why this benchmark matters

The earlier cardiac project showed why accuracy alone is dangerous in imbalanced clinical classification. A majority-class-heavy model can look accurate while missing most patients who develop the complication. The previous class-weighted SVM achieved 38.24% sensitivity, 87.58% specificity, and ROC-AUC 0.7215.

This repository asks a narrow follow-up question: does a feed-forward neural network improve that trade-off when evaluated on the same task?

Matched protocol

• UCI dataset id 579
• target FIBR_PREDS
• remove features with at least 40% missingness
• median imputation for numeric variables and mode imputation for categorical variables
• 80/20 stratified train/test split
• random_state=42
• StandardScaler fitted without using the held-out test set
• 5-fold stratified cross-validation for MLP hyperparameter selection
• headline metrics: sensitivity, specificity, ROC-AUC
• held-out test set used only for final evaluation

Important fairness decision

Current scikit-learn versions support sample_weight for MLPClassifier. The MLP is therefore trained with balanced sample weights so that the primary comparison is conceptually closer to the earlier class-weighted SVM.

The primary MLP result uses the normal 0.50 probability threshold. A second MLP row uses Youden's J, selected only from out-of-fold training probabilities. That second row is useful for clinical operating-point analysis, but it is not described as an estimator-only comparison because the threshold has also changed.

MLP search space

The benchmark tests these architectures:

• one hidden layer with 64 units
• one hidden layer with 128 units
• two hidden layers with 64 and 32 units

For each architecture it tests L2 regularisation values 1e-4, 1e-3, and 1e-2. ReLU activation and Adam optimisation are held fixed.

Results

Selected model

• Architecture: one hidden layer with 64 units
• L2 regularisation (alpha): 0.01
• Mean 5-fold training CV AUC: 0.6661
• Held-out test AUC: 0.7505
• Alternative-seed mean AUC: 0.7138 ± 0.0222

Model	Imbalance handling	Threshold	Sensitivity	Specificity	AUC
Previous class-weighted SVM	class weights	default boundary	38.24% (13/34)	87.58%	0.7215
MLP (64,) PRIMARY	balanced sample weights	0.500	73.53% (25/34)	68.30%	0.7505
MLP (64,) SECONDARY	balanced sample weights	Youden 0.382	76.47% (26/34)	44.12%	0.7505

Honest discussion

On the held-out test split, the selected MLP (64,) achieved an AUC of 0.7505 compared with 0.7215 for the previous class-weighted SVM, while sensitivity increased substantially from 38.24% (13/34) to 73.53% (25/34). This improvement came with lower specificity, which fell from 87.58% to 68.30%.

The held-out AUC of 0.7505 was also higher than the mean 5-fold training CV AUC of 0.6661, which provides an additional reason not to overinterpret performance on this single test split.

The single-run AUC improvement should not, however, be interpreted as evidence that the MLP is robustly superior. Across three alternative model initialisation seeds using the same split and selected hyperparameters, mean MLP AUC was 0.7138 ± 0.0222, with sensitivity of 57.84% ± 5.00% and specificity of 75.27% ± 7.78%. The previous SVM AUC of 0.7215 therefore lies comfortably within the range of performance observed across MLP initialisations.

The most defensible finding is that the MLP produced a more sensitivity-oriented prediction profile, identifying substantially more atrial fibrillation cases but at the cost of additional false positives. Overall discrimination remained broadly comparable to the SVM and was sensitive to neural-network initialisation.

The secondary Youden threshold of 0.382 increased sensitivity only modestly, from 73.53% to 76.47%, while specificity fell sharply from 68.30% to 44.12%. It is therefore reported as an operating-point analysis rather than evidence of superior estimator performance.

These results reinforce the purpose of the benchmark: model complexity alone does not justify model selection. Performance, stability, clinical error trade-offs, and reproducibility all need to be considered together.

Limitations

• Only 1,700 patient records are available, which is small for a neural network.
• The data come from one source and there is no external validation cohort.
• The benchmark retains the preprocessing conventions from the previous project for comparability. Feature filtering and imputation occur before the outer split, which can allow limited distributional information from the held-out set into preprocessing. A stricter future version should fit every preprocessing step inside training folds and rerun both SVM and MLP from scratch.
• The SVM numbers are carried forward from the previous project rather than recomputed inside this repository.
• Threshold tuning changes the operating point, so the tuned MLP row is secondary rather than the primary estimator comparison.

Reproduce

Create and activate an environment, then install dependencies:

python -m venv .venv
# Windows PowerShell: .venv\Scripts\Activate.ps1
# macOS/Linux: source .venv/bin/activate
pip install -r requirements.txt

For normal installation, use requirements.txt. For exact reproduction of the environment used for the published benchmark, use requirements-lock.txt, which pins the package versions from the project virtual environment.

Smoke-test the pipeline with fake data:

python mlp_benchmark.py --synthetic

This creates results_synthetic.md and results_synthetic.json. They are intentionally ignored by Git and must never be pasted into the README.

Run the real UCI benchmark:

python mlp_benchmark.py

The real run creates:

• results.md, human-readable result table and stability summary
• results.json, machine-readable metrics and selected hyperparameters

What this demonstrates

This small project demonstrates controlled model benchmarking, class-imbalance handling, cross-validation, threshold selection without touching the test set, sensitivity/specificity interpretation, reproducible experiment design, and honest reporting of negative results.

Data source

Golovenkin, S. et al. (2020). Myocardial infarction complications. UCI Machine Learning Repository. DOI: 10.24432/C53P5M. The UCI dataset is distributed under CC BY 4.0.