Datathon 2026 Postmortem

This repository contains our experimental solution attempts, leaderboard feedback analysis, and postmortem notes for Datathon 2026.

The goal of the competition was to predict career_success_score for student profiles using tabular, categorical, and Turkish mentor feedback text features. The evaluation metric was mean squared error (MSE), so lower is better.

Result

Item	Value
Best public score	84.628583
Best submission	v17_v7a0446_plus_v9safe_g1p5.csv
Final rank	129 / 528
Winner public score	80.348718
Total submissions used	25

This was not a winning solution. The repository is kept as a technical postmortem: what worked, what failed, and what should be done differently in a future tabular/text leaderboard competition.

What Worked

• AutoGluon tabular/text stacking (v7) produced a direction that worked much better on the public leaderboard than our internal recent-year validation suggested.
• Text residual modeling (v9) improved the final public score when blended on top of the V7 direction.
• Public-feedback one-dimensional blend sweeps were useful once a working direction was found.
• Final V9-safe gamma sweep improved the score from 85.085835 to 84.628583.

What Failed

• Random KFold and many local OOF metrics did not represent the public leaderboard well.
• Formula-style attacks (v8) looked strong locally but did not transfer to the public leaderboard.
• Early submissions spent too much budget on classical ML blends and calibration variants before the strongest V7/V9 directions were discovered.
• Aggressive extrapolation (g2.5) worsened the score, showing the final text-residual direction had a limited useful range.

Repository Structure

.
├── docs/
│   ├── postmortem.md
│   ├── submission_history.md
│   └── working_log_tr.txt
├── notebooks/
│   └── datathon_2026_colab_leaderboard.ipynb
├── scripts/
│   ├── leaderboard_candidates.py
│   ├── make_colab_notebook.py
│   ├── postprocess_submissions.py
│   └── public_hedge_blends.py
├── src/
│   ├── datathon_2026_leaderboard_pipeline.py
│   └── blend_stages/
│       ├── v2_weighted_lgbm_run.py
│       ├── v3_formula_attack.py
│       ├── ...
│       └── v17_final_push.py
├── requirements-colab.txt
└── .gitignore

Data

The competition dataset is not included in this repository.

Place the competition files in the working directory before running the scripts:

train.csv
test_x.csv
sample_submission.csv

Generated outputs such as datathon_outputs*/, model files, OOF arrays, and submission CSV files are also intentionally excluded.

Main Pipeline

The original baseline pipeline is available as both a notebook and a Python script:

python src/datathon_2026_leaderboard_pipeline.py

For Colab, use:

notebooks/datathon_2026_colab_leaderboard.ipynb

Install dependencies:

pip install -r requirements-colab.txt

Some later experiments require AutoGluon:

pip install autogluon.tabular

Important Experiment Stages

• v3_formula_attack.py: formula-style feature engineering and text-template features.
• v4_residual_attack.py: residual correction on top of the best public hedge.
• v5_formula_text_gated.py: formula/text/year-role gated residual models.
• v6_public_parabola_blends.py: public-feedback blend curve fitting.
• v7_autogluon_stack.py: AutoGluon tabular/text stack. This became one of the most useful public directions.
• v8_formula_decoder_attack.py: stronger formula/text/censored target attack. It looked good locally but did not transfer well.
• v9_text_similarity_attack.py: TF-IDF/KNN/cluster text residual model. This became the final useful correction direction.
• v14_final_day_pack.py to v17_final_push.py: final-day public feedback blend candidates.

Lessons Learned

The biggest mistake was treating the task too long as a generic model/ensemble competition. The data likely had a synthetic target-generation mechanism and strong template signals in mentor_feedback_text. The strongest future approach would start with:

1. Public-like validation based on application_year, graduation_year, target_role, and adversarial train/test similarity.
2. Early AutoML sweeps across multiple seeds and weighting schemes.
3. Dedicated text-template decoding before spending submissions on calibration.
4. Public feedback used as hypothesis tests, not as blind leaderboard tuning.

See docs/postmortem.md for the longer write-up.

Note

This repository is for learning and retrospective analysis. It does not include competition data or generated submissions.