Real-time drift detection for multilingual LLM traffic, with adaptive response strategies under domain shift.
DriftGuard-GALA is a research-oriented ML systems project built around a simple production question: when English-heavy traffic suddenly shifts toward lower-resource languages like Hindi or Yoruba, how should a deployed LLM respond?
This repository implements and compares multiple adaptation strategies on top of a real Qwen/Qwen2.5-7B-Instruct runtime, Aya multilingual data, PSI-based drift detection, selective LoRA weight updates, and adaptive prompt retrieval.
The most important result from the current project is not theoretical. It is practical:
In this setup, selective retrieval-based adaptation works better than online LoRA weight updates.
Run the portfolio demo with one command:
./scripts/demo.shThat command regenerates:
docs/latest_results.jsondocs/RESULTS.mddocs/assets/*.svg
If dependencies are not installed yet:
pip install -r requirements.txt
- Real LLM systems work: Qwen 2.5 7B runs in 4-bit mode with PEFT adapters.
- Streaming-style monitoring: PSI is used to detect multilingual distribution shift over replayed inference traffic.
- Multiple adaptation families: the repo compares online weight updates against adaptive prompting.
- Ablation discipline: not just one idea, but several variants tested side by side.
- Honest engineering conclusions: the best method is the one that actually improves metrics, not the one that sounds most sophisticated.
| Area | Current implementation |
|---|---|
| Base model | Qwen/Qwen2.5-7B-Instruct |
| Runtime | 4-bit quantization with bfloat16 compute |
| Drift signal | PSI over real model-derived features |
| Shift simulation | English baseline -> Hindi / Yoruba shift |
| Weight adaptation methods | selective LoRA, replay-regularized LoRA, warmup + broader LoRA ablations |
| Prompt adaptation methods | retrieval-based in-context adaptation |
| Outputs | JSON metrics, Markdown report, SVG charts, demo page |
| Path | Role |
|---|---|
streaming/ |
replay ingestion, drift features, PSI calculation, trigger logic |
model/ |
model loading, feature extraction, adaptation planning, selective update runtime |
eval/ |
metrics and evaluation runners |
scripts/ |
experiment runners, portfolio pipeline, report/chart generation |
docs/ |
generated results, charts, and web demo assets |
configs/ |
MVP configuration |
The current best-performing adaptive method is:
- Method:
selective_retrieval_adaptive - Family: adaptive prompting
- Idea: preserve the English baseline path and only apply in-context retrieval exemplars for shifted Hindi/Yoruba prompts
| Metric | Frozen Baseline | Best Adaptive | Delta |
|---|---|---|---|
| Overall token F1 | 0.2270 |
0.2570 |
+0.0300 |
| Hindi token F1 | 0.2947 |
0.3223 |
+0.0276 |
| Yoruba token F1 | 0.0254 |
0.0879 |
+0.0625 |
| English token F1 | 0.3608 |
0.3608 |
+0.0000 |
This is the first result in the project that clearly shows adaptation improving the deployed system rather than degrading it.
The project still includes the original GALA-style weight-update experiments, and they are useful because they show what did not work in this setup.
| Target | Method | Target Delta F1 | Overall Delta F1 |
|---|---|---|---|
| hi | target_only_gala |
-0.2947 |
-0.1979 |
| hi | replay_regularized_gala |
-0.2947 |
-0.1979 |
| hi | warmup_target_gala |
-0.2947 |
-0.1979 |
| hi | warmup_replay_broad_gala |
-0.2947 |
-0.1979 |
| yo | target_only_gala |
-0.0043 |
-0.1979 |
| yo | replay_regularized_gala |
-0.0043 |
-0.1979 |
| yo | warmup_target_gala |
-0.0043 |
-0.1979 |
| yo | warmup_replay_broad_gala |
-0.0043 |
-0.1979 |
That makes the conclusion much stronger: the repository does not just claim a method works, it demonstrates that several plausible alternatives were tested and underperformed.
The cleanest story for a reviewer is:
- The system architecture works end to end: drift detection, adaptation trigger, evaluation, and reporting.
- Online weight updates were implemented and tested in multiple variants.
- In this small, fast multilingual setting, weight updates were unstable or harmful.
- Selective retrieval adaptation turned out to be the best operational strategy.
- Therefore, the project delivers both a negative research finding and a positive engineering finding.
That is a strong portfolio outcome because it shows:
- rigorous experimentation
- willingness to reject weaker methods
- practical systems judgment
- real improvement on shifted-language performance
pip install -r requirements.txt./scripts/demo.shPYTHONPATH=/workspace/DriftGuard-GALA python /workspace/DriftGuard-GALA/scripts/run_experiment_suite.py
PYTHONPATH=/workspace/DriftGuard-GALA python /workspace/DriftGuard-GALA/scripts/run_context_adaptation_eval.py
PYTHONPATH=/workspace/DriftGuard-GALA python /workspace/DriftGuard-GALA/scripts/generate_job_report.py
PYTHONPATH=/workspace/DriftGuard-GALA python /workspace/DriftGuard-GALA/scripts/generate_result_charts.py- Increase the evaluation slice so the results are less sample-size sensitive.
- Add a learned gating policy that chooses between baseline, retrieval, and weight adaptation.
- Add PSI-over-time plots and per-language recovery-over-time plots.
- Improve the LoRA branch with stronger regularization and richer validation gating.
- Explore hybrid methods that combine retrieval adaptation with later offline adapter training.