A methodology for supervised machine learning on small datasets (typically n=30-200), where standard "throw XGBoost at it" approaches fail. Works with any AI model — packaged as a ready-to-install Claude Skill and a model-agnostic methodology.
A structured methodology that teaches an AI assistant how to approach small-sample data analysis problems systematically. It comes in two interchangeable formats:
SKILL.md— a Claude Skill with an installable YAML header (for Claude users)METHODOLOGY.md— a model-agnostic version you can paste into ChatGPT, Gemini, DeepSeek, Qwen, or any capable LLM
The methodology content is identical in both; only the packaging differs. The references/ files are fully platform-neutral and work with either entry point.
flowchart TD
subgraph entry["Two entry points — same methodology"]
SKILL["SKILL.md<br/>(YAML header)"]
METH["METHODOLOGY.md<br/>(plain prompt)"]
end
subgraph shared["Shared core — platform-neutral"]
REF["references/<br/>7 deep-dive guides"]
end
CLAUDE["Claude<br/>install as Skill<br/>or paste in chat"]
OTHERS["ChatGPT · Gemini<br/>DeepSeek · Qwen · Kimi<br/>any LLM via API"]
SKILL --> CLAUDE
METH --> OTHERS
SKILL -.shares.-> REF
METH -.shares.-> REF
REF --> CLAUDE
REF --> OTHERS
style SKILL fill:#D5E8F0,stroke:#2E75B6,color:#000
style METH fill:#D5F0DC,stroke:#27AE60,color:#000
style REF fill:#FFF4D6,stroke:#E6A817,color:#000
style CLAUDE fill:#EBF1F8,stroke:#5B9BD5,color:#000
style OTHERS fill:#EBF8EE,stroke:#52BE80,color:#000
- Blue path → Claude users get the installable
SKILL.md - Green path → everyone else uses
METHODOLOGY.mdas a prompt - Yellow core → the 7
references/files serve both, unchanged
Real-world origin: Distilled from a 30+ round conversation building a complete site-selection model for an e-commerce supermarket chain (49 stores, only 48 usable for training). The methodology survived stress-testing across model selection, residual diagnosis, iterative improvement, and 5-method overfitting validation.
Standard ML workflows assume large data. At n < 200, the defaults actively harm models:
| Default approach | What goes wrong at small n |
|---|---|
| XGBoost / LightGBM | CV-R² often negative (worse than predicting mean) |
| Single train/test split | Variance too high to draw conclusions |
| Stepwise feature selection | Picks noise as signal |
| Reporting only R² | Hides systematic regression-to-mean bias |
| "Add more features = better" | Crosses n/p ratio safety threshold |
This captures a battle-tested alternative methodology.
Trigger when:
- Sample size is small (n < 200, especially n < 50)
- Feature-to-sample ratio is concerning (p/n > 0.1)
- Stakeholders need interpretable models (not black boxes)
- You need to defend model choices against "why not XGBoost" questions
- Use cases include: site selection, B2B sales analytics, biomedical studies, A/B test analysis with limited cohorts, any "we only have N stores/patients/experiments and need to predict Y" scenario
small-sample-analysis/
├── SKILL.md Claude Skill format (installable, YAML header)
├── METHODOLOGY.md Model-agnostic version (ChatGPT/Gemini/etc.)
└── references/ Deep references — platform-neutral, on-demand
├── model_selection.md Ridge vs XGBoost/Lasso/NN — rejection logic
├── feature_engineering.md Top-K selection + dual-scheme design + segment dummies
├── residual_diagnosis.md 4-quadrant diagnostic + regression-to-mean detection
├── iteration_workflow.md The diagnose-improve-verify loop + impossible triangle
├── overfit_validation.md 5-method overfitting check battery
├── cross_validation_methods.md Multi-method triangulation (Ridge + SHAP + clustering)
└── deliverable_templates.md Notebook/report/slides canonical structures
The skill enforces six principles drilled into every analysis:
- Simplicity over flexibility — Ridge beats XGBoost at small n
- Cross-validation is non-negotiable — never report training R² as the headline
- Interpretability as hard requirement — black-box can't be defended to stakeholders
- Multi-method triangulation — confirm key findings via 2+ independent methods
- Diagnose-improve-verify is a loop — expose model flaws openly
- Statistical significance is shaky at small n — be honest about p-value limitations
Just read the files. The methodology is plain English Markdown — useful as a personal playbook even without any AI.
Install as a Skill (when Anthropic opens skill installation): the repo already follows the standard skill structure (SKILL.md + references/), so it's installation-ready. To package locally:
git clone https://github.com/jiachengwang-punch/small-sample-analysis.git
cd .. && zip -r small-sample-analysis.skill small-sample-analysis/Or paste into a conversation:
I'm starting a small-sample analysis project. Please follow this methodology:
[paste SKILL.md contents]
Then help me with: [your task]
Use METHODOLOGY.md instead of SKILL.md — it's the same methodology without the Claude-specific YAML header.
| Model | How to apply |
|---|---|
| ChatGPT | Paste METHODOLOGY.md into a Custom GPT's "Instructions", or at the start of a chat |
| Gemini | Paste into a Gem's "Instructions", or at the start of a chat |
| DeepSeek / Qwen / Kimi / others | Paste at the start of a chat as a system primer |
| Any model via API | Include METHODOLOGY.md as the system prompt |
The references/ files are platform-neutral — pull in the relevant one when you reach that step regardless of which model you use.
- Establish sample constraint upfront — state n explicitly, derive feature budget
- Data cleaning + consistency checks — sub-totals should sum to total at 1e-12 precision
- EDA with adversarial questions — hunt for hidden subgroups, counter-intuitive correlations
- Feature engineering with budget enforcement — enforce n/5 rule, use dual-scheme designs
- Model selection with explicit rejection logic — document why each alternative was rejected
- Baseline comparison (mean prediction) — R² alone is meaningless without baseline
- 5-fold CV with out-of-fold metrics — never report single train/test
- Residual diagnosis (4-quadrant) — expose model flaws honestly
- Multi-method cross-validation — confirm via 2+ independent methods
- Iteration loop (diagnose → improve → verify) — test variants systematically
- Deliverable assembly — notebook + report + presentation + Q&A handbook
See SKILL.md (Claude) or METHODOLOGY.md (any model) for full detail.
The skill emerged from a real project on a 49-store retail chain with these characteristics:
- 48 usable samples (1 store hadn't reached the target time horizon)
- 25 features in raw data, narrowed to 10 via correlation + business sense
- Target: predict 12-month order volume per store
Results achieved using this methodology:
| Pass | Approach | RMSE | R² | MAPE |
|---|---|---|---|---|
| Baseline | Predict mean | 389.6 | 0.000 | 19.2% |
| First pass | Ridge α=30 with 10 features | 342.5 | +0.227 | 15.81% |
| Final | Ridge α=30 with 10 features + 3 data-driven segment dummies | 328 | +0.292 | 14.65% |
Improvement validated using 5 independent overfitting checks — all confirming the improvement was real (one marginally significant at p=0.080, the rest strongly supporting).
These are the moves that distinguish this methodology from naïve approaches:
- Rejected XGBoost after empirically confirming CV-R² < 0 at n=48
- Discovered counter-intuitive finding — competitor count correlates +0.27 with sales (interpretation: "competitors don't randomly choose locations; their presence is a market-validation signal")
- Discovered subsidy contamination — first-month sales artificially inflated by launch promotions; used months 3-6 as the natural-baseline window instead
- Three decision-tree algorithm comparison — ID3 / C4.5 / CART. CART won (23% train-CV gap vs 52% for ID3) demonstrating "simpler binary splits win at small n"
- SHAP-revealed bimodal non-linearity — RF SHAP showed MAU feature splits cleanly around 175k threshold, independently confirmed by CART's automatic split
- 5-method overfitting validation after adding segment dummies — all 5 methods documented in
references/overfit_validation.md
Common mistakes the skill actively prevents:
| Mistake | Skill's intervention |
|---|---|
| "Let me just throw XGBoost at it" | Shows empirical CV-R² < 0 evidence, redirects to Ridge |
| "R² 0.4 is bad, model is useless" | Compares to baseline; small-sample R² of 0.2-0.4 is often state-of-the-art |
| "Let me use all 30 features" | Enforces n/5 budget |
| "Train R² is 0.85, ship it" | Forces out-of-fold CV; expect 50%+ drop |
| "This model is perfect" | Forces residual diagnosis; finds regression-to-mean |
| "Just use 80% quantile as threshold" | Data-driven (decision tree) thresholds typically win |
Platform-agnostic by design — two entry points, same methodology:
| Platform | Entry point | How |
|---|---|---|
| Claude (web/mobile/desktop) | SKILL.md |
Paste into conversation, or install as Skill |
| Claude API / Claude Code | SKILL.md |
Include as system prompt or CLAUDE.md |
| ChatGPT | METHODOLOGY.md |
Custom GPT Instructions, or paste in chat |
| Gemini | METHODOLOGY.md |
Gem Instructions, or paste in chat |
| DeepSeek / Qwen / Kimi / others | METHODOLOGY.md |
Paste as system primer |
| Any model via API | METHODOLOGY.md |
Use as system prompt |
The references/ files work with all of the above.
Contributions welcome! Especially useful additions:
- Domain-specific extensions (medical small-sample, A/B test cohorts, agricultural trials)
- Code templates in the
scripts/folder for common pipeline steps - New reference cases in
examples/showing the methodology applied to other datasets - Translations of SKILL.md and references
Please open an issue first to discuss major changes.
MIT License — see LICENSE file for details.
- Inspired by the experience of completing a real site-selection analysis project under data constraints
- Builds on classical statistical learning theory (Hastie, Tibshirani, Friedman) and modern interpretable ML (SHAP)
- Designed for the Claude Skill format
If you use this methodology in academic work, please cite:
Small-Sample Analysis Skill (2026)
A methodology for supervised machine learning on small datasets.
Available at: https://github.com/YOUR_USERNAME/small-sample-analysis
Star this repo if you find it useful! ⭐