🌱 SPORE - Soil-based Prediction Of Resident Entities

Advanced hierarchical machine learning system for predicting soil microbes in India based on soil characteristics and location.

Features

Hierarchical Classification: Predicts at 5 taxonomic levels (Phylum → Class → Order → Family → Genus)
Dual Input Modes:
- Lat/Lon lookup (automatic soil data retrieval)
- Manual soil characteristics input
High Accuracy: F1 scores up to 0.435 (6.5x improvement over baseline)
Multi-Level Predictions: Get predictions at all taxonomic levels simultaneously
Confidence Scores: High/Medium/Low confidence indicators
Professional PDF Reports: Comprehensive scientific reports with:
- Dynamic visualizations
- AI-powered explanations using Ollama (Mistral 7B)
- Executive summary, soil analysis, methodology sections
- Professional scientific styling and formatting

Model Performance

Taxonomic Level	Classes	F1 Score	Accuracy
Phylum	21	0.435	52.6%
Class	33	0.336	42.3%
Order	74	0.209	26.8%
Family	125	0.173	23.2%
Genus	193	0.155	20.8%

Why hierarchical + XGBoost is better: Reduces label dimensionality, handles class imbalance, and provides interpretable taxonomic predictions.

Quick Start

1. Install Dependencies

pip install -r requirements.txt

2. Train Models (First Time Only)

python3 train_model.py

This will:

Link 79,648 bacteria occurrences to soil data
Train hierarchical models at 5 taxonomic levels
Save models to models/ directory
Takes ~5-10 minutes

3. Setup Ollama (Optional - for AI explanations)

# Install Ollama
brew install ollama

# Pull Mistral model
ollama pull mistral:7b

# Start Ollama service (runs in background)
ollama serve

4. Run the Application

python3 app.py

Open: http://localhost:5002

Note: Predictions are currently available for locations within India only.

Usage

Mode 1: Lat/Lon Lookup

Enter latitude and longitude (within India)
System automatically retrieves soil data from HWSD2 database
Get predictions at all taxonomic levels

Mode 2: Manual Soil Input

Enter all 8 soil characteristics:
- Gravel Content (%)
- Sand Content (%)
- Silt Content (%)
- Clay Content (%)
- Organic Carbon (g/kg)
- Soil pH
- CEC of Clay (cmol(+)/kg)
- Bulk Density (kg/dm³)
Optionally provide lat/lon for spatial features
Get predictions based on your soil data

API Usage

# Mode 1: Lat/Lon with AI explanations
curl -X POST http://localhost:5002/predict \
  -H "Content-Type: application/json" \
  -d '{"latitude": 22.793497, "longitude": 73.62895, "enable_ai": true}'

# Mode 2: Manual Soil without AI explanations
curl -X POST http://localhost:5002/predict \
  -H "Content-Type: application/json" \
  -d '{
    "soil_characteristics": {
      "COARSE": 10, "SAND": 45, "SILT": 30, "CLAY": 25,
      "ORG_CARBON": 12, "PH_WATER": 6.5, 
      "CEC_CLAY": 55, "BULK": 1.4
    },
    "enable_ai": false
  }'

Project Structure

SPORE/
├── app.py                  # Flask application (hierarchical predictor)
├── train_model.py          # Training pipeline
├── report_generator.py     # PDF report generation
├── requirements.txt        # Python dependencies
├── README.md              # This file
├── LICENSE                # MIT License
├── data/
│   ├── HWSD2.bil          # Soil raster (23GB)
│   ├── HWSD2.hdr          # Raster header
│   ├── HWSD2_SMU.csv      # Soil mapping units
│   ├── HWSD2_LAYERS.csv   # Soil characteristics
│   └── raw_gbif_data.csv  # Bacteria occurrences
├── models/
│   ├── phylum_model.pkl   # Phylum classifier
│   ├── phylum_encoder.pkl # Phylum label encoder
│   ├── class_model.pkl    # Class classifier
│   ├── class_encoder.pkl  # Class label encoder
│   ├── order_model.pkl    # Order classifier
│   ├── order_encoder.pkl  # Order label encoder
│   ├── family_model.pkl   # Family classifier
│   ├── family_encoder.pkl # Family label encoder
│   ├── genus_model.pkl    # Genus classifier
│   ├── genus_encoder.pkl  # Genus label encoder
│   ├── scaler.pkl         # Feature scaler
│   ├── features.pkl       # Feature names
│   └── metadata.json      # Model metadata & metrics
├── templates/
│   └── index.html         # Web UI
└── static/
    └── background.jpg     # UI background

How It Works

Training Phase

Data Loading: Load HWSD2 soil database + GBIF bacteria occurrences
Taxonomy Extraction: Parse hierarchical taxonomy from GBIF data
Spatial Linking: Link bacteria coordinates to soil SMU_IDs via raster
Feature Engineering: Create 13 features (8 soil + 2 spatial + 3 derived)
Hierarchical Training: Train separate Random Forest models for each taxonomic level
Evaluation: Calculate F1 scores and accuracy metrics
Model Saving: Persist all models and metadata

Prediction Phase

Input Processing: Accept lat/lon OR manual soil characteristics
Feature Preparation: Create 13-feature vector with derived features
Scaling: Normalize features using trained scaler
Hierarchical Prediction: Run through all 5 taxonomic level models
Confidence Scoring: Classify predictions as high/medium/low confidence
Response Formatting: Return predictions organized by taxonomic level
Report Generation: Create PDF with all predictions

Key Features

Hierarchical Classification

Follows biological taxonomy: Phylum → Class → Order → Family → Genus
Better F1 scores than flat multi-label classification
More interpretable and scientifically sound
Can predict at multiple levels simultaneously

Dual Input Modes

Lat/Lon Mode: Automatic soil lookup from 408K+ soil records
Manual Mode: Direct input of soil characteristics
Flexible for different use cases and data availability

Advanced ML

Random Forest classifiers (100 trees each)
Balanced class weights for imbalanced data
Spatial autocorrelation via lat/lon features
Feature engineering (ratios, texture index)

Feature Importance

Top features across all models:

Longitude (21.3%) - Geographic location
Latitude (19.6%) - Geographic location
Organic Carbon (6.6%) - Nutrient availability
CEC of Clay (6.1%) - Nutrient retention
Bulk Density (5.9%) - Soil structure

Data Sources

HWSD v2.0 (Harmonized World Soil Database)

Source: FAO/IIASA
URL: https://gaez.fao.org/pages/hwsd
Content: 29,538 soil mapping units, 408,835 records
Citation: FAO/IIASA/ISRIC/ISSCAS/JRC, 2012

GBIF (Global Biodiversity Information Facility)

Source: GBIF.org
URL: https://www.gbif.org/
Region: India only (countryCode: IN)
Content: 96,697 bacteria occurrences
Species: 5,261 unique species
Citation: GBIF.org (2 October 2025) GBIF Occurrence Download https://doi.org/10.15468/dl.z3ysfj

Technical Details

Dependencies

ML: XGBoost 2.0+, scikit-learn 1.7+, joblib
Data: pandas, numpy, rasterio
Web: Flask 2.0+
Reports: reportlab, matplotlib, seaborn
AI: Ollama (Mistral 7B for offline explanations)

System Requirements

RAM: 8GB minimum (16GB recommended)
Storage: 25GB for data + models
CPU: Multi-core recommended for training
OS: macOS, Linux, Windows

Performance

Training Time: 5-10 minutes
Prediction Time: <1 second per location
Model Loading: ~2 seconds on startup
Report Generation:
- Without AI: ~10-15 seconds
- With AI explanations: ~3-4 minutes (Mistral 7B)

Troubleshooting

Models not loading?

# Retrain models
python3 train_model.py

Missing data files?

Download required datasets:

HWSD2 Raster: https://gaez.fao.org/pages/hwsd
GBIF Data: https://www.gbif.org/ (filter to India, Bacteria)

Port already in use?

# Change port in app.py (last line)
app.run(debug=True, port=5003)  # Use different port

AI explanations not working?

# Check if Ollama is running
curl http://localhost:11434/api/tags

# Start Ollama service
ollama serve

# Or disable AI in the UI (uncheck the toggle)

Future Improvements

Add embedding-based approaches (Node2Vec on co-occurrence networks)
Implement SHAP explainability per prediction
Add temporal/seasonal features
Cross-validation for robust metrics
Deploy to production server
Add more taxonomic levels (species, strain)
Integrate climate data
Expand to other countries/regions (currently India-specific)

Citation

If you use this project, please cite:

@software{spore2025,
  title={SPORE: Soil-based Prediction Of Resident Entities},
  author={Sarvagna},
  year={2025},
  version={2.0-hierarchical},
  url={https://github.com/yourusername/spore}
}

License

MIT License - See LICENSE file

Acknowledgments

FAO for HWSD v2.0 soil database
GBIF for bacteria occurrence data
scikit-learn community for ML tools
Flask community for web framework

Made with ❤️ by Sarvagna

Name		Name	Last commit message	Last commit date
Latest commit History 32 Commits
models		models
static		static
templates		templates
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
Task_LS.pdf		Task_LS.pdf
app.py		app.py
report_generator.py		report_generator.py
requirements.txt		requirements.txt
train_model.py		train_model.py
train_model_fast.py		train_model_fast.py

Folders and files

Latest commit

History

Repository files navigation