End-to-end multimodal AI system for plant disease detection from leaf images using deep learning, CLIP validation, and RAG-based advisory delivered via WhatsApp.
AgroMind GenAI is an end-to-end AI system that enables farmers to diagnose plant diseases by sending leaf images via WhatsApp. The platform combines deep learning–based computer vision, multimodal learning, and a Retrieval-Augmented Generation (RAG) module to provide accurate diagnosis and actionable farming guidance through a conversational interface.
Plant diseases can significantly reduce crop yield if not detected early. AgroMind GenAI provides an automated solution that analyzes plant leaf images and delivers treatment recommendations without requiring physical access to agricultural experts.
• Automatic plant disease detection from leaf images
• CLIP-based validation to reject non-leaf images
• Multimodal prediction using image + environmental inputs
• AI advisory chatbot powered by RAG
• Real-time interaction via WhatsApp
• Context-aware responses based on detected disease
• Prediction history and chat storage
• Feedback collection for system improvement
Disease Classification Model: Multimodal CNN (EfficientNet-B2 backbone)
Framework: PyTorch with Transfer Learning
Leaf Validation Model: CLIP (Vision-Language Model)
Advisory System: Retrieval-Augmented Generation (RAG)
-
Farmer sends a leaf image via WhatsApp
-
WhatsApp Business API forwards the message to FastAPI webhook
-
Server downloads the image from Meta servers
-
CLIP model checks whether the image contains a plant leaf
• If NOT a leaf → Request a valid leaf image
• If leaf → Continue -
Image is preprocessed and passed to the Multimodal CNN model
-
Model predicts disease class and confidence score
• Low confidence → Ask for clearer image
• Valid prediction → Continue -
Prediction result is stored in SQLite database
-
RAG system retrieves disease knowledge
-
System generates explanation (symptoms, treatment, prevention)
-
Response is sent back to the farmer via WhatsApp
- Farmer sends a text query
- Webhook receives the message
- System retrieves the user's latest disease prediction
- Question Mapper detects intent (symptoms, cause, treatment, etc.)
- Sentence Transformer generates embeddings
- Vector search retrieves relevant knowledge from ChromaDB
- RAG pipeline generates grounded response
- Answer is sent via WhatsApp
- Chat history and feedback are stored
Dataset → Data Augmentation → DataLoader
→ Multimodal CNN Training → Loss Optimization
→ Model Evaluation → Best Model Saved
Image Size: 192 / 224
Batch Size: 8
Optimizer: Adam
Loss Function: CrossEntropyLoss
Data augmentation:
• Random crop
• Rotation
• Horizontal & vertical flip
• Color jitter
• Perspective distortion
• Gaussian blur
• Normalization
Evaluation metrics:
• Accuracy
• F1 Score
• Confusion Matrix
• Classification Report
Embedding Model: Sentence Transformers (MiniLM — Hugging Face ecosystem)
Vector Database: ChromaDB
Framework: LangChain
Provides grounded responses about:
• Symptoms
• Causes
• Treatment
• Prevention
• Recovery time
• Cost
AgroMind GenAI
│
├── README.md
├── requirements.txt
├── train.py
├── agromind.db
├── .env
│
├── app
│ ├── main.py
│ ├── config.py
│ ├── api
│ ├── data
│ ├── dataset
│ ├── ml
│ ├── models
│ ├── rag
│ ├── services
│ └── utils
│
├── venv
├── .vscode
└── __pycache__
Install dependencies:
pip install -r requirements.txt
Train the model:
python train.py
Run the API server:
uvicorn app.main:app --reload
Configure environment variables in .env:
WHATSAPP_ACCESS_TOKEN=your_token
WHATSAPP_PHONE_NUMBER_ID=your_id
WEBHOOK_VERIFY_TOKEN=your_verify_token
Python • PyTorch • Computer Vision • Deep Learning
EfficientNet • CLIP • Multimodal AI
FastAPI • NLP • RAG • LangChain
Sentence Transformers • ChromaDB • SQLite
Git • GitHub • WhatsApp Business API
• Integration with real-time weather APIs
• Multilingual support for farmers
• Cloud deployment for scalability
• Mobile application interface
• Support for additional crops and diseases
• Continuous model retraining using collected feedback
• .env, venv, and cache files are not included for security reasons
• Large model files may be excluded due to repository size limits
• Requires valid WhatsApp Business API credentials to run fully
Released under the MIT License.
Gajanand L Immannavar
AI/ML Enthusiast | Computer Vision | Generative AI