A hands-on journey into LLMs, Tokenization, RAG pipelines, PDF vector processing, API development, and specialized AI micro-applications.
Welcome to my Generative AI learning repository! This project documents my step-by-step progress in understanding the core foundations of GenAI and building practical, modular applications using these concepts.
Includes Python scripts where I learned:
- How tokenizers work
- Creating custom vocab files
- Training BPE tokenizers
- Understanding merges, vocab length, and token distribution Folder: Vocab / Root Python scripts
Hands-on implementation of a document Q&A system:
- PDF text extraction and chunking with document tracking
- Converting documents into embeddings
- Vector indexing and similarity search using FAISS
- Building a full RAG pipeline for answering queries with context attribution Folder: Rag Model
A specialized FastAPI application for Open Source Intelligence:
- Asynchronous data gathering from GitHub, Reddit, and Breach databases
- Dynamic relationship graph generation
- Vis.js frontend visualization Folder: Digital Detective
A sentiment analysis tool powered by Hugging Face:
- Uses the
distilbertinference API - Analyzes text sentiment and returns confidence scores and emojis
- Social feed analyzer to gauge sentiment of live news headlines Folder: Mood Analyzer
An automated ATS (Applicant Tracking System) style application:
- Extracts text and skills from uploaded resumes
- Compares candidate profiles against specific job descriptions
- Uses AI-based parsing for structured data extraction Folder: Resume Matcher
Beginner-level FastAPI projects to understand backend fundamentals:
- HTTP GET & POST requests, CRUD operations
- Path & query parameters
- Async routes Folders: API and fastapi-todo-main
- 🧠 Generative AI Foundations
- 🔤 Tokenization (LLM Tokenizer Concepts)
- 📚 Embeddings & Vector Databases (FAISS)
- 🔍 RAG Workflow (Retriever + Generator)
- ⚙️ FastAPI for AI Backend Integration
- 📝 Working with PDF Text & Chunking
- 🐍 Python for AI & NLP Workflows
- 🌐 Creating Visual UIs for APIs
This repo contains my personal learning journey as I explore the core building blocks behind modern LLM systems and GenAI applications. I am documenting everything from basics to advanced workflows.
✔️ Tokenizer Training ✔️ PDF Vector Processing & Basic RAG ✔️ OSINT & Sentiment Analysis APIs 🔜 Building a Unified GenAI Dashboard Web App 🔜 Advancing the RAG Chatbot (Conversational Memory) 🔜 Expanding Digital Detective with more OSINT modules 🔜 Deploying FastAPI backend
If you have ideas to improve the code or learning roadmap, feel free to open an issue or PR!
If you like this repo, consider giving it a star ⭐ — it helps me stay motivated to learn more every day!