GitHub PR/Issue Classifier

Fine-tuned DistilBERT multi-task classifier — sentiment, GitHub labels, priority scoring, web UI & n8n automation.

GitPR Analyzer · Production-ready ML for open-source triage

Language & Model

Frameworks

DevOps & Automation

Legal

📑 Table of Contents

🤖 About the Project
🔗 Quick Links
🚀 Live Demo
✨ Features
📂 Repository Structure
🏗️ Architecture & Workflow
📚 Documentation Deep-Dive
⚙️ Getting Started
💡 Usage
🧠 Model Information
⭐ Star History
🛣️ Roadmap
🤝 Contributing
🗺️ Credits & Acknowledgements
📜 License
📬 Contact & Support

About the Project

GitHub PR/Issue Classifier (also branded GitPR Analyzer in the web UI) is an end-to-end machine learning system that reads GitHub pull request and issue text, predicts multi-label GitHub taxonomy labels, three-way sentiment, and a rule-based priority tier suitable for triage and automation. The production checkpoint (models/pr_classifier.pt) is served by a FastAPI inference layer and consumed through a Flask single-page web app and an exported n8n workflow.

🧠 The model is built on distilbert-base-uncased with a shared encoder and two task-specific heads (MultiTaskDistilBERT): a 3-class sentiment head (CrossEntropy) and a 5-label multi-label GitHub head (BCEWithLogits with positive class weighting). DistilBERT was chosen for a strong accuracy-to-latency tradeoff on issue/PR text while keeping GPU memory and container size practical for Hugging Face Spaces and local development.

🚀 Dual deployment runs both services in one container: FastAPI on port 8000 (model + GitHub REST automation) and Flask on port 7860 (or the host PORT on Spaces) via deployment/start.sh. The public Hugging Face Space exposes the Flask UI; FastAPI remains available inside the container for proxying and automation.

🔗 n8n automation listens for GitHub webhook events, normalizes issue/PR payloads, calls POST /predict, optionally creates missing repository labels, and writes final_labels back to GitHub—enabling headless triage without opening the browser UI.

Why this project?

Maintainers spend significant time reading new issues and PRs, agreeing on labels, and prioritizing work. This project compresses that loop into a reproducible pipeline: curated training data → multi-task fine-tuning → a single checkpoint → the same API contract for humans (UI) and machines (n8n). The priority engine layers operational labels (critical, high-priority, normal-priority, low-priority) on top of model outputs so automation can route urgent work without a separate ranking model.

Quick Links

🌐 Live Web App

⭐ GitHub Repository

⚙️ n8n Workflow JSON

📦 GitHub Issues Dataset

💬 Sentiment Dataset

🤗 Model Hub

Note

Model Hub: The fine-tuned multi-task model is officially published and available on the Hugging Face Model Hub. You can load it directly into your pipeline using the Hugging Face transformers library, or reference the PyTorch snippet provided in the Usage section below to use the local checkpoint (models/pr_classifier.pt).

Live Demo

Web App

Open Hugging Face Space →

n8n Workflow

Download Workflow JSON →

Features

🧠 Model & Inference

Multi-task DistilBERT — Shared DistilBertModel backbone with sentiment and GitHub heads, mean-pooled over the attention mask (src/models/model_arch.py).
Multi-label GitHub classification — Sigmoid threshold at 0.5 over five labels; falls back to unclassified when no label fires (src/api/backend.py).
Sentiment classification — Softmax over Negative, Neutral, Positive with confidence scores.
Structured text input — Title and body normalized as [TITLE]: … [BODY]: … with markdown/code-fence cleaning and body truncation to 2000 words before tokenization.
Priority engine — Hybrid rule-based compute_priority() merges label severity, sentiment, and confidences into priority_label and priority_score, appended to final_labels.

⚡ API

POST /predict — Classify from manual title and body JSON.
POST /predict-url — Fetch a public GitHub issue/PR via REST, then classify and return GitHub metadata.
GET /health — Liveness probe with device and status fields.
GitHub automation API — POST /api/fetch-issue, /api/apply-labels, /api/create-issue with X-GitHub-Token for private repos and write-back.

🎨 Web UI

Dual input flows — Flow A: GitHub URL; Flow B: manual title/body (and optional repo for issue creation).
Token-aware private repos — PAT stored in sessionStorage, routed through /fetch-issue when present.
Result dashboard — Label chips, confidence bars, sentiment, priority, existing GitHub labels, and session history (last 25 analyses).
One-click write-back — Apply predicted labels to an existing item or create a new issue when a token and repo are provided.

🔗 Automation

n8n webhook pipeline — Import automation/PR Classifier n8n workflow.json; trigger path POST /webhook/github-pr.
Label inventory sync — Workflow compares final_labels to repo labels, creates missing labels, then applies the set via GitHub REST.
Headless /predict contract — Same JSON body as the UI manual path for event-driven triage.

🐳 DevOps

Docker image — Python 3.11 slim, dual-process startup (deployment/Dockerfile, deployment/start.sh).
Hugging Face Spaces — Documented deployment at isyedsufyan/GitPR-Analyzer with public port 7860.
CORS-enabled FastAPI — Permissive middleware for browser and workflow clients during development.

Repository Structure

Path	Type	Description
`automation/`	Directory	Exported n8n workflow JSON (`PR Classifier n8n workflow.json`) for webhook-driven classification and GitHub label write-back
`data/`	Directory	Training artifacts; `cleaned_github_data.csv` (65,055 rows, 5 multi-label targets) produced in the notebook
`deployment/`	Directory	Production container: `Dockerfile`, `start.sh`, `requirements.txt`, `.dockerignore`
`documentation/`	Directory	Architecture, API, frontend, deployment, training, and n8n guides (see Documentation Deep-Dive)
`models/`	Directory	Serialized checkpoint `pr_classifier.pt` loaded by FastAPI at startup
`notebooks/`	Directory	`PR_CLassifier.ipynb` — data prep, multi-task training, validation metrics, checkpoint export
`src/`	Directory	Application source: Flask UI, FastAPI backend, model definition, templates, static assets
`src/api/`	Directory	`backend.py` — FastAPI app, inference, GitHub `GitHubManager`, priority engine
`src/models/`	Directory	`model_arch.py` — `MultiTaskDistilBERT` architecture mirrored from the notebook
`src/templates/`	Directory	`index.html` — Tailwind/vanilla JS SPA served by Flask
`src/static/`	Directory	Static assets (e.g. `favicon.svg`)
`.gitattributes`	File	Git attributes (e.g. LFS hints for large binaries)
`.gitignore`	File	Ignores virtualenvs, `.env`, caches, and local secrets
`LICENSE`	File	MIT License (Copyright © 2026 Syed Muhammad Sufyan)

Architecture & Workflow

flowchart LR
    A[Data Collection\nHF github-issues-dataset\n+ tweet_eval sentiment] --> B[Preprocessing\nNotebook label_map\n→ cleaned_github_data.csv]
    B --> C[DistilBERT Fine-tuning\nMultiTaskDistilBERT\n4 epochs]
    C --> D[Model Packaging\nmodels/pr_classifier.pt]
    D --> E[Dual Deployment\nDocker / HF Spaces\nFlask :7860 + FastAPI :8000]
    E --> F[FastAPI API\n/predict · /predict-url\n/api/*]
    F --> G[n8n Pipeline\nWebhook → /predict → GitHub labels]
    E --> H[Flask Web UI\n/analyze → proxy FastAPI]
    H --> F

Data collection & preprocessing — The notebook loads sharjeelyunus/github-issues-dataset, maps raw labels into five binary targets via substring rules, filters rows with at least one label, and exports data/cleaned_github_data.csv. Sentiment supervision uses cardiffnlp/tweet_eval in the training loop.

Fine-tuning & packaging — MultiTaskDistilBERT is trained for four epochs with paired GitHub and sentiment batches, AdamW (lr=2e-5), mixed precision, and checkpoint selection by validation macro F1. The best weights are saved to models/pr_classifier.pt.

Dual deployment — deployment/start.sh launches uvicorn for FastAPI and gunicorn for Flask in one container. The Flask app proxies to API_BASE (default http://localhost:8000).

API & automation — Interactive users hit Flask (/analyze, /fetch-issue, …), which forwards to FastAPI. n8n calls POST /predict directly, then uses the GitHub REST API for label creation and application.

Note

There is no /batch endpoint in the current codebase; inference is per-request synchronous HTTP.

Documentation Deep-Dive

📄 documentation/app/1_System_Architecture.md

Describes the three-tier stack: vanilla JS UI, Flask presentation layer, FastAPI inference service, and n8n automation.
Documents Flow A (GitHub URL), Flow B (manual text), and Flow C (webhook automation).
Includes trust-boundary table for browser tokens, Flask→FastAPI proxying, and GitHub API access.
Explains why Flask and FastAPI are split (reusable model service for UI and n8n).
Links code anchors for API_BASE, model load path, and startup configuration.

View →

📄 documentation/app/2_Backend_API_Reference.md

Full endpoint index for Flask and FastAPI layers with auth requirements.
Request/response tables for /predict, /predict-url, and automation routes.
Documents PredictionResponse fields including final_labels and priority_label.
n8n contract: webhook path, normalization code, and host.docker.internal:8000 caveat.
GitHub token forwarding via X-GitHub-Token header.

View →

📄 documentation/app/3_Frontend_Integration.md

Maps UI regions (token card, flow panels, result card, history) to DOM IDs in index.html.
Explains sessionStorage token persistence and Flow A vs Flow B routing.
Documents client-side validation rules and loading/error state behavior.
Describes conditional Apply Labels and Create Issue automation buttons.
Session history capped at 25 entries in analysis_history.

View →

📄 documentation/app/4_Deployment_Environment.md

Docker build steps: Python 3.11 slim, git-lfs, EXPOSE 7860, start.sh entrypoint.
Two-process runtime table: uvicorn :8000, gunicorn :7860 or $PORT.
deployment/requirements.txt dependency roles and .dockerignore model artifact rules.
Environment variable contract aligned with src/.env.example.
Hugging Face Spaces operational notes and /health probing.

View →

📄 documentation/automation/n8n Automation Workflow.md

End-to-end n8n node inventory (15 nodes) including label-existence branch and create-label path.
Webhook metadata: POST, path github-pr, active in export.
Normalization handles root or wrapped body payloads for issues and pull requests.
Consumes final_labels (not only topical labels) for priority tag write-back.
Production hardening notes: tunnel URLs, token scopes, Docker networking, troubleshooting matrix.

View →

📄 documentation/notebook/Pre-Training Documentation.md

Goal: five multi-label targets from Hugging Face GitHub issues dataset.
label_map substring rules for bug, enhancement, documentation, test, request.
Filtering keeps rows with at least one positive label; output schema with id.
EDA: label frequencies, correlation heatmap, text length stats (median 170 words).
Artifact: data/cleaned_github_data.csv with 65,055 rows.

View →

📄 documentation/notebook/Model Training Documentation.md

Dual-head architecture with masked mean pooling and task routing flag.
Loss design: CrossEntropyLoss + BCEWithLogitsLoss with pos_weight.
Training loop: zipped GitHub loader + cycled Twitter/sentiment loader, AMP, AdamW.
Four-epoch validation trace and checkpoint selection by sum of macro F1 scores.
Final validation metrics and per-label breakdown; handoff to src/api/backend.py.

View →

📄 documentation/notebook/Model Training Formulas.md

Reference sheet for pooling, projection, dropout, softmax, sigmoid, and composite loss.
Documents inference rules: argmax sentiment, 0.5 multi-label threshold.
Evaluation formulas: precision, recall, F1 variants, accuracy, AdamW update.
Notes that warmup scheduler is imported but not stepped in executed cells.
Suitable for thesis-style methods sections.

View →

Getting Started

Prerequisites

Requirement	Version / notes
Python	3.11 (matches `deployment/Dockerfile`)
pip + venv	Recommended for local installs
Node.js	Not required (vanilla JS + CDN Tailwind in `index.html`)
Docker	Optional; 20.10+ recommended for container workflows
GPU	Optional; CUDA used automatically when available (`backend.py`)
Git LFS	Recommended if cloning large `models/pr_classifier.pt`

Clone & Install

git clone https://github.com/syedsufyan-coder/Git-PR-Analyzer.git
cd Git-PR-Analyzer
python -m venv .venv
# Windows
.venv\Scripts\activate
# macOS / Linux
source .venv/bin/activate
pip install -r deployment/requirements.txt

Run commands from the repository root so models/pr_classifier.pt resolves correctly.

Environment Variables

Copy src/.env.example to src/.env (or export variables in your shell):

Variable	Required	Description	Example Value
`FLASK_SECRET_KEY`	Recommended	Flask session/cookie signing	`your-random-secret`
`API_BASE`	Required (Flask proxy)	FastAPI base URL used by Flask	`http://localhost:8000`
`PORT`	Set by host only	Public port for gunicorn in containers	`7860` (Hugging Face Spaces)
GitHub token	Optional / required for write-back	PAT or GitHub App token; UI uses `sessionStorage`, n8n uses credentials	`ghp_…`

Run Locally

Terminal 1 — FastAPI (inference + GitHub API):

uvicorn src.api.backend:app --reload --host 0.0.0.0 --port 8000

Terminal 2 — Flask (web UI):

# From repo root, with API_BASE pointing at FastAPI
set API_BASE=http://localhost:8000   # Windows CMD
# export API_BASE=http://localhost:8000   # macOS/Linux
python src/app.py

Open http://localhost:5000 (Flask debug server). The UI proxies analysis requests to FastAPI on port 8000.

Docker

docker build -f deployment/Dockerfile -t git-pr-classifier .
docker run --rm -p 7860:7860 -p 8000:8000 \
  -e FLASK_SECRET_KEY=change-me \
  -e API_BASE=http://localhost:8000 \
  git-pr-classifier

Note

Layout note: deployment/start.sh expects api.backend:app and app:app at the container root (as deployed on Hugging Face Spaces). Building from this repository may require mirroring src/api/backend.py → api/backend.py and src/app.py → app.py at the image root, or running the local uvicorn/gunicorn commands instead. See 4_Deployment_Environment.md.

Usage

REST API

Health check

curl -s http://localhost:8000/health

{"status":"ok","device":"cpu","token_loaded":false}

Predict from title + body

curl -s -X POST http://localhost:8000/predict \
  -H "Content-Type: application/json" \
  -d "{\"title\": \"Fix failing CI pipeline\", \"body\": \"Tests fail on main after the merge.\"}"

Predict from public GitHub URL

curl -s -X POST http://localhost:8000/predict-url \
  -H "Content-Type: application/json" \
  -d "{\"url\": \"https://github.com/owner/repo/issues/1\"}"

Fetch private issue/PR and classify (token required)

curl -s -X POST http://localhost:8000/api/fetch-issue \
  -H "Content-Type: application/json" \
  -H "X-GitHub-Token: YOUR_GITHUB_TOKEN" \
  -d "{\"repo\": \"owner/repo\", \"number\": 42, \"kind\": \"auto\"}"

Apply labels to an existing issue/PR

curl -s -X POST http://localhost:8000/api/apply-labels \
  -H "Content-Type: application/json" \
  -H "X-GitHub-Token: YOUR_GITHUB_TOKEN" \
  -d "{\"repo\": \"owner/repo\", \"number\": 42, \"labels\": [\"bug\", \"critical\"]}"

Create a new issue with labels

curl -s -X POST http://localhost:8000/api/create-issue \
  -H "Content-Type: application/json" \
  -H "X-GitHub-Token: YOUR_GITHUB_TOKEN" \
  -d "{\"repo\": \"owner/repo\", \"title\": \"New bug report\", \"body\": \"Details…\", \"labels\": [\"bug\", \"high-priority\"]}"

Python client snippet

import requests

API = "http://localhost:8000"

payload = {
    "title": "Add OAuth2 login",
    "body": "We need Google and GitHub sign-in for the dashboard.",
}

resp = requests.post(f"{API}/predict", json=payload, timeout=30)
resp.raise_for_status()
result = resp.json()

print("Labels:", result["github_labels"])
print("Sentiment:", result["sentiment"], result["sentiment_confidence"])
print("Priority:", result["priority_label"], result["priority_score"])
print("Apply to GitHub:", result["final_labels"])

To load the same architecture locally with PyTorch (as the API does):

import torch
from transformers import DistilBertTokenizer
from src.models.model_arch import MultiTaskDistilBERT

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = MultiTaskDistilBERT(num_sentiment=3, num_github=5)
model.load_state_dict(torch.load("models/pr_classifier.pt", map_location=device))
model.eval()
tokenizer = DistilBertTokenizer.from_pretrained("distilbert-base-uncased")
# Tokenize and call model(..., task="sentiment") or task="github" as in src/api/backend.py

Web UI walkthrough

Open the app (local :5000 or the Hugging Face Space).
Choose Flow A (GitHub URL) or Flow B (manual title/body); optionally paste a GitHub token for private repos or write-back.
Click Analyze — view predicted labels, sentiment, priority, confidences, and use Apply Labels or Create Issue when a token is set.

n8n Integration

Import automation/PR Classifier n8n workflow.json into n8n and activate the workflow.
Point your GitHub webhook to the n8n URL for path github-pr (POST).
Ensure the Predict Request node reaches your FastAPI service (default export: http://host.docker.internal:8000/predict — change for production).
Configure HTTP Header Auth credentials for GitHub on Existing Labels, Create Labels, and Apply Labels nodes.
Execution flow: Webhook → Normalize → Predict Request → Define Logic → … → Apply Labels.

Example I/O

Request

{
  "title": "Fix failing CI pipeline",
  "body": "The build fails when tests run on the main branch."
}

Response

{
  "github_labels": ["bug"],
  "github_confidences": {"bug": 0.9412},
  "sentiment": "Negative",
  "sentiment_confidence": 0.8734,
  "priority_score": 0.9421,
  "priority_label": "critical",
  "final_labels": ["bug", "critical"],
  "structured_text": "[TITLE]: Fix failing CI pipeline [BODY]: The build fails when tests run on the main branch."
}

Model Information

Property	Value
Base model	`distilbert-base-uncased`
Task	Multi-label GitHub classification + 3-way sentiment + rule-based priority
GitHub training data	`sharjeelyunus/github-issues-dataset` — 65,055 cleaned rows in `data/cleaned_github_data.csv`
Sentiment training data	`cardiffnlp/tweet_eval` — auxiliary corpus in `notebooks/PR_CLassifier.ipynb`
Labels	GitHub: `bug`, `enhancement`, `documentation`, `test`, `request` (+ `unclassified` fallback); Sentiment: `Negative`, `Neutral`, `Positive`; Priority: `critical`, `high-priority`, `normal-priority`, `low-priority`
Max sequence length	256 (tokenizer `max_length` in `run_prediction`)
Training epochs	4
Optimizer	AdamW, learning rate 2e-5
Checkpoint	`models/pr_classifier.pt`
Hugging Face model card	View the Model on Hugging Face

📊 Validation performance (from notebook / documentation)

Task	Metric	Value
Sentiment	Accuracy	0.71
Sentiment	Macro F1	0.69
GitHub labels	Micro F1	0.75
GitHub labels	Macro F1	0.67
GitHub labels	Samples F1	0.77

Epoch	GitHub macro F1	Sentiment macro F1
1	0.6219	0.7070
2	0.6574	0.7103
3	0.6849 (best checkpoint)	0.7058
4	0.6722	0.6927

Metrics are validation-split results from notebooks/PR_CLassifier.ipynb as documented in Model Training Documentation.md. Run a held-out test evaluation separately for publishable benchmarks.

Star History

View interactive chart on star-history.com →

Roadmap

From code comments and documentation (planned / suggested enhancements):

Refactor duplicated parse_github_url in Flask into a shared helper (src/app.py note).
Add webhook signature validation for n8n (documentation/automation/n8n Automation Workflow.md).
Add explicit error branch for unsupported n8n Normalize payloads (production hardening).
Add CI workflow for lint/test and container smoke checks (no .github/workflows/ yet).

Contributing

We'd love your help making this project better. Whether you fix a bug, improve docs, or extend the model — every contribution counts. 🙌

How to contribute

Fork Git-PR-Analyzer and create a branch: feat/short-description or fix/short-description.
Match conventions — Flask for UI proxying, FastAPI for inference, Pydantic schemas, and MultiTaskDistilBERT in src/models/model_arch.py.
Install & run locally — pip install -r deployment/requirements.txt, then start both FastAPI and Flask before opening a PR.
Open a PR with a clear description and link any related issue.

PR checklist

Changes tested locally (curl /health, /predict, and UI flows)
API changes reflected in documentation/app/2_Backend_API_Reference.md
No secrets committed (.env, tokens, PATs)
Model or notebook updates documented if the prediction contract changed
README updated if user-facing behavior changed

Not sure where to start? Open a GitHub Issue and we'll help you find a good first task.

Credits & Acknowledgements

Maintainer: syedsufyan-coder (Syed Muhammad Sufyan)

Contributors: Syed Muhammad Sufyan

Libraries & projects

Hugging Face Transformers — DistilBERT tokenizer and encoder
DistilBERT — Sanh et al., 2019
FastAPI — Inference and GitHub automation API
Flask — Web UI and request proxy
n8n — Workflow automation
Docker — Container packaging
Datasets: sharjeelyunus/github-issues-dataset · cardiffnlp/tweet_eval

License

This project is licensed under the MIT License — see LICENSE.

You may use, modify, and distribute the code with attribution and without warranty, per the MIT terms.

Contact & Support

Channel	Link
🐛 Issues	github.com/syedsufyan-coder/Git-PR-Analyzer/issues
🌐 Live demo	Hugging Face Space — GitPR-Analyzer
💡 Discussions	Join our GitHub Discussions to ask questions, share ideas, and get support

If this project helps your triage workflow, star the repo — it helps others discover the work. ⭐

Name		Name	Last commit message	Last commit date
Latest commit History 27 Commits
automation		automation
data		data
deployment		deployment
documentation		documentation
models		models
notebooks		notebooks
src		src
.gitattributes		.gitattributes
.gitignore		.gitignore
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md

Folders and files

Latest commit

History

Repository files navigation

GitHub PR/Issue Classifier

📑 Table of Contents

About the Project

Why this project?

Quick Links

Live Demo

Web App

n8n Workflow

Features

🧠 Model & Inference

⚡ API

🎨 Web UI

🔗 Automation

🐳 DevOps

Repository Structure

Architecture & Workflow

Documentation Deep-Dive

Getting Started

Prerequisites

Clone & Install

Environment Variables

Run Locally

Docker

Usage

REST API

Python client snippet

Web UI walkthrough

n8n Integration

Example I/O

Model Information

📊 Validation performance (from notebook / documentation)

Star History

Roadmap

Contributing

How to contribute

PR checklist

Credits & Acknowledgements

License

Contact & Support

About

Topics

Resources

License

Code of conduct

Contributing

Uh oh!

Stars

Watchers

Forks

Contributors

Uh oh!

Languages