An open-source framework to identify, deduplicate, and track intent-level similarity in AI prompts and datasets without relying on large AI models.
The Intent Identity Framework is a deterministic, explainable system that detects duplicate prompts and datasets by understanding their intent, not just matching words.
- Developers create the same prompts with different wording
- Datasets contain hidden duplicates
- Existing solutions either fail (exact matching) or are expensive (AI models)
A 5-layer deterministic pipeline that converts prompts into canonical representations and compares them using rule-based logic.
| Feature | Description |
|---|---|
| 🚀 Zero AI Dependencies | Core functionality uses no ML models |
| 📊 Deterministic | Same input → Same output (always) |
| 🔍 Explainable | Every score includes detailed reasoning |
| 🎨 Community-Driven | Extend vocabularies without code changes |
| ⚡ Fast | Process 100+ prompts/second |
| 🎯 Accurate | >85% accuracy on test datasets |
| 🗑️ Auto-Clean | Remove duplicates automatically |
git clone https://github.com/yourusername/DuplicateDetector.git
cd DuplicateDetectorpython3 demo.pypython3 check_duplicates.pyNo installation required! Works with Python 3.9+ standard library.
Clean your prompt collections:
from iif import SimilarityScorer
scorer = SimilarityScorer()
result = scorer.compare(prompt1, prompt2)
print(f"Similarity: {result['similarity_score']}")Find and remove duplicates in training datasets:
python3 check_duplicates.py
# Choose option 2 to remove duplicatesDetect intent-level copying in academic/research prompts:
# Compare submissions
for i, submission in enumerate(submissions):
for j in range(i+1, len(submissions)):
similarity = scorer.compare(submission, submissions[j])
if similarity['similarity_score'] > 0.85:
print(f"Potential plagiarism: {i} and {j}")Track how prompts evolve over time:
# Compare versions
v1 = "You are a helpful assistant..."
v2 = "Act as a helpful AI assistant..."
result = scorer.compare(v1, v2)
print(f"Version similarity: {result['similarity_score']}")Extend the framework for your domain:
// vocabularies/intent_tags.json
{
"medical_advice": {
"description": "Provides medical guidance",
"rules": {
"required": ["goal"],
"keywords": ["medical", "health", "diagnosis"]
}
}
}Process large datasets efficiently:
# See examples/batch_processing.py
python3 examples/batch_processing.pyInput Text
↓
┌─────────────────────────────────┐
│ Layer 1: Canonicalization │ → Normalize & structure
├─────────────────────────────────┤
│ Layer 2: Fingerprinting │ → Extract features
├─────────────────────────────────┤
│ Layer 3: Intent Tagging │ → Apply rule-based tags
├─────────────────────────────────┤
│ Layer 4: Similarity Scoring │ → Compute explainable scores
├─────────────────────────────────┤
│ Layer 5: Embedding Filter (opt) │ → Semantic similarity
└─────────────────────────────────┘
↓
Similarity Report (JSON)
See docs/architecture.md for details.
- ✅ Clean prompt libraries
- ✅ Detect duplicate API calls
- ✅ Version control for prompts
- ✅ Quality assurance for datasets
- ✅ Academic plagiarism detection
- ✅ Dataset deduplication
- ✅ Bias detection in prompts
- ✅ Reproducible similarity metrics
- ✅ AI governance & auditing
- ✅ Prompt management systems
- ✅ Data quality pipelines
- ✅ Compliance checking
- ✅ Assignment similarity checking
- ✅ Student work verification
- ✅ Teaching dataset curation
- Dataset: 386 conversation entries
- Duplicates Found: 1 pair (0.08% duplication rate)
- Accuracy: Correctly identified duplicate with different formatting
- Speed: 1,225 comparisons in ~2 seconds
- Similar prompts: 98.3% match ✓
- Different prompts: 61.2% (correctly low) ✓
- Batch detection: 100% accuracy ✓
# No installation needed!
python3 demo.py
python3 examples/basic_usage.pypython3 -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
pip install -e .pip install sentence-transformers| Document | Description |
|---|---|
| README.md | This file - Quick start guide |
| PRP_Report.md | Problem-Research-Proposal report |
| docs/architecture.md | Technical architecture |
| CONTRIBUTING.md | How to contribute |
| examples/ | Usage examples |
from iif import SimilarityScorer
scorer = SimilarityScorer()
prompt1 = "You are Sheldon Cooper. Always use Bazinga!"
prompt2 = "Act as Sheldon Cooper. Use the catchphrase Bazinga!"
result = scorer.compare(prompt1, prompt2)
print(f"Similarity: {result['similarity_score']}") # 0.983
print(f"Verdict: {result['verdict']}") # high_similarity
print(f"Matched Tags: {result['explanation']['matched_tags']}")python3 examples/batch_processing.pypython3 check_duplicates.py
# Choose option 2 to remove duplicates
# Creates: sheldon_cleaned.jsonpython3 examples/custom_vocabulary.pyWe welcome contributions! Here's how you can help:
- 📝 Add new intent tags to
vocabularies/intent_tags.json - 🎨 Add structural patterns to
vocabularies/patterns.json - 📖 Improve documentation
- 🐛 Report bugs
- ✨ Add new examples
- 🧪 Write tests
- 🔧 Improve core layers
- 🚀 Build integrations (IDE plugins, web API)
- 📊 Add visualization tools
- 🔬 Research improvements
See CONTRIBUTING.md for detailed guidelines.
- ✅ Core 5-layer architecture
- ✅ Deterministic similarity scoring
- ✅ Explainable results
- ✅ Batch processing
- ✅ Duplicate removal
- Web API server
- CLI tool
- Performance benchmarks
- More intent tags
- Version lineage tracking
- Bias fingerprint detection
- Prompt evolution graphs
- IDE integrations (VS Code, PyCharm)
- GitHub Action integration
MIT License - see LICENSE for details.
TL;DR: Free to use, modify, and distribute. No restrictions.
Current duplicate detection approaches:
- Exact matching (hashing) → Fails for reworded content ❌
- Large AI models (embeddings) → Expensive, opaque, non-deterministic ❌
IIF provides a third way:
- ✅ Deterministic (reproducible results)
- ✅ Explainable (clear reasoning)
- ✅ Fast (no model loading)
- ✅ Free (no API costs)
- ✅ Open-source (community-driven)
- 🐛 Issues: GitHub Issues
- 💬 Discussions: GitHub Discussions
- 📧 Email: your.email@example.com
- 🌐 Website: Coming soon!
- Inspired by the need for transparent AI tools
- Built with Python standard library
- Community-driven vocabulary system
- Open-source from day one
- Language: Python 3.9+
- Dependencies: Zero (core), Optional (embeddings)
- Lines of Code: ~2,000
- Test Coverage: Core layers tested
- Documentation: Comprehensive
- License: MIT
git clone https://github.com/yourusername/DuplicateDetector.git
cd DuplicateDetector
python3 demo.pyStar ⭐ this repo if you find it useful!
Built with ❤️ for the open-source community