Screenshot OCR with Coordinate Extraction

Production-ready OCR model for building camera translation apps
Extract text with precise pixel coordinates • GPU-accelerated • Multi-language • Deploy to Replicate

Quick Start

🚀 Use the Deployed Model

import replicate
import json

output = replicate.run(
    "zsxkib/easyocr",
    input={
        "image": "https://example.com/your-image.jpg",
        "include_bboxes": True,
        "preprocessing": True,
        "min_confidence": 0.25
    }
)

# Extract coordinates for overlay translation
metadata = json.loads(output["metadata"])
for region in metadata["regions"]:
    text = region["text"]
    x1, y1, x2, y2 = region["x1"], region["y1"], region["x2"], region["y2"]
    # Now you can overlay translations at these exact coordinates

🔧 Deploy Your Own

git clone https://github.com/zsxkib/EasyOCR.git
cd EasyOCR
cog predict -i image=@your-image.jpg
cog push r8.im/your-username/easyocr

Why This OCR Model?

Perfect for camera translation apps like Google Translate's camera feature:

1. 📍 Precise Coordinates: Get exact pixel positions for text overlay
2. 🎯 Layout Preservation: Maintains original text positioning and reading order
3. 🌍 Multi-Language: 80+ languages with Unicode support
4. ⚡ GPU Optimized: Fast inference with automatic CPU fallback
5. 📱 Mobile-Ready: Handles low-resolution screenshots and mobile captures

Input & Output

Input Parameters

{
    "image": "Screenshot or image file",
    "languages": "Comma-separated codes (e.g., 'en,es,fr')",
    "min_confidence": 0.25,          # Filter low-confidence detections
    "preprocessing": True,           # Enhanced image processing
    "include_bboxes": True,         # Essential for coordinate overlay
    "include_polygons": False       # Optional detailed shapes
}

Output Structure

{
    "markdown": "path/to/extracted_text.md",
    "metadata": {
        "total_regions": 9,
        "avg_confidence": 0.944,
        "languages_used": ["en", "es"],
        "regions": [
            {
                "text": "Hello World",
                "confidence": 0.99,
                "x1": 100, "y1": 50,    # Top-left coordinates
                "x2": 200, "y2": 80     # Bottom-right coordinates
            }
            // ... more text regions
        ]
    }
}

Camera Translation App Architecture

📱 Camera Input
    ↓
🔍 Screenshot OCR (this model)
    ↓
📍 Text + Coordinates Extracted
    ↓
🌐 Translation API (Google/Azure/etc.)
    ↓
🎨 Overlay Translated Text
    ↓
📱 Augmented Camera View

This model handles the crucial first step: extracting text with pixel-perfect coordinates so you can overlay translations in the exact same positions.

Examples

Note: render_from_coordinates.py is a local helper for visualization; it is not included in the Replicate container or API.

Quick usage:

• Save OCR regions to a JSON file with a top-level key "regions", each item containing text, x1,y1,x2,y2
• Optionally pass a background image to draw on

Example:

python render_from_coordinates.py ocr_output.json --background path/to/image.png

Text Detection Results

# Input: Screenshot of a menu
# Output: Structured regions ready for translation overlay

regions = [
    {"text": "Pasta Carbonara", "x1": 120, "y1": 200, "x2": 280, "y2": 230},
    {"text": "€12.50", "x1": 350, "y1": 200, "x2": 400, "y2": 230},
    {"text": "Fresh ingredients", "x1": 120, "y1": 235, "x2": 260, "y2": 255}
]

Integration Example

def translate_camera_view(image_path):
    # Step 1: Extract text with coordinates
    ocr_result = replicate.run("zsxkib/easyocr", input={"image": image_path})
    regions = json.loads(ocr_result["metadata"])["regions"]
    
    # Step 2: Translate each text region
    for region in regions:
        original_text = region["text"]
        translated = translate_api(original_text, target_lang="es")
        
        # Step 3: Overlay translation at exact coordinates
        overlay_text(image, translated, region["x1"], region["y1"], 
                    region["x2"], region["y2"])
    
    return augmented_image

Technical Details

• Framework: Cog + EasyOCR + PyTorch
• GPU: CUDA 12.1 with automatic CPU fallback
• Languages: 80+ supported with Unicode text
• Preprocessing: DPI upscaling, CLAHE enhancement, denoising
• Performance: Sub-second inference on typical screenshots
• Memory: Efficient cleanup prevents GPU memory leaks

Repository Structure

├── predict.py                   # Core OCR predictor
├── cog.yaml                    # Deployment configuration
├── requirements.txt            # Minimal dependencies
├── render_from_coordinates.py  # Local helper for visualization (not in Replicate)
└── README.md                   # This file

Contributing

This is designed to be the minimal, focused OCR backbone for camera translation apps.

• Keep it simple and readable
• No unnecessary features or bloated code
• Focus on coordinate accuracy and performance
• World-class open source standards

License

MIT License — see LICENSE

---

Ready to build the next Google Translate Camera? This OCR foundation has you covered. ✨

---

🐦 Follow @zsxkib on X
💻 Check out more projects @zsxkib on GitHub