ForgeEye

Industrial Vision-Language Model toolkit — evaluate open VLMs on factory-style tasks, then deploy with OpenVINO (primary) or ONNX Runtime where it fits.

CLI: forgeye · Repo: baldellimtt/ForgeEye

What this is

ForgeEye helps you answer: can a VLM replace or augment classical CV on an industrial line?

Goal	Use
Open-vocab detection, defect QA, counting, reasoned OCR	Qwen3-VL-2B / 4B (primary)
Fast per-frame boxes / localized OCR overlays	Florence-2 (recommended for OCR boxes)
Production on Intel CPU / GPU / NPU	OpenVINO IR + GenAI
Cross-vendor / NVIDIA / DirectML	ONNX Runtime (practical for Florence; not ideal for full Qwen)

Start simple
  Camera → Qwen3-VL → boxes / PASS-FAIL / QA

Localized OCR / line-rate boxes
  Camera → Florence-2 (OCR_WITH_REGION / OD) → overlay
         → optional Qwen inspect on ROI crops

Tested on

Validated on a Windows laptop development machine:

Component	Detail
OS	Windows 10/11
GPU	NVIDIA GeForce RTX 3050 4GB Laptop GPU + Intel UHD Graphics
Runtime paths exercised	Florence-2 (transformers 4.49 + CUDA fp16), OpenVINO GenAI INT4 for Qwen3-VL-2B (--device GPU = Intel plugin)
Example timings (school-bus, RTX 3050 + Intel iGPU)	Florence detect via forgeye serve ≈ 0.5–1 s warm; Florence CLI detect ≈ 3–4 s infer; Qwen OV detect ≈ 15–17 s (TTFT ≈ 10 s @ ~10 tok/s on iGPU)

On 4 GB VRAM, full-precision Qwen via PyTorch may offload to CPU/disk. Prefer OpenVINO INT4 for usable laptop inference. Florence needs CUDA PyTorch for speed on NVIDIA.

Default device is GPU for OpenVINO / CLI (--device GPU). Use --device CPU or NPU when needed.

Resident server: forgeye serve loads the model once (warmup), then each HTTP request pays only inference — the main production path for the Python demo before a C# OpenVINO GenAI deploy.

Installation

Prerequisites

Requirement	Notes
Python 3.10+	uv can download a matching interpreter
uv	Package manager for this repo
Git	Clone the repository
Disk	~5–15 GB for env + first model download
RAM / VRAM	4 GB laptop GPU works with OpenVINO INT4; ≥8 GB VRAM is more comfortable for PyTorch lab runs

Install uv

# Windows (PowerShell)
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

# macOS / Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

uv --version

Clone and sync

git clone https://github.com/baldellimtt/ForgeEye.git
cd ForgeEye

# Lab (Qwen + Florence via PyTorch / transformers)
uv sync --extra lab

# Lab + OpenVINO export / GenAI runtime
uv sync --extra lab --extra openvino

# Resident HTTP server (FastAPI)
uv sync --extra lab --extra openvino --extra serve

# Everything
uv sync --extra all

Goal	Command
CLI only	uv sync
Qwen / Florence	uv sync --extra lab
+ OpenVINO	uv sync --extra lab --extra openvino
+ HTTP server	uv sync --extra lab --extra openvino --extra serve
All extras	uv sync --extra all

Verify:

uv run forgeye models
uv run forgeye --help

First inference downloads weights into the Hugging Face cache (%USERPROFILE%\.cache\huggingface on Windows). Optional login for higher HF rate limits:

uv run hf auth login

NVIDIA CUDA PyTorch (Florence + Qwen lab)

ForgeEye pins torch/torchvision cu124 in pyproject.toml so uv sync --extra lab prefers the CUDA wheels (NVIDIA).

uv sync --extra lab
# If torch is still CPU-only (rare):
uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124 --reinstall

Backend	Device
--backend florence	NVIDIA CUDA (fp16) when torch CUDA is installed
--backend transformers (Qwen)	NVIDIA via device_map=auto
--backend openvino	OpenVINO device (GPU = Intel plugin by default, not CUDA)

On 4 GB VRAM, prefer Florence-base / Qwen OpenVINO INT4; full Qwen via transformers may offload layers to CPU.

Command cheat sheet

All commands assume you are in the repo root and have run uv sync with the needed extras. Replace path/to/image.jpg with your file (e.g. school-bus.jpg).

Discovery

uv run forgeye models
uv run forgeye --help
uv run forgeye infer --help
uv run forgeye export --help
uv run forgeye hybrid --help
uv run forgeye benchmark --help
uv run forgeye serve --help

Resident server (recommended for repeated inference)

Loads the model once, then serves /infer over HTTP. Eliminates the 6–35 s per-process load of the CLI.

# Florence (CUDA) — fast detect / OCR overlays
uv run forgeye serve --backend florence --model florence-base-ft --port 8080

# Qwen OpenVINO INT4 (Intel GPU plugin)
uv run forgeye serve --backend openvino --model-dir models/ov/qwen3-vl-2b-int4 --device GPU --port 8080

# Health
curl.exe -s http://127.0.0.1:8080/health

# Detect
curl.exe -s -F "image=@path/to/image.jpg" -F "task=detect" -F "classes=bus" \
  http://127.0.0.1:8080/infer

# OCR
curl.exe -s -F "image=@path/to/image.jpg" -F "task=ocr_region" \
  http://127.0.0.1:8080/infer

Warmup runs at startup (--no-warmup to skip). Override backend/model per request with form fields backend, model, model_dir, device.

Qwen — transformers (lab)

# Object / defect detection (JSON boxes)
uv run forgeye infer --backend transformers --model qwen3-vl-2b \
  --image path/to/image.jpg --task detect --classes "scratch,dent,missing_screw"

# Pass / fail inspection
uv run forgeye infer --backend transformers --model qwen3-vl-2b \
  --image path/to/image.jpg --task inspect \
  --criteria "scratches, dents, missing fasteners" \
  --golden-desc "clean undamaged assembly"

# Counting
uv run forgeye infer --backend transformers --model qwen3-vl-2b \
  --image path/to/image.jpg --task count --classes "screw"

# OCR (transcription; boxes best-effort — prefer Florence for overlays)
uv run forgeye infer --backend transformers --model qwen3-vl-2b \
  --image path/to/image.jpg --task ocr_region

# Grounded QA
uv run forgeye infer --backend transformers --model qwen3-vl-2b \
  --image path/to/image.jpg --task grounded_qa \
  --question "Where is the barcode and is it readable?"

# Larger Qwen
uv run forgeye infer --backend transformers --model qwen3-vl-4b \
  --image path/to/image.jpg --task detect --classes "part,connector"

Florence-2 — detection / localized OCR

# Localized OCR with boxes (best overlay)
uv run forgeye infer --backend florence --model florence-large-ft \
  --image path/to/image.jpg --task ocr_region

# Smaller / faster Florence
uv run forgeye infer --backend florence --model florence-base-ft \
  --image path/to/image.jpg --task ocr_region

# Object detection
uv run forgeye infer --backend florence --model florence-large-ft \
  --image path/to/image.jpg --task detect

# Open-vocab grounding
uv run forgeye infer --backend florence --model florence-large-ft \
  --image path/to/image.jpg --task detect --classes "scratch, dent, screw"

OpenVINO — export + infer (quantized, GPU default)

# One-time export Qwen3-VL-2B → INT4 IR
uv run forgeye export --model qwen3-vl-2b --weight-format int4 \
  --out models/ov/qwen3-vl-2b-int4

# Other exports
uv run forgeye export --model qwen3-vl-4b --weight-format int4 --out models/ov/qwen3-vl-4b-int4
uv run forgeye export --model qwen25-vl-3b --weight-format int4 --out models/ov/qwen25-vl-3b-int4

# Florence-2 is *not* exportable via optimum-cli / `forgeye export` (custom arch).
# Use --backend florence (PyTorch). OpenVINO path = Qwen above.

# OCR / inspect / detect on INT4 (GPU is default — omit --device or set explicitly)
uv run forgeye infer --backend openvino --model-dir models/ov/qwen3-vl-2b-int4 \
  --image path/to/image.jpg --task ocr_region

uv run forgeye infer --backend openvino --model-dir models/ov/qwen3-vl-2b-int4 \
  --image path/to/image.jpg --task inspect

uv run forgeye infer --backend openvino --model-dir models/ov/qwen3-vl-2b-int4 \
  --image path/to/image.jpg --task detect --classes "bus,person,car"

# Optional: force CPU plugin
uv run forgeye infer --backend openvino --model-dir models/ov/qwen3-vl-2b-int4 \
  --image path/to/image.jpg --task detect --classes "bus" --device CPU

OpenVINO path notes (also apply in C# GenAI):

• Compiled-model cache under models/ov/.../.ovcache (faster subsequent loads)
• Input images larger than ~1.3 MP are downscaled before prefill (boxes stay 0–1000 normalized)
• Per-task max_new_tokens defaults: detect 256, inspect 320, ocr_region 1024 (override with --max-new-tokens)
• OCR stops early if the model starts repeating the same row (INT4 Qwen often re-scans dense pages)

Hybrid — Florence detect → Qwen inspect on crops

uv run forgeye hybrid --image path/to/image.jpg \
  --detector florence-large-ft \
  --reasoner-backend transformers \
  --reasoner-model qwen3-vl-2b \
  --classes "screw,connector,cable" \
  --criteria "damage, misassembly, contamination"

# Reasoner = OpenVINO INT4 (GPU default)
uv run forgeye hybrid --image path/to/image.jpg \
  --detector florence-large-ft \
  --reasoner-backend openvino \
  --model-dir models/ov/qwen3-vl-2b-int4 \
  --classes "screw,connector" \
  --criteria "damage, missing part"

Benchmark

uv run forgeye benchmark --backend openvino --model-dir models/ov/qwen3-vl-2b-int4 \
  --image path/to/image.jpg --task detect --warmup 5 --runs 20

uv run forgeye benchmark --backend florence --model florence-large-ft \
  --image path/to/image.jpg --task ocr_region --warmup 2 --runs 5

Outputs

Every infer writes:

File	Content
outputs/last_result.json	Full result (load_s, latency_s, OCR/detections, raw text; OpenVINO also reports ttft_s / tok_per_s in meta)
outputs/last_overlay.jpg	Image overlay when boxes or OCR spans exist

Timing lines at the end of the CLI (seconds):

load_s=...s
infer_s=...s
total_s=...s

Custom output paths:

uv run forgeye infer --backend openvino --model-dir models/ov/qwen3-vl-2b-int4 \
  --image path/to/image.jpg --task ocr_region \
  --out-json outputs/ocr.json --out-viz outputs/ocr_overlay.jpg

OCR notes

Backend	Boxes on overlay	Notes
Florence ocr_region	Yes (OCR_WITH_REGION quads)	Prefer florence-large-ft for accurate text; base is faster but misreads small/stylized labels. OCR uses beam search (num_beams=3); detect stays greedy.
Qwen transformers / OpenVINO	Best-effort	May omit bbox_2d or re-scan long pages; OpenVINO stops on repeated OCR rows

Architecture (short)

Florence-2 is not a chat VLM. It is a small Microsoft multi-task model (MIT, 0.23B / 0.77B) with task tokens (<OD>, <OCR_WITH_REGION>, …). Use it as a fast detector/OCR; use Qwen for open-vocab reasoning and inspection.

Lab pin: Florence remote code needs transformers 4.49.x (lab extra). Transformers 5.x produces garbage Florence generations — keep the pin.

See ARCHITECTURE.md.

Deploy: OpenVINO vs ONNX

Runtime	Best for	Avoid for
OpenVINO GenAI (forgeye export + --backend openvino)	Qwen2.5 / Qwen3-VL INT4/INT8	Florence-2
PyTorch / transformers (--backend florence)	Florence-2 OD / OCR boxes	—
ONNX Runtime	Optional Florence / classical detectors (manual)	Full generative Qwen as one .onnx

Needs OpenVINO ≥ 2026.1 for Qwen3-VL. Qwen2.5-VL usually works from 2025.3.

--weight-format	When
int4	Default edge / laptop deploy
int8	If INT4 hurts OCR / tiny defects
nf4	Intel NPU (Lunar Lake+)
fp16	Accuracy reference

ONNX sketch (not wrapped by the CLI yet):

optimum-cli export onnx --model microsoft/Florence-2-large-ft \
  --task image-to-text models/onnx/florence-large-ft

Model shortlist

Key	Model	License	Role
qwen3-vl-2b	Qwen3-VL-2B-Instruct	Apache-2.0	Primary VLM
qwen3-vl-4b	Qwen3-VL-4B-Instruct	Apache-2.0	Higher quality
qwen25-vl-3b	Qwen2.5-VL-3B-Instruct	Apache-2.0	Mature OV baseline
florence-large-ft	Florence-2-large-ft	MIT	Fast OD / localized OCR
florence-base-ft	Florence-2-base-ft	MIT	Smaller Florence
zwz-2b	ZwZ-2B	Apache-2.0	Perception boost

Project layout

src/forgeye/
  cli.py              # `forgeye` entrypoint
  server.py           # `forgeye serve` (FastAPI, model resident)
  pipeline.py
  backends/           # transformers | florence | openvino
  config/             # models + task prompts
outputs/              # last_result.json, last_overlay.jpg
models/ov/            # local OpenVINO IR (gitignored; `.ovcache/` for compile cache)

License

Code in this repository is MIT — see LICENSE.

Upstream model weights keep their own licenses (Apache-2.0 / MIT). This project does not redistribute weights; ForgeEye downloads them from Hugging Face / exports IR locally. Do not commit models/ binaries to git.