Groundhog is a local-first autonomous web agent that navigates the internet to accomplish natural language goals.
Unlike most agents that rely on expensive, closed-source APIs (like GPT-4o), Groundhog runs entirely on open-source models (Qwen2.5-VL) that you can run yourself. On top of that, 🦫 was built without using any agent-building frameworks (Langchain, AutoGen, etc).
I have prepared a Google Colab notebook to use 🦫 that installs Chrome, loads the model, and launches a web interface for you to watch the agent work.
The core is groundhog-v1, a model I fine-tuned using Qwen2.5-VL-7B-Instruct as the base model. It was trained using QLoRA on a preprocessed subset of the Mind2Web dataset to specialize in web navigation, turning visual UI elements into actionable JSON commands.
Raw HTML is too noisy for VLMs. Groundhog uses a custom inference pipeline:
- A custom JavaScript engine "stamps" every interactive element in the browser with a unique ID (
[1],[2]). - It then extracts a simplified DOM tree containing only elements currently visible in the viewport.
- The model receives the Screenshot and the Distilled DOM. It correlates the visual location of a button with its stamped ID.
- The model outputs a JSON command (e.g.,
{"action": "click", "element_id": "15"}).
Language models hallucinate. Groundhog wraps the model in a Python controller that validates predictions against the live browser state. If the model tries to click a non-existent ID, the controller catches it and triggers a retry or scroll sequence instead of crashing.
To run Groundhog locally, you need a machine with an NVIDIA GPU (8GB+ VRAM) because the model uses 4-bit quantization (BitsAndBytes).
- Clone the repository
git clone https://github.com/shivamg05/groundhog.git cd groundhog
for more information on me visit https://shivamgarg.me!