virgo

Turn web pages into Markdown using the Chrome DevTools Protocol

Install

You can install the primary CLI utility using go install:

go install github.com/mjc-gh/virgo/cmd/virgo@latest

Usage

virgo
NAME:
   virgo - A tool for converting webpages to Markdown and plaintext.

USAGE:
   virgo [global options] [command [command options]]

VERSION:
   0.0.0

COMMANDS:
   screenshot  Screenshot one or more URLs
   markdown    Get the markdown content of a URL
   plaintext   Get the plantext content of a URL
   links       Search for links on a page using fuzzy matching
   help, h     Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --help, -h     show help
   --version, -v  print the version

Markdown subcommand

virgo markdown --help
NAME:
   virgo markdown - Get the markdown content of a URL

USAGE:
   virgo markdown [options] url

OPTIONS:
   --include-images, -i       include images in markdown output (default: false)
   --include-forms, -f        include forms in markdown output (default: false)
   --debug, -d                enable debug logging (default: false)
   --json-logs                output logs as JSON (default: false)
   --headfull, -H             run browser in headfull mode (default: false)
   --concurrency int, -c int  number of concurrent workers (default: 0)
   --remote-port int          remote DevTools port (default: 0)
   --remote-host string       remote DevTools host
   --device-type string       device type (desktop/mobile/tablet) (default: "desktop")
   --device-size string       device size preset (default: "large")
   --user-agent string        browser user-agent preset (default: "chrome")
   --help, -h                 show help

Plaintext subcommand

virgo plaintext --help
NAME:
   virgo plaintext - Get the plantext content of a URL

USAGE:
   virgo plaintext [options] url

OPTIONS:
   --selector string, -S string  CSS selector for content extraction (default: "body")
   --debug, -d                   enable debug logging (default: false)
   --json-logs                   output logs as JSON (default: false)
   --headfull, -H                run browser in headfull mode (default: false)
   --concurrency int, -c int     number of concurrent workers (default: 0)
   --remote-port int             remote DevTools port (default: 0)
   --remote-host string          remote DevTools host
   --device-type string          device type (desktop/mobile/tablet) (default: "desktop")
   --device-size string          device size preset (default: "large")
   --user-agent string           browser user-agent preset (default: "chrome")
   --help, -h                    show help

Links subcommand

virgo links --help
NAME:
   virgo links - Search for links on a page using fuzzy matching

USAGE:
   virgo links [options] url

OPTIONS:
   --search string, -s string  search term for fuzzy matching
   --threshold int, -t int     fuzzy match threshold (0=exact, higher=fuzzier) (default: 3)
   --debug, -d                 enable debug logging (default: false)
   --json-logs                 output logs as JSON (default: false)
   --headfull, -H              run browser in headfull mode (default: false)
   --concurrency int, -c int   number of concurrent workers (default: 0)
   --remote-port int           remote DevTools port (default: 0)
   --remote-host string        remote DevTools host
   --device-type string        device type (desktop/mobile/tablet) (default: "desktop")
   --device-size string        device size preset (default: "large")
   --user-agent string         browser user-agent preset (default: "chrome")
   --help, -h                  show help

Screenshot subcommand

virgo screenshot --help
NAME:
   virgo screenshot - Screenshot one or more URLs

USAGE:
   virgo screenshot [options] url [url ...]

OPTIONS:
   --output-dir string, -o string  directory for screenshots (default: "tmp/")
   --debug, -d                     enable debug logging (default: false)
   --json-logs                     output logs as JSON (default: false)
   --headfull, -H                  run browser in headfull mode (default: false)
   --concurrency int, -c int       number of concurrent workers (default: 0)
   --remote-port int               remote DevTools port (default: 0)
   --remote-host string            remote DevTools host
   --device-type string            device type (desktop/mobile/tablet) (default: "desktop")
   --device-size string            device size preset (default: "large")
   --user-agent string             browser user-agent preset (default: "chrome")
   --help, -h                      show help

Headless Shell

You can use the chromedp headless shell container if you do not have Chrome installed locally. This works well on Linux servers:

docker pull chromedp/headless-shell:latest
docker run -d -p 9222:9222 --rm --name headless-shell chromedp/headless-shell
virgo markdown --remote-port 9222 --remote-host 127.0.0.1 cnn.com

Docker

Using the virgo-web REST API Container

The virgo-web service is available as a Docker container with Chrome pre-installed. This is useful for running the REST API in isolated environments like Kubernetes, Docker Compose, or any container orchestration platform.

Quick Start

# Build the container locally
make docker.build

# Run the container
docker run -d -p 8888:8888 virgo-web:latest

# Test the API
curl http://localhost:8888/markdown?url=https://example.com

Using Pre-built Images

Pre-built container images are available from GitHub Container Registry:

# Pull the latest image
docker pull ghcr.io/mjc-gh/virgo:latest

# Run the container
docker run -d -p 8888:8888 ghcr.io/mjc-gh/virgo:latest

Images are tagged with semantic versions (e.g., v1.2.3) and latest for the most recent release.

Environment Variables

Configure the virgo-web service using environment variables:

Variable	Default	Description
VIRGO_HOST	0.0.0.0	Web server bind host
VIRGO_PORT	8888	Web server bind port
VIRGO_CONCURRENCY	1	Number of concurrent workers
VIRGO_DEBUG	false	Enable debug logging
VIRGO_JSON_LOGS	true	Output logs as JSON

Example: Running with Custom Configuration

docker run -d \
  -p 8888:8888 \
  -e VIRGO_CONCURRENCY=4 \
  -e VIRGO_DEBUG=true \
  -e VIRGO_JSON_LOGS=false \
  ghcr.io/mjc-gh/virgo:latest

Docker Compose Example

version: '3.8'

services:
  virgo-web:
    image: ghcr.io/mjc-gh/virgo:latest
    ports:
      - "8888:8888"
    environment:
      VIRGO_CONCURRENCY: 4
      VIRGO_DEBUG: false
      VIRGO_JSON_LOGS: true
    restart: unless-stopped

REST API Endpoints

The virgo-web service provides the following endpoints:

• POST /markdown - Convert URL to Markdown

  curl -X POST http://localhost:8888/markdown \
    -H "Content-Type: application/json" \
    -d '{"url": "https://example.com"}'

• POST /plaintext - Convert URL to plaintext

  curl -X POST http://localhost:8888/plaintext \
    -H "Content-Type: application/json" \
    -d '{"url": "https://example.com"}'

• POST /screenshot - Get screenshot of URL

  curl -X POST http://localhost:8888/screenshot \
    -H "Content-Type: application/json" \
    -d '{"url": "https://example.com"}' \
    --output screenshot.png

• GET /health_check - Service health status

  curl http://localhost:8888/health_check

Development

You need Go and golangci-lint

mise use -g go@1.26.3
mise use -g golangci-lint@v2.12.2

You can build and test the CLI tool with the following:

make build.cli
make test