ddg-search-mcp

.gitignore

@@ -37,4 +37,9 @@ go.work.sum
 .DS_Store
 
 # AI agent commands
-.roo/
+.roo/
+
+draft.txt
+
+/bin
+/ddg-search-mcp

.golangci.yml

@@ -31,11 +31,3 @@ linters:
       - linters:
           - errcheck
         text: 'Error return value of `.*Close` is not checked'
-      # Allow longer lines in test files
-      - linters:
-          - lll
-        path: _test\.go
-      # Test files don't need separate package
-      - linters:
-          - testpackage
-        path: _test\.go

.mise.toml

@@ -14,7 +14,7 @@ GOLANGCI_VERSION = "v2.9.0"
 # Task definitions
 [tasks.lint]
 description = "Run golangci-lint with auto-fix"
-run = "golangci-lint run --fix ./..."
+run = "golangci-lint run --whole-files --fix ./..."
 
 [tasks.test]
 description = "Run all Go tests with coverage and race detector"

.serena/.gitignore

@@ -0,0 +1 @@
+/cache

.serena/memories/code-style-and-conventions.md

@@ -0,0 +1,111 @@
+# Code Style and Conventions for ddg-search
+
+## Go Code Style
+
+### Naming Conventions
+- **Package names**: Lowercase, single word (e.g., `search`, `config`, `perplexity`)
+- **Exported symbols**: PascalCase (e.g., `NewClient`, `SearchOptions`)
+- **Private symbols**: camelCase (e.g., `httpClient`, `retryOptions`)
+- **Constants**: PascalCase (e.g., `DefaultMaxRetries`, `apiBaseURL`)
+- **Interfaces**: Usually simple names ending with behavior (e.g., `Searcher`)
+
+### Error Handling
+- Use sentinel errors for common error cases (e.g., `ErrRateLimited`, `ErrMaxRetries`)
+- Wrap errors with context using `fmt.Errorf` or `errors.Wrap`
+- Return errors as the last return value
+- Check errors immediately after function calls
+
+### Struct Design
+- Use pointer receivers for methods that modify the struct
+- Use value receivers for methods that don't modify the struct
+- Exported fields use PascalCase
+- Private fields use camelCase
+
+### Functions and Methods
+- Keep functions focused and small
+- Use descriptive names that indicate purpose
+- Constructor functions use `New` prefix (e.g., `NewClient`, `NewSearcher`)
+- Methods that return multiple values follow Go conventions (e.g., `result, err`)
+
+### Comments and Documentation
+- Use godoc-style comments for exported symbols
+- Package comments at the top of each file
+- Function/method comments describe what it does, parameters, and return values
+- Inline comments for complex logic
+
+### Testing
+- Test files named `*_test.go` in the same package
+- Use table-driven tests for multiple test cases
+- Test both success and error paths
+- Use `t.Run()` for subtests
+- Integration tests for external API interactions
+
+### Configuration
+- Use structs for configuration options
+- Provide default values via `Default*` functions or constants
+- Use functional options pattern for complex configuration
+
+## Project-Specific Conventions
+
+### Retry Logic
+- Both DuckDuckGo and Perplexity clients use exponential backoff with jitter
+- Default retry options: `DefaultRetryOptions()` in `internal/config`
+- Retry configuration: `MaxRetries`, `BaseDelay`, `MaxDelay`, `BackoffMultiplier`, `Debug`
+
+### Output Formats
+- Search results support both JSON and Markdown output
+- Markdown output is optimized for LLM consumption
+- JSON output for programmatic use
+
+### Rate Limiting
+- Automatic detection of rate limit responses (HTTP 202, 429, 5xx)
+- Graceful failure after max retries with clear error messages
+- Debug mode logs rate limit information to stderr
+
+### CLI Structure
+- Use `urfave/cli/v3` for CLI framework
+- Commands defined in `cmd/*/main.go`
+- Each tool has a `main()` and a `run*()` function
+- Version constants defined in each main.go
+
+## Linting Configuration
+
+The project uses golangci-lint with the following notable settings:
+- **Version**: v2.9.0
+- **Disabled linters**:
+  - `exhaustruct` - too noisy for partial struct initialization
+  - `ireturn` - returning interfaces is acceptable
+  - `varnamelen` - short variable names are idiomatic in Go
+  - `tagliatelle` - JSON tags follow external API conventions
+  - `dupl` - duplicate detection too noisy for small project
+  - `cyclop` - cyclomatic complexity covered by other linters
+  - `depguard` - project-internal imports trigger false positives
+  - `wsl` - deprecated, replaced by wsl_v5
+- **Settings**:
+  - `gocognit.min-complexity`: 15
+  - `funlen.lines`: 80
+  - `funlen.statements`: 50
+- **Exclusions**: errcheck for deferred `Close()` calls
+
+## Testing Requirements
+
+- **Coverage threshold**: 50% minimum (enforced in CI)
+- **Race detector**: Always enabled in tests (`-race` flag)
+- **Coverage mode**: atomic (`-covermode=atomic`)
+- **Test timeout**: 5 minutes (configured in .golangci.yml)
+
+## File Organization
+
+- `cmd/` - Entry points for CLI tools
+- `internal/` - Internal packages not meant for external use
+- `skills/` - Roo skills for automation
+- `openspec/` - OpenSpec change management
+- `.github/` - CI/CD workflows
+
+## Dependencies
+
+Key external dependencies:
+- `github.com/urfave/cli/v3` - CLI framework
+- `github.com/go-resty/resty/v2` - HTTP client
+- `github.com/PuerkitoBio/goquery` - HTML parsing
+- `github.com/JohannesKaufmann/html-to-markdown/v2` - Markdown conversion

.serena/memories/project-overview.md

@@ -0,0 +1,96 @@
+# ddg-search Project Overview
+
+## Purpose
+
+Command-line tools for web search and content fetching. The project provides three main tools:
+
+1. **ddg-search** - DuckDuckGo Search Client (no API key required)
+2. **page-dump** - URL to Markdown Converter
+3. **perplexity-search** - Perplexity API Search Client (requires API key)
+
+## Tech Stack
+
+- **Language**: Go 1.25.0
+- **CLI Framework**: urfave/cli/v3
+- **HTTP Client**: go-resty/resty/v2
+- **HTML Parsing**: goquery
+- **Markdown Conversion**: html-to-markdown/v2
+
+## Project Structure
+
+```
+ddg-search/
+├── cmd/                    # Entry points for CLI tools
+│   ├── ddg-search/        # DuckDuckGo search CLI
+│   ├── page-dump/         # URL to markdown converter
+│   └── perplexity-search/ # Perplexity API search CLI
+├── internal/              # Internal packages
+│   ├── config/           # Configuration structures (RetryOptions, SearchOptions, Result)
+│   ├── search/           # DuckDuckGo search implementation
+│   │   ├── client.go     # HTTP client with retry logic
+│   │   ├── parser.go     # HTML result parser
+│   │   └── search.go     # Searcher interface
+│   ├── perplexity/       # Perplexity API client
+│   │   ├── client.go     # API client with retry logic
+│   │   └── search.go     # Search implementation
+│   └── dump/             # Page fetching and markdown conversion
+├── skills/               # Roo skills for automation
+│   ├── ddg-search/
+│   └── perplexity-search/
+├── openspec/             # OpenSpec change management
+│   ├── changes/          # Active changes
+│   ├── changes/archive/  # Archived changes
+│   └── specs/            # Main specifications
+└── .github/              # CI/CD workflows
+```
+
+## Key Components
+
+### internal/config
+- `RetryOptions`: Configures retry behavior (MaxRetries, BaseDelay, MaxDelay, BackoffMultiplier, Debug)
+- `SearchOptions`: Search parameters (Query, MaxResults, Site, Region, TimeFilter, SafeSearch)
+- `Result`: Search result structure (Title, URL, Snippet)
+
+### internal/search
+- `Client`: HTTP client with automatic rate limit detection and retry with exponential backoff
+- `Parser`: Parses DuckDuckGo HTML results
+- `Searcher`: High-level search interface with JSON and Markdown output
+
+### internal/perplexity
+- `Client`: Perplexity API client with retry logic
+- `SearchOptions`: Query, MaxResults, Model
+- `SearchResults`: Answer, Citations, References
+
+### internal/dump
+- `Fetch`: Fetches web pages with configurable timeout and user-agent
+- `Convert`: Converts HTML to markdown preserving structure
+
+## Features
+
+### ddg-search
+- Markdown output for LLM consumption
+- Automatic rate limit detection and retry with exponential backoff
+- Site-specific, regional, and time-bounded searches
+- Configurable retry behavior
+
+### page-dump
+- Fetch any HTTP/HTTPS URL and convert to markdown
+- Preserves document structure (headings, links, lists, code blocks)
+- Configurable timeout and user-agent
+
+### perplexity-search
+- AI-powered search results with citations
+- Configurable model selection (sonar-small-online, sonar-medium-online, sonar-pro-online)
+- Automatic retry with exponential backoff
+- Markdown output for LLM consumption
+
+## Rate Limiting
+
+Both DuckDuckGo and Perplexity clients automatically:
+1. Detect rate-limit responses (HTTP 202, 429, 5xx)
+2. Retry with exponential backoff + jitter
+3. Fail gracefully with clear error messages after max retries
+
+## License
+
+MIT

.serena/memories/suggested-commands.md

@@ -0,0 +1,146 @@
+# Suggested Commands for ddg-search
+
+## Development Commands
+
+### Building
+```bash
+# Build all binaries
+go build -o bin/ddg-search ./cmd/ddg-search
+go build -o bin/page-dump ./cmd/page-dump
+go build -o bin/perplexity-search ./cmd/perplexity-search
+
+# Build specific binary
+go build -o bin/ddg-search ./cmd/ddg-search
+```
+
+### Testing
+```bash
+# Run all tests with coverage and race detector
+go test -cover -race ./...
+
+# Run tests with verbose output
+go test -v ./...
+
+# Run tests for specific package
+go test -v ./internal/search
+go test -v ./internal/perplexity
+go test -v ./internal/dump
+go test -v ./internal/config
+
+# Run tests with coverage profile
+go test -coverprofile=coverage.out -covermode=atomic -race ./...
+
+# View coverage report
+go tool cover -html=coverage.out
+go tool cover -func=coverage.out
+```
+
+### Linting
+```bash
+# Run golangci-lint with auto-fix
+golangci-lint run --new-from-merge-base main --whole-files --fix ./...
+
+# Run golangci-lint without auto-fix
+golangci-lint run ./...
+
+# Run specific linters
+golangci-lint run --disable-all --enable=gofmt,goimports ./...
+```
+
+### Using mise (task runner)
+```bash
+# Run linting
+mise run lint
+
+# Run tests
+mise run test
+```
+
+## Running the Tools
+
+### ddg-search
+```bash
+# Basic search
+ddg-search golang
+
+# With options
+ddg-search --max-results 5 --site github.com docker compose
+ddg-search --region uk-en premier league
+ddg-search --time w news today
+ddg-search --max-retries 5 --retry-delay 2s --max-retry-delay 60s slow query
+ddg-search --debug golang
+```
+
+### page-dump
+```bash
+# Basic usage
+page-dump https://example.com
+
+# With options
+page-dump --timeout 60s --user-agent "my-agent/1.0" https://example.com
+```
+
+### perplexity-search
+```bash
+# Set API key first
+export PERPLEXITY_API_KEY="your-api-key"
+
+# Basic search
+perplexity-search "What is Go programming language?"
+
+# With options
+perplexity-search --max-results 3 golang tutorial
+perplexity-search --model sonar-pro-online "machine learning fundamentals"
+perplexity-search --debug "kubernetes deployment strategies"
+```
+
+## System Commands (Darwin/macOS)
+
+```bash
+# List files
+ls -la
+
+# Find files
+find . -name "*.go"
+find . -type f -name "*.go" | head -20
+
+# Search in files
+grep -r "Searcher" ./internal/
+grep -r "func New" ./internal/
+
+# Check git status
+git status
+git diff
+git log --oneline -10
+
+# Change directory
+cd /path/to/directory
+
+# Show file contents
+cat file.txt
+head -n 20 file.txt
+tail -n 20 file.txt
+
+# Process management
+ps aux | grep ddg-search
+kill <pid>
+```
+
+## CI/CD Commands
+
+The project uses GitHub Actions for CI/CD:
+- **Lint**: Runs golangci-lint v2.9.0
+- **Test**: Runs tests with coverage, race detector, and enforces 50% coverage threshold
+- **Build**: Builds all binaries and verifies they work with `--help`
+
+## Installation
+
+```bash
+# Install from source
+go install github.com/Djarvur/ddg-search@latest
+
+# Or build from source
+git clone https://github.com/Djarvur/ddg-search
+cd ddg-search
+make build
+```