Testing, Linting, and Code Coverage with UV

This guide covers how to run tests, perform linting, and generate code coverage reports using UV after the migration from setuptools to modern UV-based dependency management.

Quick Reference

# Install dependencies (production)
uv sync --frozen

# Install with development tools
uv sync --extra dev --frozen

# Run all tests
uv run pytest

# Run tests with coverage
uv run coverage run -m pytest
uv run coverage report

# Run linting
uv run ruff check .
uv run ruff check --fix .  # Auto-fix issues
uv run pylint aiagents4pharma/

# Run security scanning
uv run bandit -r aiagents4pharma/
uv run pip-audit

# Format code
uv run ruff format .

# Pre-commit hooks (runs all checks at once)
uv run pre-commit run --all-files

# Run individual pre-commit checks
uv run pre-commit run ruff --all-files         # Linting only
uv run pre-commit run ruff-format --all-files  # Formatting only
uv run pre-commit run bandit --all-files       # Security only
uv run pre-commit run pip-audit --all-files    # Vulnerabilities only

Setup & Installation

1. Install UV

Follow the official UV installation guide:

# macOS/Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

# Windows
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"

# With pip
pip install uv

2. Install Dependencies

# Clone repository
git clone https://github.com/VirtualPatientEngine/AIAgents4Pharma
cd AIAgents4Pharma

# Sync dependencies (creates .venv and installs everything)
uv sync --frozen

# Install with development dependencies (recommended)
uv sync --extra dev --frozen

3. System Prerequisites

macOS/Linux users:

# macOS
brew install libmagic

# Linux (Ubuntu/Debian)
sudo apt-get install libmagic1

# Linux (RHEL/CentOS)
sudo yum install file-libs

Windows users: libmagic is bundled automatically with python-magic.

Package Management with UV

Adding New Dependencies

Add Runtime Dependencies

# Add a new package to main dependencies
uv add "package-name"

# Add with specific version constraint
uv add "package-name>=1.0.0,<2.0.0"

# Add with exact version
uv add "package-name==1.2.3"

# Add from specific index
uv add "package-name" --index-url https://pypi.org/simple/

Add Development Dependencies

# Add to development dependencies
uv add --dev "pytest-mock"

# Add to specific extra group
uv add --extra dev "pip-audit==2.9.0"

Add from Git Repository

# Install from Git repository
uv add "git+https://github.com/user/repo.git"

# Install from specific branch/tag
uv add "git+https://github.com/user/repo.git@main"
uv add "git+https://github.com/user/repo.git@v1.0.0"

Removing Dependencies

# Remove a package
uv remove "package-name"

# Remove development dependency
uv remove --dev "package-name"

Updating Dependencies

# Update all dependencies
uv lock --upgrade

# Update specific package
uv add "package-name" --upgrade

# Update to latest compatible versions
uv sync --upgrade

Managing Virtual Environments

# Create virtual environment (automatic with uv sync)
uv venv

# Activate virtual environment
source .venv/bin/activate  # Unix/macOS
.venv\Scripts\activate     # Windows

# Install current project in development mode
uv pip install -e .

# List installed packages
uv pip list

# Show package information
uv pip show "package-name"

Lock File Management

# Generate/update uv.lock
uv lock

# Install from lock file (exact versions)
uv sync --frozen

# Check for dependency conflicts
uv pip check

Testing

Running Tests

Basic Test Execution

# Run all tests
uv run pytest

# Run with verbose output
uv run pytest -v

# Run specific test file
uv run pytest aiagents4pharma/talk2scholars/tests/test_main_agent.py

# Run specific test function
uv run pytest aiagents4pharma/talk2scholars/tests/test_main_agent.py::test_specific_function

# Run tests matching a pattern
uv run pytest -k "test_pdf"

Component-Specific Testing

# Test individual agents
uv run pytest aiagents4pharma/talk2scholars/tests/
uv run pytest aiagents4pharma/talk2biomodels/tests/
uv run pytest aiagents4pharma/talk2knowledgegraphs/tests/
uv run pytest aiagents4pharma/talk2aiagents4pharma/tests/
uv run pytest aiagents4pharma/talk2cells/tests/

Submodule-Specific Quality Checks

Pylint for Individual Submodules

# Run pylint on specific submodules (configuration in pyproject.toml)
uv run pylint aiagents4pharma/talk2scholars/
uv run pylint aiagents4pharma/talk2biomodels/
uv run pylint aiagents4pharma/talk2knowledgegraphs/
uv run pylint aiagents4pharma/talk2aiagents4pharma/
uv run pylint aiagents4pharma/talk2cells/

Coverage for Individual Submodules

# Run coverage on specific submodules
uv run coverage run --include="aiagents4pharma/talk2scholars/*" -m pytest --cache-clear aiagents4pharma/talk2scholars/tests/ && uv run coverage report
uv run coverage run --include="aiagents4pharma/talk2biomodels/*" -m pytest --cache-clear aiagents4pharma/talk2biomodels/tests/ && uv run coverage report
uv run coverage run --include="aiagents4pharma/talk2knowledgegraphs/*" -m pytest --cache-clear aiagents4pharma/talk2knowledgegraphs/tests/ && uv run coverage report
uv run coverage run --include="aiagents4pharma/talk2aiagents4pharma/*" -m pytest --cache-clear aiagents4pharma/talk2aiagents4pharma/tests/ && uv run coverage report
uv run coverage run --include="aiagents4pharma/talk2cells/*" -m pytest --cache-clear aiagents4pharma/talk2cells/tests/ && uv run coverage report

Pre-commit Hooks for Specific Files/Directories

# Run pre-commit on specific submodule
uv run pre-commit run --files aiagents4pharma/talk2scholars/*.py
uv run pre-commit run --files aiagents4pharma/talk2biomodels/*.py
uv run pre-commit run --files aiagents4pharma/talk2knowledgegraphs/*.py

# Run specific hook on specific submodule
uv run pre-commit run ruff --files aiagents4pharma/talk2scholars/*.py
uv run pre-commit run bandit --files aiagents4pharma/talk2knowledgegraphs/*.py

Test Configuration

Tests are configured in pyproject.toml:

[tool.pytest.ini_options]
asyncio_default_fixture_loop_scope = "function"
markers = ["integration: marks tests as integration tests"]
filterwarnings = ["ignore::DeprecationWarning"]

Environment Variables for Testing

# Required for certain tests
export OPENAI_API_KEY="your-openai-key"
export ZOTERO_API_KEY="your-zotero-key"  # For Talk2Scholars tests
export ZOTERO_USER_ID="your-zotero-id"   # For Talk2Scholars tests

# Optional for LangSmith tracing
export LANGCHAIN_TRACING_V2=true
export LANGCHAIN_API_KEY="your-langsmith-key"

Code Coverage

Basic Coverage Commands

Generate Coverage Reports

# Run tests with coverage
uv run coverage run -m pytest

# Show coverage report in terminal
uv run coverage report

# Generate HTML coverage report
uv run coverage html

# Generate XML coverage report (for CI/CD)
uv run coverage xml

Advanced Coverage Usage

Component-Specific Coverage:

# Coverage for specific component (standard configuration)
# Note: Use quotes on macOS/zsh shell, GitHub Actions CI doesn't require them
uv run coverage run --include="aiagents4pharma/talk2scholars/*" -m pytest --cache-clear aiagents4pharma/talk2scholars/tests/ && uv run coverage report

# Run coverage on each subfolder individually (local development - use quotes for macOS/zsh)
uv run coverage run --include="aiagents4pharma/talk2scholars/*" -m pytest --cache-clear aiagents4pharma/talk2scholars/tests/ && uv run coverage report
uv run coverage run --include="aiagents4pharma/talk2biomodels/*" -m pytest --cache-clear aiagents4pharma/talk2biomodels/tests/ && uv run coverage report
uv run coverage run --include="aiagents4pharma/talk2knowledgegraphs/*" -m pytest --cache-clear aiagents4pharma/talk2knowledgegraphs/tests/ && uv run coverage report
uv run coverage run --include="aiagents4pharma/talk2aiagents4pharma/*" -m pytest --cache-clear aiagents4pharma/talk2aiagents4pharma/tests/ && uv run coverage report
uv run coverage run --include="aiagents4pharma/talk2cells/*" -m pytest --cache-clear aiagents4pharma/talk2cells/tests/ && uv run coverage report

# Alternative format for CI environments (without quotes)
# uv run coverage run --include=aiagents4pharma/talk2scholars/* -m pytest --cache-clear aiagents4pharma/talk2scholars/tests/

# Coverage with source specification (all components)
uv run coverage run --source=aiagents4pharma -m pytest

# Show missing lines
uv run coverage report -m

# Show coverage by file with line numbers
uv run coverage report --show-missing

Coverage Configuration (pyproject.toml):

[tool.coverage.run]
source = ["aiagents4pharma"]
omit = [
    "*/tests/*",
    "*/test_*",
    "docs/*",
    "app/*",
]

[tool.coverage.report]
exclude_lines = [
    "pragma: no cover",
    "def __repr__",
    "if self.debug:",
    "if settings.DEBUG",
    "raise AssertionError",
    "raise NotImplementedError",
    "if 0:",
    "if __name__ == .__main__.:",
    "class .*\\bProtocol\\):",
    "@(abc\\.)?abstractmethod",
]

[tool.coverage.xml]
output = "coverage.xml"

Coverage Thresholds

Check Coverage Threshold:

# Check if coverage meets minimum threshold
COVERAGE=$(uv run coverage report -m | awk 'END {print int($NF)}')
if [[ $COVERAGE -lt 80 ]]; then
    echo "Coverage is below 80%"
    exit 1
fi

HTML Reports: Open htmlcov/index.html after running uv run coverage html to see detailed coverage information with highlighted missing lines.

Linting & Code Quality

Ruff (Fast Python Linter)

Basic Ruff Commands

# Check all files
uv run ruff check .

# Check specific directory
uv run ruff check aiagents4pharma/

# Fix auto-fixable issues
uv run ruff check --fix .

# Check with specific rules
uv run ruff check --select E,W,F .

# Format code
uv run ruff format .

Ruff Configuration (`pyproject.toml`)

[tool.ruff]
target-version = "py312"
line-length = 100

[tool.ruff.lint]
select = [
    "E",  # pycodestyle errors
    "W",  # pycodestyle warnings
    "F",  # pyflakes
    "I",  # isort
    "B",  # flake8-bugbear
    "C4", # flake8-comprehensions
    "UP", # pyupgrade
]
ignore = [
    "B008",  # do not perform function calls in argument defaults
    "C901",  # too complex
]

[tool.ruff.lint.per-file-ignores]
"__init__.py" = ["F401"]

Pylint (Comprehensive Static Analysis)

Basic Pylint Commands

# Run pylint on entire codebase
uv run pylint aiagents4pharma/

# Run on specific component
uv run pylint aiagents4pharma/talk2scholars/

# Standard configuration (disabled rules configured in pyproject.toml)
uv run pylint aiagents4pharma/

# Run on each subfolder individually
uv run pylint aiagents4pharma/talk2scholars/
uv run pylint aiagents4pharma/talk2biomodels/
uv run pylint aiagents4pharma/talk2knowledgegraphs/
uv run pylint aiagents4pharma/talk2cells/

# Generate JSON report
uv run pylint aiagents4pharma/ --output-format=json --reports=no > pylint-report.json

# Show only errors and warnings
uv run pylint aiagents4pharma/ --errors-only

Pylint Configuration (`pyproject.toml`)

[tool.pylint.MASTER]
extension-pkg-allow-list = ["pcst_fast"]

[tool.pylint.messages_control]
disable = ["R0801", "R0902", "W0221", "W0122"]

Security Scanning

Bandit (Security Linter)

Basic Bandit Commands

# Scan entire codebase
uv run bandit -r aiagents4pharma/

# Use configuration from pyproject.toml
uv run bandit -c pyproject.toml -r aiagents4pharma/

# Generate JSON report
uv run bandit -c pyproject.toml -f json -o bandit-report.json -r aiagents4pharma/

# Show only high severity issues
uv run bandit -r aiagents4pharma/ -ll

Bandit Configuration (`pyproject.toml`)

[tool.bandit]
exclude_dirs = ["tests", "test_*"]
skips = ["B101", "B601"]

[tool.bandit.assert_used]
skips = ["*_test.py", "*/test_*.py"]

Vulnerability Scanning

pip-audit (Dependency Vulnerability Scanner)

# Basic vulnerability scan
uv run pip-audit

# Generate reports
uv run pip-audit --desc --format=json --output=audit-report.json
uv run pip-audit --desc --format=markdown --output=audit-report.md

# Scan specific requirements
uv run pip-audit --desc --format=table

safety (Alternative Vulnerability Scanner)

# Basic safety check
uv run safety check

# JSON output
uv run safety check --json --output safety-report.json

# Check specific files
uv run safety check --file=uv.lock

Type Checking (MyPy)

Current Status

MyPy is currently disabled in pre-commit due to 1,121 type annotation errors that require dedicated cleanup effort.

Manual MyPy Execution

# Run mypy (will show many errors)
uv run mypy aiagents4pharma/

# Run with lenient settings
uv run mypy aiagents4pharma/ --ignore-missing-imports --show-error-codes

# Check specific component
uv run mypy aiagents4pharma/talk2scholars/ --ignore-missing-imports

MyPy Configuration (`pyproject.toml`)

[tool.mypy]
python_version = "3.12"
warn_return_any = true
warn_unused_configs = true
disallow_untyped_defs = true
disallow_incomplete_defs = true
check_untyped_defs = true
disallow_untyped_decorators = true
no_implicit_optional = true
warn_redundant_casts = true
warn_unused_ignores = true
warn_no_return = true
warn_unreachable = true
strict_equality = true

[[tool.mypy.overrides]]
module = [
    "setuptools.*",
    "pkg_resources.*",
]
ignore_missing_imports = true

Pre-commit Hooks

Installation & Setup

# Install pre-commit hooks
uv run pre-commit install

# Run hooks on all files
uv run pre-commit run --all-files

# Run specific hook
uv run pre-commit run ruff

# Update hooks to latest versions
uv run pre-commit autoupdate

Pre-commit Configuration (`.pre-commit-config.yaml`)

The repository includes comprehensive pre-commit hooks:

Ruff: Fast linting and code formatting
Bandit: Security vulnerability scanning
General hooks: Trailing whitespace, YAML validation, large file checks, etc.
pip-audit: Dependency vulnerability scanning

Individual Pre-commit Commands

# Run all hooks
uv run pre-commit run --all-files

# Run specific hooks
uv run pre-commit run ruff --all-files         # Linting with auto-fix
uv run pre-commit run ruff-format --all-files  # Code formatting
uv run pre-commit run bandit --all-files       # Security scanning
uv run pre-commit run pip-audit --all-files    # Dependency vulnerabilities

# General quality checks
uv run pre-commit run trailing-whitespace --all-files
uv run pre-commit run end-of-file-fixer --all-files
uv run pre-commit run check-yaml --all-files
uv run pre-commit run check-added-large-files --all-files

# Run on specific files
uv run pre-commit run ruff --files aiagents4pharma/talk2scholars/main.py
uv run pre-commit run bandit --files aiagents4pharma/talk2knowledgegraphs/tools/*.py

Bypassing Hooks (Emergency Use)

# Skip all hooks for urgent commits
git commit -m "urgent fix" --no-verify

# Skip specific hooks
SKIP=ruff git commit -m "work in progress"

Automation & CI Integration

GitHub Actions Integration

All these tools are integrated into our CI/CD pipeline:

# Simulate CI locally
uv run pytest                                    # Tests
uv run coverage run -m pytest && coverage report # Coverage
uv run ruff check .                              # Linting
uv run pylint aiagents4pharma/                   # Static analysis
uv run bandit -c pyproject.toml -r aiagents4pharma/ # Security
uv run pip-audit                                 # Vulnerabilities

Local Development Workflow

# 1. Make changes to code
git checkout -b feature/my-feature

# 2. Run tests and quality checks
uv run pytest
uv run coverage run -m pytest && uv run coverage report
uv run ruff check --fix .

# 3. Run pre-commit hooks
uv run pre-commit run --all-files

# 4. Commit changes
git add .
git commit -m "feat: add new feature"

# 5. Push and create PR
git push origin feature/my-feature

Performance & Optimization

UV Advantages

Speed: UV is 10-100x faster than pip
Reliability: Deterministic dependency resolution
Caching: Smart caching reduces repeat installations
Lock files: uv.lock ensures reproducible environments

Development Tips

Faster Test Iteration

# Run tests with pytest-xdist for parallel execution
uv add --dev pytest-xdist
uv run pytest -n auto

# Use pytest cache for faster re-runs
uv run pytest --cache-clear  # Clear cache when needed
uv run pytest               # Reuse cache for speed

Efficient Coverage

# Run coverage only on changed files
git diff --name-only | grep "\.py$" | xargs uv run coverage run -m pytest

# Use coverage's fast mode
uv run coverage run --parallel-mode -m pytest
uv run coverage combine

Troubleshooting

Common Issues

Dependency Conflicts

# Fix dependency conflicts using uv
uv add "package>=new-version"
uv lock --upgrade

# Example: Fix PyArrow compatibility
uv add "pyarrow>=14.0.0"
uv add "datasets>=4.0.0"

# Clear and reinstall if needed
rm -rf .venv
uv sync --extra dev --frozen

Import Errors

# Ensure you're in the right environment
uv run python -c "import sys; print(sys.executable)"

# Check installed packages
uv pip list

Permission Issues (Linux/macOS)

# Fix ownership if needed
sudo chown -R $USER:$USER .venv

# Alternative: use --user flag
uv sync --user

Windows-Specific Issues

# Use PowerShell execution policy
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

# Use forward slashes in paths
uv run pytest aiagents4pharma/talk2scholars/tests/

Getting Help

UV Documentation: https://docs.astral.sh/uv/
pytest Documentation: https://docs.pytest.org/
Repository Issues: https://github.com/VirtualPatientEngine/AIAgents4Pharma/issues

This guide provides a comprehensive overview of testing, linting, and code coverage using UV. The modern toolchain ensures fast, reliable, and secure development practices.

Uh oh!

FilesExpand file tree

TESTING_LINTING.md

Latest commit

History