Skip to content

feat: Refactor scraping pipeline with DB-backed persistence, UK spelling migration, and legacy cleanup - #323

Merged
Pierre-VF merged 25 commits into
mainfrom
pvf/maintainability-fix
Jul 19, 2026
Merged

feat: Refactor scraping pipeline with DB-backed persistence, UK spelling migration, and legacy cleanup#323
Pierre-VF merged 25 commits into
mainfrom
pvf/maintainability-fix

Conversation

@Pierre-VF

Copy link
Copy Markdown
Owner

This branch introduces several interconnected changes across the scraping system, database layer, CLI tooling, and frontend templates. The primary goal is to improve maintainability, correctness of error handling, and consistency in naming conventions throughout the codebase.

New repository scraper system with DB-backed persistence

  • src/oss4climate/src/repository_scraper.py — A new RepositoryScraper class that orchestrates the full scraping lifecycle: syncing from TOML index (upsert orgs/repos, discover repos, prune inactive), incremental re-scraping based on a configurable refresh interval (refresh_days=28 by default), and exporting results to feather files plus summary/failure TOML outputs.
  • src/oss4climate/src/database/repos.py — New SQLModel-backed tables for Organisation and Repository, replacing the flat-file approach with structured, queryable database records including error tracking (last_error, error_count) and staleness management via mark_repos_inactive.
  • src/oss4climate/src/database/scrape_cache.py — Extracted the old cache logic into its own module to separate concerns between scrape caching (HTTP-level) and repository persistence.

Database refactoring

  • The database init (__init__.py) was restructured: models are now lazily imported from their respective modules, engine creation uses a connection string from settings instead of hardcoded paths, and the global _ENGINE is initialized on first use rather than at import time to avoid eager table creation.
  • Error mapping in parsers was refactored for correctness — individual platform scrapers (GitHub, GitLab, Bitbucket) now have clearer error handling per scraper type.

UK spelling migration (licenselicence)

  • Renamed src/oss4climate/src/parsers/licenses.py to licences.py.
  • Updated all references across the codebase: Typesense schema fields (organisation, licenseorganisation_id, licence), model field names, and template bindings. This aligns with the project's UK English convention.

Legacy cleanup

  • Templates: Removed old v1 templates (_base.html, _navbar.html, results.html) in favour of consolidated v2 templates under src/oss4climate_app/templates/v2/.
  • CLI search feature removed: Deleted src/oss4climate_scripts/src/search.py (TypeScript-style CLI search) as it was superseded by the Typesense-powered web UI.
  • Repo-assist workflow files: Removed .github/workflows/repo-assist.lock.yml and .github/workflows/repo-assist.md.

Logging improvements

  • Replaced all print() calls in logging utilities with proper logging.info() / logging.warning(), configured at INFO level by default.
  • Updated callers across the codebase to use structured log functions instead of raw prints, improving traceability and consistency between library (oss4climate) and app (oss4climate_app).

Performance fix

  • Resolved an O(m × n) inefficiency in repository matching logic (likely in parsers or scraper orchestration), reducing redundant lookups when processing multiple repositories.

Typesense schema updates

  • Made several fields optional in the search index: readme, embedding_readme, organisation_id, licence, language, url, and last_commit_timestamp. This prevents indexing failures for repos with incomplete metadata.
  • Changed idx field type from int32 to string for compatibility with database IDs.

Documentation

  • Added docs/scraping-process.md documenting the scraping pipeline, its stages (TOML sync → scrape → export), and configuration options.

@Pierre-VF
Pierre-VF merged commit 62720d5 into main Jul 19, 2026
1 check passed
@Pierre-VF
Pierre-VF deleted the pvf/maintainability-fix branch July 19, 2026 11:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant