Skip to content

Repoint hardcoded bbop-sqlite S3 URLs to the semanticsql CDN before raw-bucket access is retired #41

Description

@caufieldjh

Repoint hardcoded bbop-sqlite S3 URLs to the SemanticSQL CDN

universalizer downloads pre-built OAK/SemanticSQL SQLite databases by hardcoding the raw S3 bucket URL. The maintainers of that bucket (Berkeley BOP / INCATools SemanticSQL) are retiring direct public access to the raw S3 bucket and moving all traffic behind a CDN. Because this URL is hardcoded (rather than resolved through OAK'''s sqlite:obo: selector or the semsql package), it will not auto-migrate and will break once raw-bucket access is removed.

Where this repo is affected

  • universalizer/oak_utils.pyf"https://s3.amazonaws.com/bbop-sqlite/{db}.db"

What to change

  • Old: https://s3.amazonaws.com/bbop-sqlite/{db}.db
  • New: https://semanticsql.berkeleybop.io/{db}.db

Important

You must also send a non-default User-Agent. The CDN sits behind a Browser Integrity Check that returns HTTP 403 to default client User-Agents (Python's Python-urllib/3.x, R's download.file/httr, and bare curl/wget defaults). A host-only swap will start returning 403 — set an explicit, non-default User-Agent header on the request.

If you fetch with urllib/requests, add an explicit header, e.g. urllib.request.Request(url, headers={"User-Agent": "universalizer/1.0"}).

Why now

Filed as part of the coordinated downstream migration tracked in INCATools/semantic-sql#115. If you've already migrated or this is a non-issue, feel free to close — thanks!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions