feat(dev): add mockservices — mocks IA/third-party services for local dev (replaces /internal/fake/* stubs) by mekarpeles · Pull Request #13045 · internetarchive/openlibrary

mekarpeles · 2026-06-25T19:06:25Z

Summary

Replaces the four inline web.py stub classes in account.py with a standalone
FastAPI container, mockservices, that mocks all external IA/third-party
services that dev environments need to hit. Adds mailpit for email capture.

Stubs removed from account.py:

/internal/fake/availability → POST /services/availability/
/internal/fake/loans → POST /services/loans/loan/
/internal/fake/xauth → POST /services/xauthn/
/internal/fake/s3auth → GET /services/s3auth/

New endpoints in mockservices:

Endpoint	Was
`POST /services/xauthn/`	`/internal/fake/xauth`
`GET /services/s3auth/`	`/internal/fake/s3auth`
`POST /services/loans/loan/`	`/internal/fake/loans`
`GET /services/loans/loan/?action=changes`	not previously mocked — needed by #12689's near-realtime loan availability updater
`POST /services/availability/`	`/internal/fake/availability`
`GET /services/borrow/{id}`	hardcoded in lending.py:599
`POST /recaptcha/api/siteverify`	was hitting recaptcha.net
`GET /fts/v1/search`	implemented but NOT wired as the default — `be-api.us.archive.org` works fine for local dev as-is
`POST /paapi5/getItems`	Amazon PA-API stub
mailpit SMTP :1025 / web :8025	dummy_sendmail=True (stdout logging)

Loan changes feed: seeds 1000 borrow/return events on startup using real
ia identifiers pulled from local Solr (falls back to a static pool if Solr
is empty/unreachable), timestamps spread over the last 14 days to exercise
catchup/backfill. A background loop then adds 50 events and evicts the oldest
50 every 60s to exercise ongoing steady-state polling.

Config changes in openlibrary.yml:
Most external service URLs now point at http://mockservices:8090/....
smtp_server: mockservices, smtp_port: 1025, dummy_sendmail: False route
email to mailpit (browse at http://localhost:8025). search_endpoint is left
pointing at the real be-api.us.archive.org — it works fine for local dev.

S3 loan URL coverage:
Added ia_s3_loan_url config key to lending.py. The get_groundtruth_availability()
and s3_loan_api() functions previously used S3_LOAN_URL % config_bookreader_host
which hardcoded archive.org as the host. They now fall back to the new config key.

smtp_port fix:
web.sendmail() reads web.config.smtp_port but infogami only copied smtp_server
to web.config. Originally patched in the infogami submodule, but that commit was
never pushed to internetarchive/infogami, which broke CI's submodule fetch. Moved
the fix to openlibrary/config.py's load_config() instead — no infogami changes
needed, and smtp_port now correctly forwards so mailpit receives email.

Naming note: this was briefly renamed to "ghostwriter" mid-PR; the team decided
MockServices was the better name, so a later commit reverts that rename rather than
rewriting history — the intermediate commits (including the rename itself) stay intact.

Testing

# Start dev environment — starting `web` also starts `mockservices` (depends_on)
OL_MOUNT_DIR=$(pwd) docker compose up -d web

# Verify mockservices is healthy
curl http://localhost:8090/health
# {"status":"ok","service":"ol-mockservices"}

# Browse captured email
open http://localhost:8025

# Run the e2e suite against the live mockservices container
docker compose run --rm home pytest docker/mockservices/tests/test_e2e.py -v
# 21 passed

Dockerfile build context bug: COPY requirements.txt/main.py/start.sh resolved against the build context (repo root, per compose.override.yaml), not the Dockerfile's own directory — silently pulled in the root requirements.txt (with its git+https deps) instead of the minimal mockservices one, breaking the build outright.
Healthcheck bug: the healthcheck shelled out to wget, which the Dockerfile purges immediately after using it to install mailpit — the container could never report healthy. Switched to python3 (always present in the base image).
xauthn contract bug: the mock read op from form data and returned a status field, but the real client (InternetArchiveAccount.xauth in accounts/model.py) sends op as a query param, the payload as JSON, and reads success — every op was effectively broken. redeem_otp also checked the wrong field (otp instead of password, which is what the client sends the OTP in), and issue_key/activate were missing entirely (breaks S3 key issuance and account activation flows). Verified all 7 ops (authenticate, info, issue_otp, redeem_otp, create, issue_key, activate) against the exact wire format the real client sends.

Added

GET /services/loans/loan/?action=changes&after_uid=&limit= — mocks IA's loan changes feed needed by feat(solr): near-realtime loan availability updater #12689's near-realtime loan availability updater for local testing (per this comment). Seeds 1000 borrow/return events on startup using real ia identifiers pulled from local Solr (falls back to a static pool if Solr is empty/unreachable, e.g. on a fresh dev environment), timestamps spread over the last 14 days to exercise catchup/backfill. A background loop then adds 50 events and evicts the oldest 50 every 60s to exercise ongoing steady-state polling.

Known limitation (pre-existing, not a regression)

The info op unconditionally succeeds for any email, which means InternetArchiveAccount.create()'s pre-check (cls.get(email=email)) always finds a "match" — so /account/create can never actually create a new account against this mock. This is inherited from the original /internal/fake/xauth stub (also unconditional success), not introduced by this PR. Worth a small follow-up to have the mock track a set of registered emails in memory so create-account flows are testable locally. Tracked for follow-up.

…e2e testing Adds docker/mockservices/ — a standalone FastAPI service that replaces the web.py inline stubs (/internal/fake/*) with real isolated endpoints, and mocks additional external services that previously hit archive.org in dev. Endpoints implemented: - POST /services/xauthn/ (was /internal/fake/xauth) - GET /services/s3auth/ (was /internal/fake/s3auth) - POST /services/loans/loan/ (was /internal/fake/loans) - POST /services/availability/ (was not mocked — pointed at archive.org) - GET /services/borrow/{id} (was hardcoded in lending.py) - POST /recaptcha/api/siteverify (was not mocked — pointed at recaptcha.net) - GET /fts/v1/search (available, but NOT wired as the default — be-api.us.archive.org works fine for local dev as-is, so search_endpoint keeps pointing there) - POST /paapi5/getItems (Amazon PA-API stub) mailpit runs alongside FastAPI in the same container: - SMTP on :1025 — receives email instead of dummy_sendmail stdout logging - Web UI on :8025 — browse captured mail at http://localhost:8025 openlibrary.yml updated to point all ia_* URLs at mockservices:8090. Closes #13044

… cover S3 loan URL - Remove the four /internal/fake/* stub classes from account.py (availability, loans, xauth, s3auth); they are now served by the mockservices container. - Add ia_s3_loan_url config key to lending.py so get_groundtruth_availability and s3_loan_api can be redirected to mockservices (previously they used S3_LOAN_URL % config_bookreader_host which hardcoded archive.org). - Wire ia_s3_loan_url in openlibrary.yml pointing at mockservices. - Forward smtp_port from openlibrary.yml to web.config in openlibrary/config.py's load_config() (previously only smtp_server was forwarded by infogami, so smtp_port: 1025 in openlibrary.yml was silently ignored and mailpit never received email). Done on the OL side rather than patching the infogami submodule, since that commit was never pushed to the infogami remote and broke CI's submodule fetch.

COPY requirements.txt/main.py/start.sh in the Dockerfile resolved against the build context (repo root, per compose.override.yaml), not the Dockerfile's own directory — this silently pulled in the root requirements.txt (with its git+https deps) instead of the minimal mockservices one, breaking the build. The healthcheck also called wget, which the Dockerfile purges right after using it to install mailpit, so the container never reported healthy. Switched the healthcheck to python3, always present in the base image.

xauthn read `op` from the form body and returned a `status` field, but the real client (InternetArchiveAccount.xauth) sends `op` as a query param, the payload as a JSON body, and reads `success` — so every op was effectively broken (op always empty, "Unknown op" for every call). redeem_otp also checked the wrong field name (`otp` instead of `password`, which is what the client actually sends the OTP in), and issue_key/activate were missing entirely, breaking S3 key issuance and account activation. Added python-multipart, needed for the (now correctly parsed) request bodies. Also adds a GET /services/loans/loan/?action=changes handler mocking IA's loan changes feed, needed by the near-realtime loan availability updater (#12689) for local testing. Seeds 1000 borrow/return events on startup using real `ia` identifiers from local Solr (falling back to a static pool if Solr is empty/unreachable), with timestamps spread over the last 14 days to exercise catchup/backfill. A background loop then adds 50 events and evicts the oldest 50 every 60s to exercise ongoing steady-state polling.

…iner Runs directly against mockservices over the webnet docker network (docker compose run --rm home pytest docker/mockservices/tests/test_e2e.py), exercising every endpoint with the exact wire format the real OL client code sends. Skips gracefully (not a hard failure) when mockservices isn't reachable, so it's safe to include in a full test run without Docker up.

Renames docker/mockservices/ -> docker/ghostwriter/, the compose service, container hostname in openlibrary.yml, and every internal reference. Dev-only — confirmed zero references in compose.yaml (the prod/staging base file); the service is defined entirely in compose.override.yaml, which never applies outside local dev.

mekarpeles · 2026-07-03T05:27:49Z

@RayBB I think this PR should be a big improvement in terms of creating a microservice sidecar that local Open Library can hit to safely test external dependencies like account creation and loans (which are currently difficult for developers to test).

mekarpeles · 2026-07-13T19:51:50Z

@cdrini; Mocker instead of Ghostwriter?

mekarpeles · 2026-07-16T04:24:28Z

Feel free to change to Mocker, for now, would like this functionality available to the team and we can open a follow-up PR. I'd like to unblock @benbdeitch who can benefit from the changes API that is in this service. Gave PR a 2 week grace period for review.

…es naming Team decided MockServices was the better name after all. Reverts the prior rename commit as a new commit rather than rewriting history -- the intermediate commits stay intact. This reverts commit 6c6daa4.

mekarpeles changed the title ~~feat(dev): scaffold mockservices FastAPI container (replaces /internal/fake/* stubs)~~ feat(dev): GhostWriter: mockservices FastAPI container (replaces /internal/fake/* stubs) Jul 2, 2026

mekarpeles force-pushed the 13044/mockservices-fastapi-container branch from cf1abe0 to a9f070c Compare July 3, 2026 04:52

mekarpeles force-pushed the 13044/mockservices-fastapi-container branch from a9f070c to 9252ac1 Compare July 3, 2026 05:08

mekarpeles marked this pull request as ready for review July 3, 2026 05:10

mekarpeles added 6 commits July 2, 2026 23:12

mekarpeles force-pushed the 13044/mockservices-fastapi-container branch from 9252ac1 to 6c6daa4 Compare July 3, 2026 05:20

mekarpeles changed the title ~~feat(dev): GhostWriter: mockservices FastAPI container (replaces /internal/fake/* stubs)~~ feat(dev): add ghostwriter — mocks IA/third-party services for local dev (replaces /internal/fake/* stubs) Jul 3, 2026

mekarpeles assigned RayBB Jul 3, 2026

github-project-automation Bot added this to Ray's Project Jul 3, 2026

github-project-automation Bot moved this to Waiting Review/Merge from Staff in Ray's Project Jul 3, 2026

mekarpeles assigned cdrini and unassigned RayBB Jul 3, 2026

mekarpeles added this to the Sprint 2026-07 milestone Jul 6, 2026

mekarpeles added the Priority: 2 Important, as time permits. [managed] label Jul 13, 2026

mekarpeles changed the title ~~feat(dev): add ghostwriter — mocks IA/third-party services for local dev (replaces /internal/fake/* stubs)~~ feat(dev): add mockservices — mocks IA/third-party services for local dev (replaces /internal/fake/* stubs) Jul 16, 2026

mekarpeles merged commit eca4ee4 into master Jul 16, 2026
8 checks passed

github-project-automation Bot moved this from Waiting Review/Merge from Staff to Done in Ray's Project Jul 16, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

feat(dev): add mockservices — mocks IA/third-party services for local dev (replaces /internal/fake/* stubs)#13045

feat(dev): add mockservices — mocks IA/third-party services for local dev (replaces /internal/fake/* stubs)#13045
mekarpeles merged 7 commits into
masterfrom
13044/mockservices-fastapi-container

mekarpeles commented Jun 25, 2026 •

edited

Loading

Uh oh!

mekarpeles commented Jul 2, 2026 •

edited

Loading

Uh oh!

openlibrary-bot commented Jul 3, 2026

Uh oh!

mekarpeles commented Jul 3, 2026

Uh oh!

mekarpeles commented Jul 13, 2026

Uh oh!

mekarpeles commented Jul 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Uh oh!

Conversation

mekarpeles commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Related

Uh oh!

mekarpeles commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openlibrary-bot commented Jul 3, 2026

Fixed

Added

Known limitation (pre-existing, not a regression)

Uh oh!

mekarpeles commented Jul 3, 2026

Uh oh!

mekarpeles commented Jul 13, 2026

Uh oh!

mekarpeles commented Jul 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mekarpeles commented Jun 25, 2026 •

edited

Loading

mekarpeles commented Jul 2, 2026 •

edited

Loading