OPDS feed silently drops books that have no cover on Open Library

[mekarpeles]

Found while verifying #200 against production on 2026-08-29.

Symptom

metadata.numberOfItems does not match the number of publications the feed returns:

GET /v1/api/opds?limit=200          -> numberOfItems: 96,  publications: 95
GET /v1/api/opds?modified_since=…   -> numberOfItems: 49,  publications: 48

Consistently one short, across filters.

The real problem

The count is only how this surfaced. The actual issue is that a book in the library's collection is absent from its own catalogue — it cannot be discovered, and cannot be borrowed by any OPDS client.

Cause

Two different searches run per feed build, and only one of them filters:

1. LennyAPI._enrich_items → OpenLibrary.search(...) — Lenny's own class, no filtering.
2. LennyAPI.opds_feed → LennyDataProvider.search(...) → OpenLibraryDataProvider.search(...), whose signature is require_cover: bool = True.

With require_cover=True, pyopds2_openlibrary drops records that lack a cover image or a description:

if require_cover:
    records = [r for r in records if _has_acquisition_options(r) and _has_cover(r)]
else:
    records = [r for r in records if _has_acquisition_options(r)]

That default is right for Open Library's own homepage carousels — it avoids broken "Cover Unavailable" cards. It is wrong for Lenny, where the feed is not a curated shelf but the complete inventory of what this library lends. A missing cover is not a reason to hide a book the library owns.

metadata.numberOfItems comes from Item.count() — a straight DB count, correctly unaware of any of this. Hence the mismatch.

Suggested fix

Pass require_cover=False from LennyAPI.opds_feed (and search_feed, which has the same issue). _has_acquisition_options filtering should stay — a record with no way to acquire it genuinely cannot be represented in OPDS.

If some items legitimately cannot be rendered, numberOfItems should be reconciled with what is actually emitted rather than left as a raw DB count, so a harvester is not waiting on a record that will never arrive.

Note on #200

#200 changed numberOfItems from len(publications) (the current page's size) to Item.count() (the whole matching set). The new value is far more useful — a harvester needs the total, not the page size — but it is what made this pre-existing gap visible. The dropped book was already missing before #200; nothing reported it.

Reproducing

BASE=https://lennyforlibraries.org/v1/api
curl -s "$BASE/opds?limit=200" | jq '.metadata.numberOfItems, (.publications|length)'

To identify the missing item, diff the openlibrary_edition values in the items table against the self link ids in the feed.

[openlibrary-bot]

Correction: the symptom is real, the diagnosed cause is not

Re-verified against production today (2026-09-04) on 0.2.16, after #200, #204, #195 and pyopds2_lenny#31. The symptom is unchanged and still live — but the cause in the issue body is wrong, and the suggested fix would not compile. Correcting the record.

The cause named here does not exist in the dependency Lenny runs

require_cover: bool = True is real, but it lives only on pyopds2_openlibrary main (added 2026-04-15). requirements.txt:52 pins e18e79f, dated 2026-03-03, which is not an ancestor of main — it sits on a side branch. Introspecting the installed package:

OpenLibraryDataProvider.search signature: (query, limit=50, offset=0, sort=None, facets=None)
has _has_cover: False

There is no require_cover parameter and no _has_cover function in the build Lenny runs. The only filter is _has_acquisition_options, and it applies solely when facets['mode'] == 'ebooks'; Lenny passes no facets, so mode is everything and nothing is filtered client-side at all.

So require_cover=False would raise TypeError: unexpected keyword argument. The suggested fix cannot be applied as written.

Also worth noting: #195 has since removed the two-search structure the "Cause" section describes. There is now exactly one Open Library search per feed build. The mismatch survived that change, which is itself evidence the diagnosis was wrong — removing one of the "two different searches" changed nothing.

What is actually happening

The missing record is dropped server-side, by Open Library's own search, not by any client-side filter. Localised precisely by bisecting on modified_since (numberOfItems comes from Item.count(), the publications come from Item.get_many(), so narrowing the window until the gap vanishes isolates the row):

no filter                                   -> numberOfItems=96  emitted=95  gap=1
modified_since=2026-08-28T04:28:31.023644Z  -> numberOfItems=38  emitted=37  gap=1
modified_since=2026-08-28T04:28:34.037774Z  -> numberOfItems=36  emitted=36  gap=0

Exactly one items row has updated_at between 2026-08-28T04:28:31.023644Z and 2026-08-28T04:28:34.037774Z and produces no publication. It sits between "Heathen Earth" (28354744) and "Jerusalem commands" (28153865) in import order.

It is also absent from /v1/api/items, which takes the separate get_enriched_items path. Two independent code paths, both querying Open Library, both losing the same row — so the row is lost upstream, in what OL returns for edition_key:(...), not in anything Lenny does with the response.

Ruled out along the way:

• Work-collapse (two owned editions of one work returning a single doc): checked every local record's work key — 0 works hold more than one local edition. Not this.
• Mis-keying (pyopds2_lenny#31): all 95 emitted publications have unique edition ids, and each one's title matches the edition its own borrow link points at, verified by fetching all 95 editions directly from openlibrary.org/books/OL{id}M.json — 0 mismatches.

To identify the book

One query on the Lenny host settles it:

SELECT id, openlibrary_edition, created_at, updated_at
FROM items
WHERE updated_at >  '2026-08-28T04:28:31.023644Z'
  AND updated_at <  '2026-08-28T04:28:34.037774Z';

Then check that edition against Open Library directly — the likely explanations are an edition OL's search does not index (a redirect, or a record created after the last index run), or an edition key that no longer resolves.

What still stands

The framing under "The real problem" is correct and unaffected: a book the library owns is absent from its own catalogue and cannot be borrowed by any OPDS client. That remains true today.

The closing suggestion — reconcile numberOfItems with what is actually emitted, rather than leaving it a raw DB count — also still stands, and matters more than when this was filed, because a harvester now consumes this feed. See the separate issue filed alongside this comment for a far more severe instance of the same defect class.

Suggested retitle: "OPDS feed emits one fewer publication than it counts (a book the library owns is unborrowable)" — the current title asserts the disproven cause.