feat(retrievers): add Caesar web search retriever

[TF-Caesar]

What

Adds Caesar as a new web-search retriever, selectable via RETRIEVER=caesar, alongside the existing providers (Tavily, Exa, Serper, Brave, fastCRW, and others).

Caesar is an agentic web-search API. The retriever POSTs to its /v1/search endpoint and maps results into GPT Researcher's standard {href, body} shape.

Disclosure: I work on GTM for Caesar, so this is an affiliated contribution.

Why

Caesar returns the passages it selected for your query inline with each search result, so the body of each {href, body} carries the text that addresses the query rather than the page's marketing blurb.

Its snippet field is the page's meta description, identical for every query that surfaces the document. Its passages are query-conditioned. Same document, two different queries, zero overlapping passages:

tokio.rs/blog/2019-10-scheduler

  q: "work stealing balancing load across worker threads"
     -> "That notified processor will steal half the tasks in the batch, and in turn notify..."

  q: "atomic reference counting overhead in the old scheduler"
     -> "There are many outstanding references to the task structure: the scheduler and each waker..."

So the retriever builds body from the passages and falls back to the flat fields when a result has none. For a research agent that reads body before deciding what to scrape, that is the difference between a meta description and roughly 1,200 characters of relevant text per source.

An honest note on history: when I opened this PR the rationale was that Caesar was keyless. Caesar removed the anonymous tier on 2026-07-07, so that rationale is gone and a CAESAR_API_KEY is required, created from an account at https://app.trycaesar.com. What replaced it is the argument above, which I think is the better one.

Changes

• Add the CaesarSearch retriever under gpt_researcher/retrievers/caesar/.
• Register caesar in the retriever factory (actions/retriever.py) and in VALID_RETRIEVERS (retrievers/utils.py), and export it from retrievers/__init__.py.
• Document RETRIEVER=caesar in the search-engines guide.
• Add tests/test_caesar_retriever.py.

All changes are additive. No existing retriever is modified.

Implementation notes

• The key requirement mirrors the exa retriever: get_api_key raises a clear error naming CAESAR_API_KEY and pointing at where to obtain it. The Authorization: Bearer header is always sent.
• body prefers passages, then falls back across snippet, content, and passage. href falls back across url, canonical_url, and source_url. Results with no URL are dropped, mirroring how brave normalizes.
• query_domains is now honored. It maps to Caesar's source_policy.include_domains. The earlier revision of this PR accepted query_domains and silently ignored it, so a domain-scoped report was quietly searching the whole web. exa and tavily both pass include_domains, and now so does this.
• CAESAR_API_URL (or a caesar_api_url header) can override the base URL, following the same override pattern crw uses.

One tradeoff, stated plainly: preferring passages makes each body roughly 4x larger than a meta description, which costs more tokens up front. For this project I think that is the right trade, since a better body is exactly what improves source selection before scraping. It is one method (_body) if you would rather keep bodies short.

Testing

tests/test_caesar_retriever.py covers: a missing CAESAR_API_KEY raises, a present key produces the Bearer header, a header-supplied key is honored, passage preference over the meta description, blank-passage handling, the field-name fallbacks, query_domains producing a source_policy, its absence omitting one, and the empty-on-error path.

$ python -m unittest tests.test_caesar_retriever
Ran 9 tests in 0.004s

OK

Live, against a keyed account (the keyless transcript in the original description no longer reproduces):

$ python -c "...CaesarSearch('why did tokio avoid the atomic increment in wake_by_ref').search(max_results=3)"
 1208 chars  https://tokio.rs/blog/2019-10-scheduler
  965 chars  https://docs.rs/tokio/0.2.9/src/tokio/task/waker.rs.html
   33 chars  https://github.com/tokio-rs/tokio/issues/1388
body answers the question: True

And domain scoping, which now actually scopes:

query_domains=["tokio.rs"]  -> hosts: ['tokio.rs']
unscoped                    -> hosts: ['en.wikipedia.org', 'github.com', 'hackage.haskell.org', ...]

Happy to adjust naming or anything else to fit your conventions.

[TF-Caesar]

Correction, and a pushed fix.

Caesar began requiring an API key on every request on 2026-07-07. When I opened this PR, the API served anonymous traffic, and "keyless" was the whole reason I proposed it. That rationale is now gone, and I would rather say so plainly than leave a stale claim in the description.

Pushed to this branch:

• get_api_key now raises when CAESAR_API_KEY is unset, mirroring the exa retriever, and the Bearer header is always sent.
• The module docstring, the search-engines guide entry, and the tests no longer describe anonymous access.
• The PR description is rewritten.

This is now just one more key-required provider rather than a keyless option. If you would rather not carry another one, say the word and I will close this. If you do want it, I will re-run a live query against a keyed account and post the output before it merges.

Sorry for the churn.

[TF-Caesar]

Re-verified against a keyed account, as promised.

With CAESAR_API_KEY set:

>>> from gpt_researcher.retrievers.caesar.caesar import CaesarSearch
>>> CaesarSearch("rust async runtime comparison").search(max_results=5)

5 results, each mapped to {"href": ..., "body": ...}
  https://www.matterai.so/guides/rust-networking-tokio-vs-async-std-vs-smol-for-async-network-programming
  https://lucaberton.com/blog/rust-async-runtimes-tokio-2026
  https://rustify.rs/articles/rust-async-runtimes-tokio-vs-async-std-2026
  https://lobste.rs/s/jkct2m/avoid_async_rust
  https://corrode.dev/blog/async

With CAESAR_API_KEY unset, construction raises, mirroring the exa retriever:

Exception: Caesar API key not found. Please set the CAESAR_API_KEY environment variable.
You can obtain your key from https://app.trycaesar.com

That closes the testing note in the description. The offer still stands: if you would rather not carry another key-required provider, say the word and I will close this.

[TF-Caesar]

Update, and I would like to withdraw the offer to close this.

Above, I said that once Caesar stopped being keyless this was "simply one more key-required provider." I had not actually read its search response carefully. Having done that, I no longer think that is true, and I found two things wrong with my own retriever. Both are fixed in 8bc8ae3.

The retriever was discarding the useful half of the response. Caesar returns a passages array per result, and those passages are selected for the query rather than being a fixed chunking of the page. Same document, two queries, zero overlapping passages:

tokio.rs/blog/2019-10-scheduler

  q: "work stealing balancing load across worker threads"
     -> "That notified processor will steal half the tasks in the batch, and in turn notify..."

  q: "atomic reference counting overhead in the old scheduler"
     -> "There are many outstanding references to the task structure: the scheduler and each waker..."

The snippet I was mapping into body is the page's meta description, the same text for every query. So body now comes from the passages, falling back to the flat fields when a result has none. On a live query, the top source's body goes from a blurb to 1,208 characters that answer the question. Since body is what the agent reads when deciding which sources to scrape, this is the part that matters here.

query_domains was accepted and silently ignored. exa and tavily both pass include_domains to their backends. My retriever stored self.query_domains and never sent it, so a domain-scoped report was searching the whole web. It now maps to Caesar's source_policy.include_domains:

query_domains=["tokio.rs"]  -> hosts: ['tokio.rs']
unscoped                    -> hosts: ['en.wikipedia.org', 'github.com', 'hackage.haskell.org', ...]

That was a real bug, and it would have shipped quietly.

Tests are up to 9 and all pass, including passage preference, blank passages, and both query_domains cases. The PR description is updated to match.

If you still would rather not carry another keyed provider, that is a completely fair call and I will close it myself. But I did not want the decision made against the weaker version of the integration.

[assafelovic]

Reviewed — the patch is well-formed (tests, docs, error handling), but it no longer applies to main after the retriever hardening that landed in #2074. Needs a rebase before it can be judged on a green run.

CI now runs on every PR, so a rebase gives you an immediate answer.

---

Why this is not merged yet, and it isn't code quality.

main already ships 22 retrievers. Six open PRs (#1814, #1850, #1867, #2015, #2048, #2068) would add seven more, and most come from the vendor whose API is being added. Each one is a few hundred lines the project maintains indefinitely, behind an API key CI cannot hold, against a service whose uptime and response shape nobody here controls — and the recently-merged work included roughly twenty separate fixes for retrievers returning payload shapes their parsers did not expect.

That is a scope decision for @assafelovic, not a verdict on this patch. The sustainable answer is probably a documented plugin path — an entry-point group so a vendor ships and versions their own package — rather than accepting or rejecting each of these case by case, which is exactly why they have all been sitting.

Not closing it. Flagging it so the decision gets made once instead of six times.