Add public REST search API for document discovery

[JSv4]

Summary

This PR adds a new public REST search API endpoint (/api/search/) that enables crawlers, AI agents, and lightweight integrations to search across public corpuses and documents without requiring GraphQL or MCP connections.

Key Changes

• New Search API Endpoint (GET /api/search/)

  • Accepts query parameters: q (required), corpus (optional), limit (optional, capped at MAX_SEARCH_RESULTS)
  • Returns JSON results with document/corpus metadata
  • Respects visibility permissions (only returns public corpuses/documents to anonymous users)

• Dual Search Strategies

  • Corpus-scoped search: Attempts semantic vector search first, falls back to text matching
  • Global search: Text-based search across all public corpus and document titles/descriptions

• Documentation Updates

  • Updated module docstring to mention the new search API
  • Added comprehensive REST API documentation to both llms.txt and llms-full.txt discovery endpoints
  • Includes usage examples and parameter descriptions

• SEO & Discovery Improvements

  • Enhanced structured data (schema.org) in frontend/index.html with SearchAction and SoftwareSourceCode types
  • Enabled search engine crawlers (Googlebot, Bingbot, DuckDuckBot) in Cloudflare OG worker for better indexing
  • Updated web app manifest with proper branding

• Comprehensive Test Coverage

  • Added 15 new test cases covering:
    • Query validation and error handling
    • Permission enforcement (public/private corpus filtering)
    • Corpus-scoped vs. global search behavior
    • Limit parameter validation and capping
    • Result structure validation

Implementation Details

• Search results are limited to prevent abuse (configurable via MAX_SEARCH_RESULTS constant)
• Results include type, slug, title, description (truncated to 200 chars), and optional similarity_score
• Vector search gracefully falls back to text search if embeddings are unavailable
• Uses Django's visible_to_user() queryset filtering to enforce access control
• Endpoint is GET-only and requires the q parameter; missing/empty queries return 400 errors

https://claude.ai/code/session_015NyrhCvpdCxtju7jhWwYmP

[claude]

Code Review

Overall this is a well-structured addition — clean endpoint design, good documentation, and a solid test suite foundation. A few issues worth addressing before merge:

---

Bug: Document visibility in tests and production logic

Test setup missing is_public=True on documents

BaseOCModel.is_public defaults to False. The visible_to_user(AnonymousUser()) path in BaseVisibilityManager (line 70 of Managers.py) filters strictly by is_public=True. In the test setup:

cls.public_doc = Document.objects.create(
    title="Sample Indemnity Agreement",
    creator=cls.owner,
    # is_public not set → defaults to False
)

So test_global_search_finds_public_documents and test_corpus_scoped_search are likely failing or passing vacuously. The fix is to set is_public=True on public_doc.

Deeper design concern: In _search_within_corpus and _search_global, the code filters with Document.objects.visible_to_user(AnonymousUser()), which requires is_public=True on the document itself. But the common pattern in this codebase is that documents inherit public visibility through their corpus membership — documents typically aren't independently marked public. If that assumption holds, corpus-scoped search will return empty results for most real deployments even when the corpus is public.

For corpus-scoped search, it would be more correct to skip the document-level visibility check (since the corpus access was already verified) and query documents directly via DocumentPath:

# Instead of:
Document.objects.visible_to_user(user).filter(id__in=corpus_doc_ids)
# Consider:
Document.objects.filter(id__in=corpus_doc_ids)

For the global search, filtering via document_paths__corpus_id__in=public_corpus_ids is the right pattern, but the preceding visible_to_user(user) call may prematurely exclude public corpus documents.

---

Performance: select_related() without arguments

In _search_global:

.select_related()  # No args = deep JOIN on ALL related objects
.distinct()
.order_by("-modified")[:remaining]

select_related() with no arguments traverses all FK/O2O relationships recursively, which is wasteful for a lightweight search endpoint. Either remove it (the DocumentManager applies sensible select_related("creator", "user_lock") via visible_to_user anyway) or specify only the fields actually needed (none in this case, since only slug, title, description are used from the results).

---

Security: No rate limiting

The MCP and llms endpoints use RATE_LIMIT_REQUESTS for abuse protection. The search API has no rate limiting at all, making it an attractive target for content harvesting or resource exhaustion — especially since corpus-scoped search triggers corpus.embed_text(query) which may call an external embedding service on every request.

Recommend at minimum adding a @cache_page decorator for common queries, and ideally adding IP-based rate limiting consistent with other discovery endpoints.

---

Overly broad exception handling in vector search

except (ValueError, TypeError, AttributeError):
    pass

AttributeError in particular is too broad — it would silently swallow programming errors like a missing method or misconfigured embedder. Consider narrowing to only the exceptions that embed_text or search_by_embedding are documented to raise, or at least log them at DEBUG level like the RuntimeError catch below does.

---

Minor: Import inside setUpTestData

@classmethod
def setUpTestData(cls):
    ...
    from opencontractserver.documents.models import DocumentPath  # inside method

Document is already imported at the top of the test file. DocumentPath should join it there.

---

Non-issues (just noting they look intentional)

• Always uses AnonymousUser(): The design choice to always treat search as anonymous is sensible for a crawler-facing endpoint, but worth a comment in the docstring since authenticated callers might be surprised.
• URL placement: api/search/ in the discovery app mounting at root resolves cleanly to /api/search/ with no conflicts against existing routes.
• MAX_SEARCH_RESULTS = 50 constant: Good use of the constants module.
• Structured data / manifest / SEO updates: Clean and well-formed.

[claude]

Code Review: Add public REST search API for document discovery

Overall this is well-structured and the feature is a useful addition. A few issues worth addressing before merge, including one that will cause test failures.

---

Bug: Test documents not visible to anonymous users

File: opencontractserver/discovery/tests/test_discovery_views.py

DocumentQuerySet.visible_to_user(AnonymousUser()) filters to is_public=True only (see shared/QuerySets.py:213-214). The test setup creates documents without is_public=True:

cls.public_doc = Document.objects.create(
    title="Sample Indemnity Agreement",
    description="Contains standard indemnification language",
    creator=cls.owner,
    # missing: is_public=True
)

This will cause test_global_search_finds_public_documents and test_corpus_scoped_search to fail — documents in a public corpus are not automatically public themselves. The fix is to add is_public=True to public_doc:

cls.public_doc = Document.objects.create(
    title="Sample Indemnity Agreement",
    description="Contains standard indemnification language",
    creator=cls.owner,
    is_public=True,
)

---

Performance: select_related() without arguments

File: opencontractserver/discovery/views.py, _search_global

matching_docs = list(
    Document.objects.visible_to_user(user)
    .filter(...)
    .select_related()   # <-- joins ALL related models
    .distinct()
    ...
)

select_related() without arguments traverses every foreign key relationship, which is very expensive. Since only doc.slug, doc.title, and doc.description are used in the result serialization, the select_related() call should simply be removed.

---

Code quality: Overly broad exception swallowing AttributeError

File: opencontractserver/discovery/views.py, _search_within_corpus

except (ValueError, TypeError, AttributeError):
    pass

Including AttributeError in the silent pass block means programming errors (e.g. embed_text returning an unexpected type, search_by_embedding not existing on the queryset) are silently swallowed and fall back to text search. These are not expected runtime conditions — they're bugs. Consider narrowing this to only expected failure cases, or at least logging them at DEBUG level the same way RuntimeError is handled.

---

Design concern: Always ignores the authenticated user

File: opencontractserver/discovery/views.py, search_api

anonymous = AnonymousUser()
...
return _search_within_corpus(query, corpus_slug, limit, anonymous)

The endpoint always uses AnonymousUser() regardless of whether the actual requester is authenticated. This means logged-in users can't search their own private corpuses via this endpoint. This may be intentional (it's a "public" search API), but the docstring only says "Only searches public corpuses visible to anonymous users" — it doesn't explain why authenticated sessions are ignored.

If this is intentional, it's worth making explicit in the docstring: "Note: this endpoint always behaves as an anonymous request regardless of session state."

If authenticated users should be able to search private content, replace anonymous = AnonymousUser() with request.user.

---

Abuse risk: No rate limiting

Other discovery endpoints use @cache_page to reduce load. The search endpoint is deliberately uncached (results vary per query), but it also has no rate limiting. Under load, it runs two database queries per request. Given it's publicly accessible, consider adding basic throttling (e.g. Django's ratelimit decorator or a simple cache-based counter), or at minimum document this as a known limitation.

---

Minor: _search_global corpus-first ordering silently starves document results

If limit=10 and 10 corpuses match, the document search is never run (remaining = 0). This is probably acceptable behavior, but it's worth a brief comment explaining the intentional priority ordering:

# Corpuses are returned first; documents fill remaining slots.

---

Missing: CHANGELOG update

Per CLAUDE.md, significant changes should be documented in CHANGELOG.md. Adding a new public API endpoint qualifies.

---

Summary

Issue	Severity
Test documents not is_public=True → 2 tests will fail	High
select_related() without arguments	Medium
AttributeError silently swallowed	Medium
Authenticated user always treated as anonymous	Low / design question
No rate limiting	Low
Missing CHANGELOG entry	Low
Document corpus-first ordering	Nit

[claude]

PR Review: Add public REST search API for document discovery

Overall this is a well-structured PR with good test coverage and a clear purpose. The domain rename is clean and consistent. A few issues worth addressing before merge.

---

Critical: Traefik routing gap

The web-secure-router in traefik.yml only forwards /graphql, /admin, /mcp, /sse, /robots.txt, /llms.txt, /llms-full.txt, /sitemap.xml, /.well-known to Django. /api/ is not listed, so the frontend-secure-router (priority 1) matches /api/search/ first and sends it to the frontend static file server. The endpoint will 404 in production. The web-secure-router rule needs PathPrefix for /api/ added alongside /graphql, /admin, etc. Worth also verifying api/health/ works in production.

---

Performance: select_related() without arguments in _search_global

select_related() with no arguments JOINs every FK/OneToOne on Document. None of those related fields appear in the result dict. Remove it or replace with .only() scoped to the needed fields.

---

Code quality: Over-broad exception swallowing in _search_within_corpus

Catching AttributeError silently swallows programming errors like renamed methods or missing attributes. Narrow to only the expected exceptions or add a logger.debug() call. The RuntimeError branch already logs properly; the bare pass is inconsistent.

---

Design: Global search exhausts limit with corpuses before documents

If the corpus query returns the full limit, remaining = 0 and the document query is skipped entirely. A search matching many corpuses will silently suppress all document results. Consider a fixed split (e.g. limit // 2 per type) or interleaving by a common ranking signal.

---

Test gaps

test_limit_capped_at_max only asserts status_code == 200. It should also verify the result count is bounded by MAX_SEARCH_RESULTS. test_result_structure checks type/slug/title but not description or similarity_score, both documented in llms-full.txt as part of the API contract.

---

Minor observations

• The search_api docstring should note that all requests are evaluated as anonymous regardless of authentication (intentional public-only search, but non-obvious to future maintainers).
• similarity_score is documented as always present but is None for text-search and global results. A note like 'null when vector search was not used' would clarify the contract for API consumers.
• Migration dependency on sites 0004_alter_options_ordering_domain looks correct.

---

What is good

• Consistent use of visible_to_user(user) for all permission checks — no IDOR risk.
• MAX_SEARCH_RESULTS constant correctly placed in constants/discovery.py (no magic numbers).
• Vector search to text fallback is a clean degradation strategy.
• 15 test cases cover the main happy/sad paths thoroughly.
• Permission enforcement tests are solid.
• Structured data improvements in index.html use proper @graph for multiple entity types.

[claude]

PR Review: Add public REST search API for document discovery

Overall this is a clean, well-structured addition. The domain rename is consistent and correct throughout. The search API is well-documented with solid test coverage. A few issues worth addressing before merge:

Bug: _search_global can exceed the requested limit

In _search_global the doc_limit math can return more results than limit when the limit is small. With limit=1: corpus_limit=max(0,1)=1, doc_limit=max(0,1)=1. After corpus takes its slot, remaining=0, but doc_limit=max(0,1)=1 — so up to 2 results for limit=1. The test_limit_parameter test likely passes only because ?q=indemnification does not match 'Sample Indemnity Agreement' (different word), but the logic is wrong. Fix: use remaining directly as the doc slice limit.

Security: No rate limiting on the new endpoint

Every other crawler-facing endpoint references RATE_LIMIT_REQUESTS, but search_api has none. A search endpoint triggering DB queries and potentially ML embedding lookups is a more attractive DoS target than static text files. Consider applying the same rate-limit middleware used for /mcp/, or at minimum a @cache_page decorator for identical queries.

Exception handling in vector search is too narrow

The except clauses only catch ValueError, TypeError, and RuntimeError. corpus.embed_text() and Document.objects.search_by_embedding() can raise pgvector/psycopg errors, AttributeError if no embedder is configured, etc. An unhandled exception here returns a 500 to the caller. A broader except Exception as exc with a debug log is safer for a public-facing fallback path.

Telemetry helper not guarded against failures

_record_discovery_event is now injected into robots.txt, sitemap.xml, llms.txt, etc. If PostHog is misconfigured and record_event raises, it breaks those endpoints. Wrapping the body in try/except Exception: pass would prevent telemetry failures from cascading.

Minor: public_corpus_ids second queryset worth a comment

In _search_global a fresh Corpus.objects.visible_to_user(user) queryset is used for the document filter — intentionally fetching all public corpus IDs, not just title-matched ones. This is correct but reads as a potential oversight. A one-line comment ('re-query to get all public corpus IDs, not just title-matched ones') would make intent clear.

---

What looks good:

• AnonymousUser() used correctly for public-only enforcement, documented in docstring
• MAX_SEARCH_RESULTS constant used (no magic numbers)
• @require_GET + 405 for non-GET methods is correct
• 15 test cases covering error paths, permission filtering, corpus-scoped vs global, and response structure
• Structured data @graph upgrade in index.html is well-formed and meaningfully improves discoverability
• Domain rename consistent across all config files, migrations, docs, and tests
• New migration has correct forward/backward functions

[claude]

PR Review summary - 8 items found. See below for details.

[claude]

PR Review: Add public REST search API for document discovery

Overall this is well-structured and the test coverage is solid. The domain rename is comprehensive. A few issues worth addressing before merge:

---

Bug 1 - Vector search early return bypasses text fallback

In _search_within_corpus, if embed_text succeeds and query_vector is truthy, the function returns immediately after the vector search even when doc_results is empty. Text fallback is only reached on exceptions, not on empty vector results. A corpus with embeddings but no matching documents returns empty even when a text match exists. Fix: only return early if results is non-empty, then fall through to text search.

Bug 2 - Total results can exceed limit in _search_global

With limit=1: corpus_limit=1, doc_limit=max(0,1)=1. After 1 corpus result, remaining=0 and doc_limit=max(0,1)=1 - up to 2 items returned for a limit=1 request. Consider adding results = results[:limit] before returning, and documenting the actual guarantee.

Bug 3 - Migration dependency may be incorrect in 0005_update_site_domain.py

The dependency on sites/0004_alter_options_ordering_domain refers to the django.contrib.sites built-in migration, not the previous migration in this app. Verify the app label and add a self-referential dependency to guarantee ordering within the custom app migrations.

Perf 4 - visible_to_user evaluated twice in _search_global

Once for corpus text-search, again to build public_corpus_ids. Both round-trips hit the database. Either reuse the IDs from the first list() call, or pass the queryset without list() to keep it as a SQL subquery.

Perf 5 - Missing rate limiting on a public unauthenticated endpoint

Every other discovery endpoint is cache-paged or rate-limited. MAX_SEARCH_RESULTS caps response size but not request frequency - this endpoint can enumerate all public content cheaply. Consider IP-level rate limiting (the MCP endpoint has a working pattern) or a short @cache_page for repeated identical queries.

Perf 6 - Overly narrow exception handling in vector search path

Only ValueError, TypeError, and RuntimeError are caught. ORM/connection errors and pgvector-specific exceptions will surface as 500s on a public endpoint. Add a broad except Exception as a final catch to ensure text fallback is always attempted.

Minor 7 - _record_discovery_event fires before validation succeeds

The telemetry call happens before the corpus lookup, so 404 responses from private/missing corpuses are counted as successful hits. Move the call to after results are determined.

Minor 8 - Inline import in setUpTestData

DocumentPath is imported inside the method body. Move it to the top-level imports alongside Document.

---

What is Good

• Permission enforcement is correct: visible_to_user(AnonymousUser()) applied consistently at every data access point - no IDOR risk.
• .only() deferred-load in _search_global is a good performance touch.
• 15 test cases with solid edge-case coverage: private corpus 404, limit capping, empty queries, method enforcement.
• Schema.org @graph restructure is clean and the SearchAction registration improves crawlability.
• Domain rename is comprehensive and the migration is correctly reversible via RunPython.

[claude]

test

[claude]

Code Review

Overall this is a well-structured PR. The domain rename is clean and consistent, the new search endpoint is useful, and test coverage is solid. A few issues worth addressing before merge.

---

Breaking Change: AUTH0_ADMIN_CLAIM_NAMESPACE default

The change to AUTH0_ADMIN_CLAIM_NAMESPACE in config/settings/base.py and config/graphql_auth0_auth/utils.py silently breaks existing production deployments that rely on the default value. Any deployment that uses Auth0 without an explicit AUTH0_ADMIN_CLAIM_NAMESPACE env var will have admin users who can no longer log in as staff/superuser after migration. This should be called out in the CHANGELOG with a migration note. The fix is straightforward (update the Auth0 Action), but it needs to be communicated.

---

Missing Rate Limiting on Search Endpoint

/api/search/ has no rate limiting, while other discovery endpoints use RATE_LIMIT_REQUESTS. This endpoint is more expensive than a static text response — it can hit both the DB and an external embedding service per request. A malicious actor could trivially trigger expensive vector search calls in a tight loop. Consider applying the same rate limiting used by other discovery endpoints.

---

Telemetry Fires Before Corpus Validation

In search_api(), _record_discovery_event("search_api", request) fires before the corpus lookup. A request with an invalid or private corpus slug will be counted as a discovery hit even though it returns a 404. This inflates the metric with noise. Consider moving the telemetry call to after successful corpus validation, or document that "attempted searches" are intentionally counted.

---

Broad except Exception on Vector Search

The except Exception block silently catches programming errors (AttributeError, TypeError, etc.) that indicate bugs, not just transient infrastructure issues. These will appear only as debug log entries. Consider catching specific expected exceptions (e.g., embedding-related errors), or at minimum logging at logger.warning so they surface in production monitoring.

---

corpus_doc_ids QuerySet Re-evaluated Twice

In _search_within_corpus, corpus_doc_ids is a lazy QuerySet used in both the vector path and the text fallback. If both branches execute (vector search returns empty results, or vector search fails), the underlying DB query runs twice. Materializing it once with list() avoids the double hit.

---

_search_within_corpus Missing .only()

_search_global uses .only("slug", "title", "description", "modified") to avoid fetching full document objects, but the text search fallback in _search_within_corpus fetches full objects without this optimization. Should add .only() for consistency and performance.

---

Migration Dependency Could Be Clearer

0005_update_site_domain.py only lists ("sites", "0004_alter_options_ordering_domain") (the built-in django.contrib.sites migration) as a dependency. Django infers the local app's ordering from the filename prefix, but being explicit about the previous local migration as a dependency would help future maintainers understand the ordering intent.

---

Missing CHANGELOG Update

Per CLAUDE.md, significant changes require a CHANGELOG.md entry. This PR introduces a new public API endpoint and a domain rename that affects Auth0 configuration for existing deployments — both warrant changelog entries.

---

Minor: test_result_structure Missing Field Assertions

The test verifies type, slug, and title but not description or similarity_score, which are part of the documented response schema. Worth adding assertions for those fields.

---

Minor: test_limit_capped_at_max Assertion Ordering

The test asserts assertLessEqual(len(data["results"]), MAX_SEARCH_RESULTS) before assertIsInstance(data["results"], list). If the response were malformed, the length check would raise an unhelpful error. The type check should come first.

---

What's Working Well

• Clean, consistent domain rename across all config, docs, and tests
• _search_global floor-split for corpus/doc slots ensures neither type starves the other
• Private corpus/document visibility correctly delegated to visible_to_user() — no IDOR risk
• Schema.org @graph structure is correct (multiple nodes with @id cross-references)
• manifest.json fix was long overdue
• _record_discovery_event added consistently to all existing discovery endpoints
• 15 test cases covering main happy paths and edge cases

[codecov]

Codecov Report

❌ Patch coverage is 96.12903% with 6 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
opencontractserver/discovery/views.py	90.76%	6 Missing ⚠️

📢 Thoughts on this report? Let us know!