Skip to content

feat: add 'obsidian' EntrySource to distinguish plugin-imported content - #1367

Open
01luyicheng wants to merge 1 commit into
khoj-ai:masterfrom
01luyicheng:feat/obsidian-entry-source
Open

feat: add 'obsidian' EntrySource to distinguish plugin-imported content#1367
01luyicheng wants to merge 1 commit into
khoj-ai:masterfrom
01luyicheng:feat/obsidian-entry-source

Conversation

@01luyicheng

Copy link
Copy Markdown

Summary

Adds a granular OBSIDIAN value to the EntrySource enum and threads a file_source parameter through the content indexing pipeline (indexerconfigure_contenttext_search.setup → content processors). When the Obsidian plugin client syncs content, entries are now persisted with file_source='obsidian' instead of the generic 'computer', enabling targeted delete/filter/query operations on Obsidian-imported content without affecting imports from the desktop client or other sources.

This re-submits the intent of #1243 (closed by the original author due to inactivity, with no technical objection), rebased onto the latest master and extended to address downstream consumers of EntrySource so the change is non-regressing.

Resolves #1239.

Changes

  • database/models/__init__.py: add EntrySource.OBSIDIAN = "obsidian".
  • text_to_entries.py: add file_source: str = None to the abstract process() signature.
  • 6 content processors (markdown/org/pdf/plaintext/image/docx): accept file_source and pass file_source or DbEntry.EntrySource.COMPUTER to update_embeddings().
  • notion_to_entries.py / github_to_entries.py: also accept file_source to keep the abstract signature consistent. They continue to use their dedicated EntrySource.NOTION / EntrySource.GITHUB and ignore the parameter — this prevents a silent TypeError in setup() that was previously swallowed by configure_content's try/except (GitHub/Notion indexing would silently fail).
  • routers/helpers.py: configure_content() accepts file_source: Optional[str] = None and forwards it to the 6 file-type text_search.setup() calls (GitHub/Notion blocks untouched — they use their own sources). Also exposes "obsidian" in enabled_content_sources so the backend reports it as a content source the user has indexed.
  • search_type/text_search.py: setup() accepts file_source and forwards it to process().
  • routers/api_content.py: indexer() derives file_source from the client arg (obsidianEntrySource.OBSIDIAN, else COMPUTER) and passes it through run_in_executor. map_config_to_object() gets an explicit OBSIDIAN branch returning the "Computer" sentinel (Obsidian has no dedicated config object, same as Computer) so delete_content_source("obsidian") routes through the file-objects + entries deletion path instead of raising ValueError. All downstream deletes use the original content_source="obsidian", so only Obsidian entries/file-objects are removed — Computer content is not affected.
  • routers/api_chat.py + database/adapters/__init__.py: get_file_filter, add_files_to_filter, and remove_files_from_filter now query both EntrySource.COMPUTER and EntrySource.OBSIDIAN files (set-unioned). Previously they only queried "computer", which would silently drop Obsidian-imported files from the chat file-filter list and erase existing Obsidian filter entries when users added/removed any filter entry (data-loss regression introduced by tagging Obsidian content with 'obsidian').

Backward compatibility

All existing call sites default to file_source or EntrySource.COMPUTER (or None → falls back to COMPUTER), preserving current behavior for the desktop client, manual ingestion, and server-side GitHub/Notion indexing. No database migration is needed — TextChoices is application-layer validation and the underlying CharField(max_length=30) already accommodates the "obsidian" value.

Validation

  • ruff check — passed
  • ruff format --check — passed (222 files already formatted)
  • python -m py_compile — all 15 modified files compile
  • AST verification confirms EntrySource.OBSIDIAN == "obsidian" and file_source parameter present in all expected signatures
  • Static scan of all 24 EntrySource reference sites confirms no call site requires updating for the new value

Out of scope (potential follow-ups)

These were intentionally left out to keep the PR focused; happy to send follow-up PRs if useful:

  • Unit tests covering the obsidian path (client=obsidian tagging, delete_content_source("obsidian"), file-filter preservation).
  • Frontend SyncedContent TypeScript type and disconnectContent handler do not yet include obsidian — runtime is unaffected (extra backend keys are ignored), but the Obsidian source won't have a disconnect button in the web settings UI. The Obsidian plugin self-manages its content via /api/content?client=obsidian, so this is cosmetic.
  • Optional micro-optimization: replace the two get_all_filenames_by_source calls with a single file_source__in=[COMPUTER, OBSIDIAN] query.

References

Add OBSIDIAN value to the EntrySource enum and thread a file_source
parameter through the indexing pipeline (indexer -> configure_content
-> text_search.setup -> content processors). When the Obsidian client
syncs content, entries are now persisted with file_source='obsidian'
instead of 'computer', enabling targeted delete/filter/query operations
on Obsidian-imported content without affecting other imports.

All processors (including Notion and GitHub, which use their own dedicated
sources) accept the new file_source parameter to keep the abstract base
class signature consistent and avoid silent TypeError in setup().

Chat file filter queries now include EntrySource.OBSIDIAN alongside
EntrySource.COMPUTER so Obsidian-synced files remain visible in the
chat file filter list, and existing file_filters are not silently
erased when users add/remove filter entries.

map_config_to_object now has an explicit OBSIDIAN branch (returns the
"Computer" sentinel, since Obsidian has no dedicated config object) so
delete_content_source routes obsidian through the file-objects + entries
deletion path and does not raise ValueError for the new source. The
obsidian source is also exposed in enabled_content_sources so the
frontend recognizes it as a content source the user can manage.

Existing call sites default to file_source or EntrySource.COMPUTER,
preserving current behavior for desktop client and manual ingestion.

Fixes khoj-ai#1239
Copilot AI review requested due to automatic review settings July 4, 2026 04:52

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@01luyicheng

Copy link
Copy Markdown
Author

Hi @debanjum, gentle ping on this PR when you have a moment. Happy to make any changes if needed. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add granular 'obsidian' EntrySource to distinguish plugin-imported content from other sources

2 participants