feat: add 'obsidian' EntrySource to distinguish plugin-imported content - #1367
Open
01luyicheng wants to merge 1 commit into
Open
feat: add 'obsidian' EntrySource to distinguish plugin-imported content#136701luyicheng wants to merge 1 commit into
01luyicheng wants to merge 1 commit into
Conversation
Add OBSIDIAN value to the EntrySource enum and thread a file_source parameter through the indexing pipeline (indexer -> configure_content -> text_search.setup -> content processors). When the Obsidian client syncs content, entries are now persisted with file_source='obsidian' instead of 'computer', enabling targeted delete/filter/query operations on Obsidian-imported content without affecting other imports. All processors (including Notion and GitHub, which use their own dedicated sources) accept the new file_source parameter to keep the abstract base class signature consistent and avoid silent TypeError in setup(). Chat file filter queries now include EntrySource.OBSIDIAN alongside EntrySource.COMPUTER so Obsidian-synced files remain visible in the chat file filter list, and existing file_filters are not silently erased when users add/remove filter entries. map_config_to_object now has an explicit OBSIDIAN branch (returns the "Computer" sentinel, since Obsidian has no dedicated config object) so delete_content_source routes obsidian through the file-objects + entries deletion path and does not raise ValueError for the new source. The obsidian source is also exposed in enabled_content_sources so the frontend recognizes it as a content source the user can manage. Existing call sites default to file_source or EntrySource.COMPUTER, preserving current behavior for desktop client and manual ingestion. Fixes khoj-ai#1239
Author
|
Hi @debanjum, gentle ping on this PR when you have a moment. Happy to make any changes if needed. Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a granular
OBSIDIANvalue to theEntrySourceenum and threads afile_sourceparameter through the content indexing pipeline (indexer→configure_content→text_search.setup→ content processors). When the Obsidian plugin client syncs content, entries are now persisted withfile_source='obsidian'instead of the generic'computer', enabling targeted delete/filter/query operations on Obsidian-imported content without affecting imports from the desktop client or other sources.This re-submits the intent of #1243 (closed by the original author due to inactivity, with no technical objection), rebased onto the latest
masterand extended to address downstream consumers ofEntrySourceso the change is non-regressing.Resolves #1239.
Changes
database/models/__init__.py: addEntrySource.OBSIDIAN = "obsidian".text_to_entries.py: addfile_source: str = Noneto the abstractprocess()signature.file_sourceand passfile_source or DbEntry.EntrySource.COMPUTERtoupdate_embeddings().notion_to_entries.py/github_to_entries.py: also acceptfile_sourceto keep the abstract signature consistent. They continue to use their dedicatedEntrySource.NOTION/EntrySource.GITHUBand ignore the parameter — this prevents a silentTypeErrorinsetup()that was previously swallowed byconfigure_content's try/except (GitHub/Notion indexing would silently fail).routers/helpers.py:configure_content()acceptsfile_source: Optional[str] = Noneand forwards it to the 6 file-typetext_search.setup()calls (GitHub/Notion blocks untouched — they use their own sources). Also exposes"obsidian"inenabled_content_sourcesso the backend reports it as a content source the user has indexed.search_type/text_search.py:setup()acceptsfile_sourceand forwards it toprocess().routers/api_content.py:indexer()derivesfile_sourcefrom theclientarg (obsidian→EntrySource.OBSIDIAN, elseCOMPUTER) and passes it throughrun_in_executor.map_config_to_object()gets an explicitOBSIDIANbranch returning the"Computer"sentinel (Obsidian has no dedicated config object, same as Computer) sodelete_content_source("obsidian")routes through the file-objects + entries deletion path instead of raisingValueError. All downstream deletes use the originalcontent_source="obsidian", so only Obsidian entries/file-objects are removed — Computer content is not affected.routers/api_chat.py+database/adapters/__init__.py:get_file_filter,add_files_to_filter, andremove_files_from_filternow query bothEntrySource.COMPUTERandEntrySource.OBSIDIANfiles (set-unioned). Previously they only queried"computer", which would silently drop Obsidian-imported files from the chat file-filter list and erase existing Obsidian filter entries when users added/removed any filter entry (data-loss regression introduced by tagging Obsidian content with'obsidian').Backward compatibility
All existing call sites default to
file_source or EntrySource.COMPUTER(orNone→ falls back toCOMPUTER), preserving current behavior for the desktop client, manual ingestion, and server-side GitHub/Notion indexing. No database migration is needed —TextChoicesis application-layer validation and the underlyingCharField(max_length=30)already accommodates the"obsidian"value.Validation
ruff check— passedruff format --check— passed (222 files already formatted)python -m py_compile— all 15 modified files compileEntrySource.OBSIDIAN == "obsidian"andfile_sourceparameter present in all expected signaturesEntrySourcereference sites confirms no call site requires updating for the new valueOut of scope (potential follow-ups)
These were intentionally left out to keep the PR focused; happy to send follow-up PRs if useful:
client=obsidiantagging,delete_content_source("obsidian"), file-filter preservation).SyncedContentTypeScript type anddisconnectContenthandler do not yet includeobsidian— runtime is unaffected (extra backend keys are ignored), but the Obsidian source won't have a disconnect button in the web settings UI. The Obsidian plugin self-manages its content via/api/content?client=obsidian, so this is cosmetic.get_all_filenames_by_sourcecalls with a singlefile_source__in=[COMPUTER, OBSIDIAN]query.References