Skip to content

fix: strip date filters at the start of a query - #1396

Open
LHMQ878 wants to merge 1 commit into
khoj-ai:masterfrom
LHMQ878:fix/date-filter-at-start-of-query
Open

fix: strip date filters at the start of a query#1396
LHMQ878 wants to merge 1 commit into
khoj-ai:masterfrom
LHMQ878:fix/date-filter-at-start-of-query

Conversation

@LHMQ878

@LHMQ878 LHMQ878 commented Aug 4, 2026

Copy link
Copy Markdown

Summary

Fixes #1395.

DateFilter.defilter required whitespace before the date filter, so a dt filter at the start of a query was never stripped. Because routers/helpers.py recovers the filters by subtracting the defiltered query from the original, that left filters_in_query empty and the date range never reached the database query — while the raw dt>="..." text stayed in the question sent to the LLM and the embedding model.

Root cause

    def defilter(self, query):
        # remove date range filter from query
        query = re.sub(rf"\s+{self.date_regex}", " ", query)
        query = re.sub(r"\s{2,}", " ", query).strip()  # remove multiple spaces
        return query

A filter at position 0 has nothing before it, so \s+ can't match. get_filter_terms uses the bare date_regex with no such requirement, so the two methods disagree about whether a filter is present. Measured on master (ae229ca):

query get_filter_terms defilter
head dt:"1984-01-01" tail ["dt:'1984-01-01'"] 'head tail'
dt:"1984-01-01" tail ["dt:'1984-01-01'"] 'dt:"1984-01-01" tail'
dt:"1984-01-01" ["dt:'1984-01-01'"] 'dt:"1984-01-01"'
head,dt:"1984-01-01" ["dt:'1984-01-01'"] 'head,dt:"1984-01-01"'
head (dt:"1984-01-01") ["dt:'1984-01-01'"] 'head (dt:"1984-01-01")'

Any non-whitespace character before the filter suppresses the removal, so start-of-query, ( and , are all affected. Two adjacent filters hit it too: the first one's trailing space is consumed as the second one's required leading \s+, so only one is removed —

dt>="1984-01-01" dt<="1984-12-31" summarize my year
   -> defilter: 'dt>="1984-01-01" summarize my year'

Impact

routers/helpers.py:1329 reconstructs the filters by string subtraction:

defiltered_query = defilter_query(q)
filters_in_query = q.replace(defiltered_query, "").strip()

When defilter returns the query unchanged, filters_in_query is "". That string is the only thing appended to each inferred search query (line 1371) and the only thing EntryAdapters.apply_filters sees, so the date range is dropped. Measured end to end:

BEFORE 'dt>="last week" what changed in my project'
        defiltered (-> LLM/embedding): 'dt>="last week" what changed in my project'
        filters_in_query            : ''
        date range hits DB filter   : NONE (filter silently dropped)
AFTER  'dt>="last week" what changed in my project'
        defiltered (-> LLM/embedding): 'what changed in my project'
        filters_in_query            : 'dt>="last week"'
        date range hits DB filter   : [1784563200.0, None]

Two user-visible effects for one input: the date filter isn't applied, and dt>="last week" is left in the text passed to extract_questions and embed_query. Neither is surfaced — the query just behaves as though no date filter were typed.

Fix

Drop the \s+ prefix and match the filter itself. The existing "remove multiple spaces" pass on the next line already tidies up the whitespace the substitution leaves behind, which is why nothing else needed to change.

Behavior parity

I enumerated all 280 combinations of 10 prefixes ("", "head ", "head", "head, ", "head,", "head (", "(", "head\n", "head\t", " ") × 4 filter forms (dt:, dt>=, dt<, dt==) × 7 suffixes:

total combinations tested : 280
outputs that differ       : 140
of those, cases where old code had ALREADY stripped the filter: 0
new code leaves a filter behind in: 0 cases (asserted)

Every output that differs is a case the old expression failed to strip. There is no combination where the old code worked and the new code changes the result.

Tests

Four tests added to tests/test_date_filter.py:

  • test_defilter_removes_date_filter_mid_query — the cases that already worked, so they can't silently regress.
  • test_defilter_removes_date_filter_at_start_of_query — the reported bug, including two adjacent filters and a query that is nothing but a filter.
  • test_defilter_removes_date_filter_after_non_space_character, and (.
  • test_defilter_removes_all_terms_it_reports — the invariant that ties the two methods together: nothing get_filter_terms finds may survive defilter.

Red on the unmodified tree:

FAILED test_date_filter.py::test_defilter_removes_date_filter_at_start_of_query
FAILED test_date_filter.py::test_defilter_removes_date_filter_after_non_space_character
FAILED test_date_filter.py::test_defilter_removes_all_terms_it_reports
3 failed, 7 passed in 1.75s

With the fix:

10 passed in 1.63s

test_defilter_removes_date_filter_mid_query passes either way by design — it pins the behavior this change must preserve rather than guarding the regression.

Verification

  • ruff check and ruff format --check clean on both files, run with ruff==0.12.3 to match the pin in .pre-commit-config.yaml.
  • No trailing whitespace, single trailing newline, LF-only in the committed blobs (trailing-whitespace / end-of-file-fixer hooks).
  • WordFilter.defilter has no leading-whitespace requirement and handles all these positions correctly, so it needed no change — I verified this rather than assuming it.
  • FileFilter.defilter has a separate bug (it never strips -file:"..."), already covered by Fix exclude file filters leaking into query in FileFilter.defilter #1345. I deliberately left it alone to avoid conflicting with that PR; this change touches only DateFilter.
  • The only other test file mentioning dt filters is tests/test_online_chat_actors.py, whose date tests assert on LLM output and never call defilter.

Disclosure: I could not run the full pytest suite locally — it needs Postgres with pgvector plus the --all-extras install. I ran tests/test_date_filter.py against both the pristine and fixed trees, and the end-to-end and 280-combination probes above, all against the real DateFilter module.

DateFilter.defilter matched a required leading whitespace before the date
filter, so a filter at position 0 was never removed. Anything other than
whitespace before the filter had the same effect, which covers start of
query, "(" and ",".

get_filter_terms uses the bare date_regex, so the two methods disagreed
about whether a filter was present. defilter_query then returned the query
unchanged, and routers/helpers.py recovers filters by subtracting the
defiltered query from the original, so filters_in_query came out empty and
the date range never reached apply_filters. The raw dt>="..." text was also
left in the query passed to extract_questions and embed_query.

For example, `dt>="last week" what changed in my project` searched the whole
knowledge base with the filter text still in the embedded query, while the
same filter written at the end of the query worked.

Dropping the \s+ prefix leaves the existing "remove multiple spaces" pass to
tidy up. Across 280 prefix/filter/suffix combinations, every output that
differs is one the old expression failed to strip; no case where it already
worked changes.

Fixes khoj-ai#1395
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Date filter at the start of a query is silently ignored

1 participant