folder_contents search filter not accepting umlauts

[thet]

BUG/PROBLEM REPORT (OR OTHER COMMON ISSUE)

What I did:

• One a Plone 6.2.1 site (classic.demo.plone.org)
• Went to: https://classic.demo.plone.org/ and logged in as admin
• Added a document with the title "täst": https://classic.demo.plone.org/en/++add++Document
• Went to https://classic.demo.plone.org/en/folder_contents and filtered for "täst"
• Nothing is shown.

What I expect to happen:

• the "täst" content being shown.

What actually happened:

• the "täst" content not being shown.

What version of Plone/ Addons I am using:

6.2.1

[kunalKumar-13]

I tried to reproduce this on the server side and could not, so I want to share
what that rules out before someone else spends the same time.

Environment: plone.app.content against the 6.2-dev constraints, which resolves
to Products.CMFPlone 6.2.1 — the version in the report.

I replayed the request the filter box makes, byte for byte from
mockup's getCriterias() (term with the appended *, Title +
string.contains, and the <path>::1 criterion), over real HTTP to
@@getVocabulary?name=plone.app.vocabularies.Catalog:

filter term	results
test	1
täst	1
täst*	1
täst (mojibake control)	0

Tested with both \uXXXX-escaped JSON and raw UTF-8 (what JSON.stringify
sends) — both find the document. I included the mojibake row deliberately: it
shows the reproduction is sensitive to the encoding damage that would explain
this, and no such damage is happening.

At the catalog layer things are consistent too — Title is a ZCTextIndex on
plone_lexicon (Splitter → I18NNormalizer), and baseNormalize folds ä→a on
both sides:

stored terms:           ['plone', 'site', 'taest', 'tast']
parseTerms('täst')  ->  ['tast']
parseTerms('täst*') ->  ['tast*']

I also checked the language hypothesis, since plone.i18n.normalizer.de maps
ä→ae and an index/query mismatch there would look exactly like this bug. It
does not apply: I18NNormalizer is a lexicon pipeline element using
baseNormalize regardless of site language, so both sides agree.

So this looks like site state rather than code. The one mechanism I can see that
produces exactly this signature — ASCII search fine, accented search empty — is
a lexicon whose stored terms were written under a different pipeline than the one
now answering queries. plone_lexicon is used for both indexing and querying, so
the two can only disagree if the pipeline changed after the content was indexed
and the affected indexes were never rebuilt. Stored täst versus a query
normalized to tast misses; test is unaffected either way.

Two things that would confirm or kill that quickly on the demo site:

1. In portal_catalog → Indexes, what does plone_lexicon's pipeline show? It
   should be Unicode Whitespace splitter + Unicode Ignoring Accents Case Normalizer.
2. Does reindexing Title make "täst" findable?

I ruled out multilingual as a cause on logic rather than testing, for what it is
worth: language filtering would suppress the ASCII search too, and that one works.

Happy to keep digging — I have the reproduction environment set up, so if the
pipeline turns out to be correct on the demo I can chase the client side next.