[Hack] Multiple spelling corrections are returned

[veloman-yunkan]

This is likely a throwaway PR intended to enable the customer to test a solution to openzim/libzim#1012 that may be implemented in a cleaner way at more development cost.

[codecov]

Codecov Report

❌ Patch coverage is 57.14286% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 42.92%. Comparing base (f82bfc0) to head (75a3363).
⚠️ Report is 78 commits behind head on main.

Files with missing lines	Patch %	Lines
src/spelling_correction.cpp	57.14%	0 Missing and 6 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1254      +/-   ##
==========================================
+ Coverage   42.87%   42.92%   +0.04%     
==========================================
  Files          60       60              
  Lines        4744     4746       +2     
  Branches     2498     2498              
==========================================
+ Hits         2034     2037       +3     
- Misses       1088     1089       +1     
+ Partials     1622     1620       -2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[veloman-yunkan]

@kelson42 @gremid

Here is a package that allows to evaluate the approach based on Levenshtein distance solution to multiple suggestions. The three misspellings used in the demo script produce the following results

Schtuhl
    Achtel
    Achtung
    Dachstuhl
    Hochstuhl
    Schach
    Schacht
    Schah
    Schakal
    Schal
    Schall
    Schaltuhr
    Schatulle
    Schau
    Schaub
    Schaube
    Schauer
    Schaufel
    Schaukel
    Schicht

gefärlich
    gefährlich
    gefänglich
    gefäßreich
    fährlich
    gebührlich
    klärlich
    spärlich
    ungefährlich

Phenomen
    Phänomen
    Phenole
    Pheromon
    Pronomen
    Pränomen
    Agnomen
    Cenoman
    Kognomen
    Phellogen
    Phenol
    Renomee
    benommen

I set a limit of 100 on the number of spelling corrections to be returned. As you can see xapian's implementation of Levenshtein distance (+my hack to have more than a single result returned) produces for Schtuhl a lot of noise without including for some reason the expected result of Stuhl (which is within a distance of 2 from Schtuhl). Note that the distance limit is hardcoded to a value of 3. In this particular case, I expect that a slightly different way of obtaining multiple results by trying increasing values of the distance limit will work better.

Please play with other misspellings and tell me if we should pursue this approach (or a variant of it) or go with a user-configurable general-purpose spell-checker instead.

[kelson42]

I let @gremid give his feedback, but it seems to me that the results we get here, even by having only the results with the shortest distance, @re not the one expected.

[kelson42]

Maybe @ojwb you would have an insightful comment here. We have been trying to implement since months an effecient spellchecking system for the german dictionary DWDS. First implementation is #1230 but it delivers only one result (we would like to have many results if many of them share the same TOP Levenstein distance) and we try here to deliver more results... but like you see we pretty much struggle to get the quality suggestions we expect. Do we do something wrong?

[ojwb]

I can't seem to reproduce the problem in current Xapian git master (and nothing in this area has changed for a while):

$ cd xapian-bindings/python3
$ ./run-python-test 
Python 3.13.11 (main, Dec  8 2025, 11:43:54) [GCC 15.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import xapian
>>> db = xapian.WritableDatabase("tmp.db")
>>> db.add_spelling("Stuhl")
>>> db.get_spelling_suggestion("Schtuhl", 3)
b'Stuhl'
>>> db.add_spelling("Achtel")
... db.add_spelling("Achtung")
... db.add_spelling("Dachstuhl")
... db.add_spelling("Hochstuhl")
... db.add_spelling("Schach")
... db.add_spelling("Schacht")
... db.add_spelling("Schah")
... db.add_spelling("Schakal")
... db.add_spelling("Schal")
... db.add_spelling("Schall")
... db.add_spelling("Schaltuhr")
... db.add_spelling("Schatulle")
... db.add_spelling("Schau")
... db.add_spelling("Schaub")
... db.add_spelling("Schaube")
... db.add_spelling("Schauer")
... db.add_spelling("Schaufel")
... db.add_spelling("Schaukel")
... db.add_spelling("Schicht")
... 
>>> 
>>> db.get_spelling_suggestion("Schtuhl", 3)
b'Stuhl'
>>> 

I see the same with 1.4.29.

If it's a Xapian bug, it could perhaps be to do with checking spellings with uncommitted spelling changes (the way that's implemented I'd expect it should just be handled transparently, but it's something you are doing here that is more unusual and so probably less well tested). You could try adding impl_->commit(); after impl_->remove_spelling(term); and if that fixes things then this probably is the cause.

As a more general point, I'd suggest you probably should call impl_->begin_transaction(); before you start removing spellings, and then you can rollback the temporary changes with impl_->cancel_transaction(); rather than having to loop over the words and add them back).

Also, I assume all your spellings are always added with frequency 1? For words where it is more remove_spelling() will just decrement the spelling frequency so you'll get repeated entries in your results. impl_->remove_spelling(word, (Xapian::termcount)-1); would take care of that (and if you rollback with cancel_transaction() the correct frequency will get restored for you; otherwise you'd need to store and restore it).

[veloman-yunkan]

I can't seem to reproduce the problem in current Xapian git master (and nothing in this area has changed for a while):

$ cd xapian-bindings/python3
$ ./run-python-test 
Python 3.13.11 (main, Dec  8 2025, 11:43:54) [GCC 15.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import xapian
>>> db = xapian.WritableDatabase("tmp.db")
>>> db.add_spelling("Stuhl")
>>> db.get_spelling_suggestion("Schtuhl", 3)
b'Stuhl'
>>> db.add_spelling("Achtel")
... db.add_spelling("Achtung")
... db.add_spelling("Dachstuhl")
... db.add_spelling("Hochstuhl")
... db.add_spelling("Schach")
... db.add_spelling("Schacht")
... db.add_spelling("Schah")
... db.add_spelling("Schakal")
... db.add_spelling("Schal")
... db.add_spelling("Schall")
... db.add_spelling("Schaltuhr")
... db.add_spelling("Schatulle")
... db.add_spelling("Schau")
... db.add_spelling("Schaub")
... db.add_spelling("Schaube")
... db.add_spelling("Schauer")
... db.add_spelling("Schaufel")
... db.add_spelling("Schaukel")
... db.add_spelling("Schicht")
... 
>>> 
>>> db.get_spelling_suggestion("Schtuhl", 3)
b'Stuhl'
>>> 

I see the same with 1.4.29.

@ojwb I was able to reproduce your example with 1.4.18

However when I populate the database with the more than 240 thousand entries found in the DWDS dictionary, db.get_spelling_suggestion("Schtuhl", 3) returns Achtel, and then removing the suggestions one by one results in the sequence reported earlier. Pasted below is the unedited transcript of my experiment (including the mistakes that I made in the process):

$ python3
Python 3.10.12 (main, Nov  4 2025, 08:48:33) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> db = xapian.Database("dwds_zim_titles.cleaned_up.spellingsdb")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'xapian' is not defined
>>> import xapian
>>> db = xapian.Database("dwds_zim_titles.cleaned_up.spellingsdb")
>>> db.get_spelling_suggestion("Schtuhl", 3)
b'Achtel'
>>> db = xapian.WritableDatabase("dwds_zim_titles.cleaned_up.spellingsdb")
>>> db.remove_spelling('Achtel')
>>> db.get_spelling_suggestion("Schtuhl", 3)
b'Achtung'
>>> db.remove_spelling('Achtung')
>>> db.get_spelling_suggestion("Schtuhl", 3)
b'Dachstuhl'
>>> db.remove_spelling('Dachstuhl')
>>> db.get_spelling_suggestion("Schtuhl", 3)
b'Hochstuhl'
>>> db.remove_spelling('Hochstuhl')
>>> db.get_spelling_suggestion("Schtuhl", 3)
b'Schach'
>>> db.remove_spelling('Schach')
>>> db.get_spelling_suggestion("Schtuhl", 3)
b'Schacht'
>>> db.remove_spelling('Schacht')
>>> db.get_spelling_suggestion("Schtuhl", 3)
b'Schah'
>>> db.remove_spelling('Schah')
>>> db.get_spelling_suggestion("Schtuhl", 3)
b'Schakal'
>>> db.remove_spelling('Schakal')
>>> db.get_spelling_suggestion("Schtuhl", 3)
b'Schal'
>>> db.remove_spelling('Schal')
>>> db.get_spelling_suggestion("Schtuhl", 3)
b'Schall'
>>> db.remove_spelling('Schall')
>>> db.get_spelling_suggestion("Schtuhl", 3)
b'Schaltuhr'
>>> db.remove_spelling('Schaltuhr')
>>> db.get_spelling_suggestion("Schtuhl", 3)
b'Schatulle'
>>> db.remove_spelling('Schatulle')
>>> db.get_spelling_suggestion("Schtuhl", 3)
b'Schau'
>>> db.remove_spelling('Schau')
>>> db.get_spelling_suggestion("Schtuhl", 3)
b'Schaub'
>>> db.remove_spelling('Schaub')
>>> db.get_spelling_suggestion("Schtuhl", 3)
b'Schaube'
>>> db.remove_spelling('Schaube')
>>> db.get_spelling_suggestion("Schtuhl", 3)
b'Schauer'
>>> db.remove_spelling('Schauer')
>>> db.get_spelling_suggestion("Schtuhl", 3)
b'Schaufel'
>>> db.remove_spelling('Schaufel')
>>> db.get_spelling_suggestion("Schtuhl", 3)
b'Schaukel'
>>> db.remove_spelling('Schaukel')
>>> db.get_spelling_suggestion("Schtuhl", 3)
b'Schicht'
>>> db.get_spelling_suggestion("Schicht", 3)
b'Schichte'
>>> db.remove_spelling('Schicht')
>>> db.get_spelling_suggestion("Schtuhl", 3)
b''
>>> 

[veloman-yunkan]

As a more general point, I'd suggest you probably should call impl_->begin_transaction(); before you start removing spellings, and then you can rollback the temporary changes with impl_->cancel_transaction(); rather than having to loop over the words and add them back).

@ojwb Thanks! That's a useful hint.

[veloman-yunkan]

Maybe @ojwb you would have an insightful comment here. We have been trying to implement since months an effecient spellchecking system for the german dictionary DWDS. First implementation is #1230 but it delivers only one result (we would like to have many results if many of them share the same TOP Levenstein distance) and we try here to deliver more results... but like you see we pretty much struggle to get the quality suggestions we expect. Do we do something wrong?

@kelson42 I am not so much concerned about a potential bug in xapian, as about the usefulness of the Levenshtein distance based approach of spelling correction to DWDS's use case. I think I have demonstrated that that approach will produce non-relevant spelling suggestions for language learners (as opposed to random mistakes made when typing the word).

[ojwb]

@veloman-yunkan Please can you put a copy of your dwds_zim_titles.cleaned_up.spellingsdb somewhere I can download it to poke at.

[veloman-yunkan]

@veloman-yunkan Please can you put a copy of your dwds_zim_titles.cleaned_up.spellingsdb somewhere I can download it to poke at.

@kelson42 @gremid Can I publicly share the word list of the DWDS dictionary?

[kelson42]

@veloman-yunkan Please can you put a copy of your dwds_zim_titles.cleaned_up.spellingsdb somewhere I can download it to poke at.

@kelson42 @gremid Can I publicly share the word list of the DWDS dictionary?

@veloman-yunkan I think this is OK. Go ahead.

[veloman-yunkan]

@veloman-yunkan Please can you put a copy of your dwds_zim_titles.cleaned_up.spellingsdb somewhere I can download it to poke at.

@ojwb Here you are - dwds_zim_titles.zip. Run the following command after unpacking the archive:

./create_dwds_spellings_db dwds_zim_titles.cleaned_up

[ojwb]

Thanks, I can reproduce the problem locally now.

[ojwb]

This much shorter list is enough to trigger it:

Schiedsrichterstuhl
Stuhl

There's an n-gram filter which identifies candidate words to calculate the edit distance for. What's happening is that Schiedsrichterstuhl matches 6 n-grams from Schtuhl (we have trigrams plus head and tail bigrams, and it matches ^Sc, Sch, cht, tuh, uhl and hl$) whereas Stuhl matches 3 (tuh, uhl and hl$) and we consider Schiedsrichterstuhl first and record that we've seen a score of 6. New candidates are required to have an n-gram score within 2 of the best, but that seems a flawed optimisation as a candidate which is actually much too long can get a high n-gram score.

I'm unsure of the logic behind this threshold - I don't see it documented in the code, but maybe it's in an old ticket somewhere. Making the threshold 3 here fixes this case, but I think I could construct examples which fail with arbitrary thresholds so it seems this may just be flawed. We may be able to just drop the check without too much impact - the edit distance calculation has been optimised and is now much faster than it used to be (I've done some experiments doing spelling correction without storing any special spelling data and just considering unstemmed terms as correctly spelled words and even that seems fast enough).

I'll investigate further and decide what the best way to address it is.

Thanks again for providing the data to reproduce.

[ojwb]

I'll investigate further and decide what the best way to address it is.

I've pushed a fix: xapian/xapian@e91ba89

FWIW, with this fixed the suggested correction for Schtuhl is actually Schmul rather than Stuhl - both are 2 edits away and all the words are added with frequency 1 so it's essentially arbitrary which is chosen (I think in this case we just return the one we saw first, which will be the one which sorts first comparing strings by byte value).