2.0.0#31
Merged
Merged
Conversation
PGO, Server GC, SkipLocalsInit, SIMD PackedIntCodec and BM25 scoring, QualifiedTermCache, BinaryWriter elimination, ISpanStemmer, BBQ PopCount, Int8Distance SIMD, metrics false-sharing padding, AggressiveOptimisation on 10 hot methods, striped TermOffsetCache, QueryCache Interlocked counters, FST enumeration stackalloc, stackalloc fast path in filters, Levenshtein AggressiveOptimisation.
- Remove PGO and Server GC from benchmark projects in Directory.Build.props to fix zero-allocation measurement artifact in 14 LuceneNet benchmarks under Server GC - Restore inline scoring for selective TermQuery queries (docFreq < 256) to avoid 2KB stackalloc + two-pass batch overhead; the SIMD batch path is preserved for high-frequency terms where it provides benefit Benchmark config fix verified: GC=Concurrent Workstation Token filter regressions reduced from +7-16% to +3-6% Gutenberg search regressions confirmed as environmental noise (LuceneNet also regressed same amount)
Include Lucene.NET parity for Dirichlet and Jelinek-Mercer, plus Boolean query variants.
…ET term-vector and re-analysis highlighters.
…on references in codec comments
… CodecKit-era codec structure
…2% in analysis, 25% in flush.
…tringField, NumericField, StoredField and BinaryField default to true; TextField and VectorField default to false. Also fixed a double-byte-copy in AddBinaryDocValue where the string overload encoded to UTF-8 then called the span overload which ToArray'd a second time.
… new scoring models, language-model similarities, highlighters, StoreDocValues flag, profiling project, and indexing hot-path performance work.
…ld boost and presence dictionaries across the first pass; second pass reads values directly from lazy enumeration, keeping peak memory at O(largest single field) instead of O(all fields)
…hannel consumer on a dedicated thread, eliminating thread-pool starvation from lock contention in async continuations; remove AcquireBackpressureSlotAsync with it swallowed ObjectDisposedException
…rmVectors helper; DWPT IndexTextField applies char filters and conditionally stores offsets based on StoreTermVectors; DWPT constructor takes IndexWriterConfig instead of bool storePayloads
…eases through ReleaseSemaphoreSlots to guard against ObjectDisposedException after Dispose drain timeout; call MarkIndexingFailed before breaking out of the drain loop
…ecKit envelope version via TermDictionaryReader.Open + EnumerateAllTerms; add version-check fast path that copies or returns when already current
…onstantExpression, rejecting property access and static member expressions that would execute arbitrary code at query translation time
… escape pairs resolved by string.Create Unescape helper, allowing literal colons in URLs, timestamps, file paths, and operator characters in field values
…ream+BinaryReader using FileShare.Read|FileShare.Delete to avoid mmap file-locking on Windows after Dispose
…tion to handle transient Windows mmap handle release delays after TermDictionaryReader.Open disposes its IndexInput
…try with retry wrappers (Move, Copy, Delete, DeleteDirectory, ReadLines, file/directory/enumeration helpers); replace IndexAtomicFileWriter retry loop and RetryFileOperation helper with FileOpenRetry primitives to eliminate Windows file-locking failures from Defender and mmap handle release delays
Owner
Author
|
"It works on my machine" Works on my local Linux, local Windows, GitHub Linux I hate Windows. |
…edAccessException so retry actually fires on Windows file-locking; bump retry parameters from 5×10ms to 25×50ms to absorb CI Defender delays; add bufferSize/FileOptions Open overload; route IndexOutput through FileOpenRetry.Open
Owner
Author
Pre-Release Benchmark AcceptanceBenchmarks have run. These need sorting before release:
There are benchmarks that are close in performance (or Lucene.Net has a very slight gain), but they're minor enough for next minor release. But even all those, we win on allocation (often by a lot). Like On a better note, somethings have a slight performance increase, best I've noticed
Thanks to work on the FST. We don't need to traverse the whole graph if the root node doesn't have a hit |
… cache MLT terms, route deletion-commit by segment Replace double-search in SearchWithAggregations, SearchWithCollapse, and SearchWithFacets with ISideCollector invoked during postings iteration. Inline FunctionScore scoring for TermQuery inners to avoid TopNCollector(reader.MaxDoc) allocation. Fix ExecuteShouldOnlyHeap to use PopRoot/InsertHeap multi-way merge instead of rebuilding the heap every iteration, and raise HeapThreshold to 64. Cache MLT candidate terms with ConcurrentDictionary generation-swap eviction. Switch MLT to string-based GetDocFreqByQualified for _termOffsetCache hits. Route numeric id-field delete terms to the single segment containing their doc ID, avoiding O(K*S) FST lookups. Add index-sort early termination path when sort matches index sort. Fix LuceneNet aggregation and collapse benchmarks to iterate all matching documents.
…ric dispatch overhead
…archWithFacets fallback
…AOT) which dropped VSTest support on .NET 10+
…le locking Please for the love of god Window, please
…s; accept version 0 in NormsReader for migration tests
…perations fail fast, read/move/delete/copy use platform-aware transient retry (5x200ms on Windows, zero on Linux) FileOpenRetry.Open no longer retries because creating files or acquiring exclusive locks is either an atomic success or a permanent conflict that retrying cannot resolve. This was causing WriteLock_ConcurrentWriterConstruction to time out after 50x100ms=5s per failed thread with 16 threads contending OpenRead and OpenReadDelete now have their own inline retry loops instead of delegating to the no-retry Open. Move/Delete/Copy/DeleteDirectory use transient retry scoped to the few hundred milliseconds that Defender AV scanning can hold a handle On Linux, TransientMaxRetries is zero so all transient-retry paths become direct calls with no overhead. Collateral: TestDirectoryFixture cleanup uses FileOpenRetry.DeleteDirectory for resilience IndexBackup.CopyFileAtomically switched to OpenReadDelete (read path, needs retry).
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Attempting to use this version with an index created with any previous version, will not work. You need to reindex, and build it again
Important
Now feature frozen. If it hasn't been ticked off in #24 then it's not made cut for this version. Bug fixes, refactorings, and updates to existing features only