Refactor CSV and Excel enumerators for improved performance and clarity - #34
Merged
Conversation
…chronous and asynchronous row enumeration - Introduced `SyncRowEnumerator` and `AsyncRowEnumerator` base classes to encapsulate common enumeration logic for CSV and Excel formats. - Updated `CsvEnumerable` and `ExcelEnumerable` to inherit from the new base classes, simplifying their implementations. - Removed redundant code and improved performance by leveraging shared functionality. - Added `ZipReaderOpen` utility class to streamline ZIP archive handling in `XlsxReader` and `XlsbReader`. - Implemented `WriterStateGuard` for consistent state management across workbook writers. - Enhanced error handling and resource management in ZIP-related operations. - Deleted obsolete `XlsExcelEnumerable` class as its functionality is now covered by the refactored enumerators.
…rformance optimization across multiple files.
…rdWriter and BiffStringEncoder for cleaner code
- Introduced ProjectionRules class to encapsulate row classification logic and error handling for missing required values. - Updated CsvEnumerable and RowProjector to utilize ProjectionRules for row processing. - Enhanced ProjectionStep enum to include BuildMap step for better control flow. - Refactored CellAccumulator to add methods for handling boolean and error cell types. - Created PooledStreamRowEnumerator base class to reduce code duplication in CSV, XLSB, and XLSX enumerators. - Improved error handling in Excel reader methods to ensure proper disposal of streams on failure. - Streamlined cell writing logic in CellFormatter to reduce redundancy. - Updated DateSerial to leverage ExcelEpoch for date serial conversions. - Simplified RowWriter methods to directly write values instead of calling formatter methods. - Added tests to cover edge cases in formatting and parsing.
… enhance performance in cell parsing
… and code clarity
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #34 +/- ##
==========================================
+ Coverage 87.66% 88.81% +1.14%
==========================================
Files 78 83 +5
Lines 5701 5464 -237
Branches 1071 1038 -33
==========================================
- Hits 4998 4853 -145
+ Misses 459 385 -74
+ Partials 244 226 -18 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Benchmark ResultsMeasured on ExcelReader.Benchmarks.ColdStartBenchmark
ExcelReader.Benchmarks.CsvParseBenchmark
ExcelReader.Benchmarks.CsvReadBenchmark
ExcelReader.Benchmarks.CsvWriteBenchmark
ExcelReader.Benchmarks.ParseBenchmark
ExcelReader.Benchmarks.ReadBenchmark
ExcelReader.Benchmarks.RealDataReadBenchmark
ExcelReader.Benchmarks.RecordWriteBenchmark
ExcelReader.Benchmarks.WriteBenchmark
ExcelReader.Benchmarks.XlsReadBenchmark
ExcelReader.Benchmarks.XlsWriteBenchmark
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request refactors and modernizes the Excel parsing core, with a focus on simplifying and generalizing the column parsing logic, improving date and time handling, and updating project configuration for consistency and maintainability. The changes consolidate multiple structurally similar parser methods into generic implementations, introduce a shared Excel epoch conversion utility, and streamline project build settings.
Parser refactoring and simplification
ColumnParserFactoryto replace many type-specific parser methods with genericBuildValueandBuildNullableValuefactories, reducing code duplication and centralizing logic for all value-type columns. This includes new generic readers forbool,DateTime,DateOnly,TimeOnly, andGuid, and uses delegates to select the correct parsing strategy based on configuration. [1] [2] [3] [4] [5]Date and time handling improvements
ExcelEpochutility class to centralize and standardize conversions between Excel serial dates and OADate, ensuring consistent handling of the 1900/1904 date systems and the Excel leap year bug.[SkipLocalsInit]for stack-allocated buffers, which can improve performance and reduce unnecessary zero-initialization. [1] [2]API and type changes
ExcelEnumerable<T, TReader, TEnumerator>for both synchronous and asynchronous XLS parsing, replacing the oldXlsExcelEnumerable<T>type. [1] [2]Project configuration updates
LangVersion,ImplicitUsings,Nullable,Deterministic) toDirectory.Build.propsfor consistency across all projects, and removed them fromExcelReader.Core.csproj. Also enabledAllowUnsafeBlocksin the core project. [1] [2] [3]Directory.Build.props.Minor improvements
Guidparsing across different .NET versions.using System.Runtime.CompilerServices;for[SkipLocalsInit]attribute.