The Vision
A lexer that uses UTF-8 is a necessity because it would make the lexing process faster and UTF-16 is overkill anyway.
The Problem
Slowness and the need for heap allocations (strings are not UTF-8).
The Solution
A custom lexer, titled Utf8Lexer, which implements interface IUtf8Lexer.
Benefits
Reduction of edge cases and heap allocations across the projects.
Potential Drawbacks
Added complexity.
User Impact
Users who need the best performance possible really need this feature.
Related Context
The PR that added this is #74, but said PR did not test the changes made.
The Vision
A lexer that uses UTF-8 is a necessity because it would make the lexing process faster and UTF-16 is overkill anyway.
The Problem
Slowness and the need for heap allocations (strings are not UTF-8).
The Solution
A custom lexer, titled
Utf8Lexer, which implements interfaceIUtf8Lexer.Benefits
Reduction of edge cases and heap allocations across the projects.
Potential Drawbacks
Added complexity.
User Impact
Users who need the best performance possible really need this feature.
Related Context
The PR that added this is #74, but said PR did not test the changes made.