Feature: Implement the UTF-8 lexer - #74
Merged
Merged
Conversation
…dex and Range types
Essentially, splitting the previously huge `Directory.Build.props`.
…ct and split it by diagnostics and panics (soon by sources as well)
They were heavy to maintain and didn't add much, as the lexer barely used the APIs they exposed. On top of that, .NET has many better ways to handle things like that. Currently code is fundamentally broken, so CI will fail.
…II extensions instead of UTF-16 string methods) [1/?] Code is fundamentally broken, so CI will fail. Lots of TODOs in code. Essentially, this is very bad code LMFAO (for now). Signed-off-by: Matthew Refs: #49
…II extensions instead of UTF-16 string methods) [2/3] Code is fundamentally broken, so CI will fail. Lots of TODOs in code. Essentially, this is very bad code LMFAO (for now). Signed-off-by: Matthew Refs: #49
Context: Refactor the lexer for performance (using UTF-8 and ASCII extensions instead of UTF-16 string methods) Code is no longer broken. Signed-off-by: Matthew Refs: #49
Member
Author
|
Sending this back to clear build errors and logic errors. Build errors found:
Logic errors found:
View CI status and reviews for more info. |
mf366-dev
commented
Aug 12, 2026
| : Result.Success(new Token(kind, startLoc, reader.Consumed - startLoc)); | ||
| } | ||
|
|
||
| private void SkipWhitespaceAndComments(ref SequenceReader<byte> reader) |
Member
Author
There was a problem hiding this comment.
Updates to position.Line are very rare and are not working as they should.
Member
Author
|
There aren't any tests yet, but this is getting merged. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before you continue
Before submitting your pull request, please check all the boxes that apply:
Summary
This adds the UTF-8 lexer that is gon' get RSML development actually started. It also removes the buffer but oh well.
Related Issues
Closes #49.
Testing
No tests were added (marked for later).
Changes Made
IUtf8LexerandUtf8LexerThank you for your contribution!