[JuliaSyntax] Stop emitting K".." and K"..." in lexer#61992
Merged
Conversation
Member
Author
|
There is one more PR in this stack, which I will put up once this one merges to save CI time. |
Member
Author
|
(There was an open question on the PR whether to go back to bump_glue to rewrite the token in place after the decision is made - I have decided for now not to go back to that. My intuition is that not rewriting tokens is a useful invariant of the data structure, although I don't have any concrete use cases in mind at the moment. If it's particularly annoying in practice, we can revisit this in another PR). |
Unfortunately, the sequences `..` and `...` do not always refer to the `..` operator or the `...` syntax. There are two and a half cases where they don't: 1. After `@` in macrocall, where they are both regular identifiers 2. In `import ...A` where the dots specify the level 3. `:(...)` treats `...` as quoted identifier Case 1 was handled in a previous commit by lexing these as identifiers after `2`. However, as a result of case 2, it is problematic to tokenize these dots together; we essentially have to untokenize them in the import parser. It is also infeasible to change the lexer to have speical context-sensitive lexing in `import`, because there could be arbitrary interpolations, `@eval import A, $(f(x..y)), ..b`, so deciding whether a particular `..` after import refers to the operator or a level specifier requires the parser. Currently the parser handles this by splitting the obtained tokens again in the import parser, but this is undesirable, because it invalidates the invariant that the tokens produced by the lexer correspond to the non-terminals of the final parse tree. This PR attempts to address this by only ever having the lexer emit `K"."` and having the parser decide which case it refers to. The new non-terminal `K"dots"` handles the identifier cases (ordinary `..` and quoted `:(...)` ). K"..." is now exclusively used for splat/slurp, and is no longer used in its non-terminal form for case 3. (cherry picked from commit 6e81eb1c70dd664a9bd888f13cf0b4b4e443f8fa)
Keno
added a commit
to KenoAIStaging/julia
that referenced
this pull request
Jun 10, 2026
`a .≔ b` (likewise `.⩴` and `.≕`) crashed the parser with an internal "Unexpected dotted operator" error since the dots were split into separate tokens (JuliaLang#61992), because `dotted()` only handled the heads `.= .&& .|| .op=`. Add dedicated `K".≔"`/`K".⩴"`/`K".≕"` heads - matching `K".="` - so these parse to `(.≔ a b)` and convert to `Expr(:.≔, a, b)` as in the reference parser. Dotted `:=` is not valid syntax (the reference parser rejects it too), so emit a proper parse error for it instead of crashing. Also a few smaller items from review of JuliaLang#62000: * The new heads also fix `Expr(:.≔, a, b)` conversion into a `SyntaxTree`, which previously went through JuliaLowering's compound assignment `unknown_head` path. That path now additionally requires a trailing `=` in the head name, so operator-named heads such as a hand-written `Expr(:⊕, a, b)` produce a lowering error rather than being mangled into an `op=` form. * Add `K"Operator"` to `_nonunique_kind_names`, since the kind name doesn't determine the token text. * Remove duplicated `·` (U+00B7) and `·` (U+0387) entries from the `PREC_TIMES` operator table (they are listed - with comments - at the end of the list). * Fix an asymmetry in `Expr` conversion of one-argument `K".op="` vs `K"op="` heads. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keno
added a commit
to KenoAIStaging/julia
that referenced
this pull request
Jun 10, 2026
`a .≔ b` (likewise `.⩴` and `.≕`) crashed the parser with an internal "Unexpected dotted operator" error since the dots were split into separate tokens (JuliaLang#61992), because `dotted()` only handled the heads `.= .&& .|| .op=`. Add dedicated `K".≔"`/`K".⩴"`/`K".≕"` heads - matching `K".="` - so these parse to `(.≔ a b)` and convert to `Expr(:.≔, a, b)` as in the reference parser. Dotted `:=` is not valid syntax (the reference parser rejects it too), so emit a proper parse error for it instead of crashing. Also a few smaller items from review of JuliaLang#62000: * The new heads also fix `Expr(:.≔, a, b)` conversion into a `SyntaxTree`, which previously went through JuliaLowering's compound assignment `unknown_head` path. That path now additionally requires a trailing `=` in the head name, so operator-named heads such as a hand-written `Expr(:⊕, a, b)` produce a lowering error rather than being mangled into an `op=` form. * Add `K"Operator"` to `_nonunique_kind_names`, since the kind name doesn't determine the token text. * Remove duplicated `·` (U+00B7) and `·` (U+0387) entries from the `PREC_TIMES` operator table (they are listed - with comments - at the end of the list). * Fix an asymmetry in `Expr` conversion of one-argument `K".op="` vs `K"op="` heads. * Bump the JuliaSyntaxHighlighting pin to drop a stray local-path `[sources]` section (left over from testing) from its Project.toml. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[This is a forward port of https://github.com/JuliaLang/JuliaSyntax.jl/pull/573, which was approved there, but didn't make the move - I'm not currently working on this code, but I'd like to get my open PRs worked in before everything becomes too stale. Rebased and pendubg review comments addressed by Claude. Original Description Follows]
Unfortunately, the sequences
..and...do not always refer to the..operator or the...syntax. There are two and a half cases where they don't:@in macrocall, where they are both regular identifiersimport ...Awhere the dots specify the level:(...)treats...as quoted identifierCase 1 was handled in a previous commit by lexing these as identifiers after
2.However, as a result of case 2, it is problematic to tokenize these dots together; we essentially have to untokenize them in the import parser. It is also infeasible to change the lexer to have speical context-sensitive lexing in
import, because there could be arbitrary interpolations,@eval import A, $(f(x..y)), ..b, so deciding whether a particular..after import refers to the operator or a level specifier requires the parser.Currently the parser handles this by splitting the obtained tokens again in the import parser, but this is undesirable, because it invalidates the invariant that the tokens produced by the lexer correspond to the non-terminals of the final parse tree.
This PR attempts to address this by only ever having the lexer emit
K"."and having the parser decide which case it refers to. The new non-terminalK"dots"handles the identifier cases (ordinary..and quoted:(...)). K"..." is now exclusively used for splat/slurp, and is no longer used in its non-terminal form for case 3.(cherry picked from commit 6e81eb1c70dd664a9bd888f13cf0b4b4e443f8fa)