[JuliaLowering] Syntax versioning and macro-expansion improvements

[mlechu]

Attach a new SyntaxContext to all syntax, which is shared between all nodes of
the same macro expansion (with some exceptions for escape, etc.).

Cool new stuff

Macros can now be syntax-version-aware, so the following is now possible:

macro some_versioned_macro(arg)
    if syntax_version(__context__.macrocall) < v"1.14"
        # ...
    elseif syntax_version(arg) < v"1.14"
        # ...
    else
        # ...
    end
end

and I've added the first one, @legacy_quote_to_syntax for testing purposes
within JuliaLowering:

julia> JuliaLowering.include_string(Main, "@legacy_quote_to_syntax :(a + b)"; expr_compat_mode=true) == Expr(:call, :+, :a, :b)
true

julia> JuliaLowering.include_string(Main, "@legacy_quote_to_syntax :(a + b)") isa SyntaxTree
true

Design

TL;DR: Racket does it right. The following should be equivalent:

JL	Racket
SyntaxTree	syntax object
Expr/Symbol/etc	plain data
K"inert"/QuoteNode	'() (quote)
K"quote"/:()	`() (quasiquote)
K"$"	,() (unquote)
syntaxinert (formerly inert_syntaxtree)	#'() (syntax)
syntaxquote (new, no surface syntax yet)	#`() (quasisyntax)
syntaxunquote(new, no surface syntax yet)	#,() (unsyntax)

This PR doesn't implement the full sets-of-scopes resolution, though it does
make it easier to implement if we wanted that. No local macros makes it
less important. We miss out on some cross-toplevel-thunk resolution, which I
think is OK or desirable for now.

After each macro expands, we want to apply a new scope layer to all new syntax
(that wasn't passed as a macro argument at some point). Racket applies the new
scope to all argument syntax, expands, then inverts that scope throughout the
expansion. This PR expands, then applies the new layer to all syntax whose
existing layer is not known to be "under" that new layer.

Fixes

• Our integer-ID-based scope layer dies with the MacroExpansionContext object,
  so the return value of macro expansion isn't a self-contained thing you can
  eval. Fix this.

• Right now, when expr_compat_mode=true, the surface syntax :(x + 1) is a
  SyntaxTree, but the :x is still Symbol. That means we have a special case
  to support the following:

  let x = :sym
  :(quoted = $x)
  

  and this causes IR validity errors:

  let x = Expr(:call, :identity, 1)
  eval(:(function f; return $x; end))
  

  This PR takes the same approach as Racket: Non-syntax interpolated into
  syntax is automatically upgraded. (This automatic upgrade also happens with
  the return value of a macro, and the body of a generated function, although
  I've left a couple cases disallowed to catch my own mistakes for now).

• On top of the above, quote; end and :() no longer produce SyntaxTree
  under any circumstance. This PR uses @legacy_quote_to_syntax as a
  stop-gap, which converts quote/$ to syntaxquote/syntaxunquote. Ideally we
  can add surface syntax for this later.

• : no longer produces SyntaxTree, but that used to be one of a whole class
  of problems now solved by syntax carrying its version: An expansion may
  contain arbitrary mixtures of syntax, so e.g. increased strictness on a form
  in a newer version can't be enforced without changing the form (like in
  syntax: Disallow using @goto to skip finally blocks #60979).

• I got rid of :scope_layer when calling old macros from new ones. This fixes
  Bug report from JETLS: :scope_layer interfering with user code JuliaLowering.jl#108, which has been
  causing some pkgeval and JETLS problems (cc @aviatesk), and clearly got in the
  way of the robot in WIP: Separately compile JuliaSyntax/JuliaLowering into a standalone frontend #62073. I've fixed our premature collapse of escape
  nodes in prepare_macro_args so we shouldn't need it, and in general, macro
  hygiene for old-style macros is not something that can be done without
  breakage or drastic changes to Symbol.

• Change my approach from [JuliaLowering] Implement macro hygiene exemptions for some forms #62013 for macro hygiene exemptions. This PR does it
  in desugaring, which is closer to how macroexpand.scm does it.

• Add macroexpand1

• Fix modules-in-expansions-with-escaped-names (pkgeval)

• Fix $ in dot-rhs (pkgeval)

• Fix some cases of invalid IR from generated functions returning weird forms

TODO

• The expr_compat_mode parameter should be deprecated in favour of the tree
  carrying its syntax version.
• Use syntax_module(st) instead of ctx.mod throughout lowering? Probably
  more correct, though may cause some "eval-into-closed-module" errors
• Performance/tree-mutation cleanup: I've been sloppy with when the syntax
  context is mutated or updated immutably.
• I haven't updated the lowering iterator (eval.jl), which probably no longer
  needs to be an iterator.
• Escaping in modules is not quite right yet, though should be rare given the questionable current behaviour.
• Mentioned by @topolarity: We should decide on what we guarantee when macros manually call macroexpand

Pinging @Keno to take a look at this approach to syntax versioning: I'll
probably want to remove Core.MacroSource (and the current implementation of
@VERSION) if it works.

@aviatesk This will likely require JETLS tweaks (e.g. ex.context.layer === ctx.syntax_context.layer instead of ex.scope_layer == 1); let me know if you
need anything on the JL end.

Some tests written with AI.

[topolarity]

I'm mildly concerned about versioning of syntax being a bit brittle for "non-local" constructs (e.g. break / @label from #60481 and also #60979). e.g. is the syntax version of the @label what's required, or the break, or do they both need a version that permits labeled break? More ambitious syntax changes that affect how structs, kwargs, closures, etc. would be lowered seem similarly entangled, or more.

I assume we'll typically have a single head / node that corresponds to the construct and can mostly use that to choose lowering / macro behavior, which seems good enough - just something to watch out for composability-wise

On the plus side, this does seem to handle interpreting a break identifier from "old" syntax quite well IIUC

[mlechu]

Changes just now:

• Fix wrong module being used in eval-call on K"toplevel" in desugaring (see comment)
• Fix wrong module being used in export and public returned from macros
• Add support for ($ (... x)) and ($ a b) in syntaxquote produced by @legacy_quote_to_syntax
• Fold macro_source into SyntaxContext: sorry for the churn (cc @aviatesk). The macro_prov, macro_prov_end helpers should function as they did before. This saves space, avoids dangling references I was hitting in interpolation, and prepares us for experimentation with a standard pointer-based tree structure.
• Move SyntaxContext, ScopeLayer into JuliaSyntax partially because of the point above
• Lots of new tests, some botted

[mlechu]

I think the "manual macroexpand" case shouldn't be too hard to support (and I might do that now, since there are a lot of "cannot document" errors in the pkgeval). I probably want a unique context per top-level thunk, and to change "unknown layer" in apply_expansion_layer to mean that st.context does not have the base layer in its .escaped chain. Manual macroexpand from a macro body should build upon the layers given to that macro, so the returned syntax should not count as "unknown".

Claude (welcome back Fable) also found some issues, which I'll fix:

• @ast copies SyntaxContext from the source node (which is intended within lowering) so this messes with the hygiene of some of the macros in syntax_macros.jl. I don't think users should be using @ast in the first place, since providing the first two args is more of a lowering-internal thing anyway.
• relayering of plain global and typegroup needs fixing and testing
• expr_to_est keeps linenodes as K"Value" (I should also leave a comment about what I did here).

[mlechu]

I think this works now. I've made the changes in the previous comment, which cleaned up the semantics around macros that emit toplevel/module. Every top-level thunk gets one and only one base layer, and each macro expansion may produce arbitrary crap, but apply_expansion_layer fixes all syntax to have the same base layer before it can be observed.

Macros should now be able to call macroexpand on their arguments without supplying a module argument:

    JuliaLowering.include_string(test_mod, raw"""
    macro undo_inert(x)
        x2 = JuliaLowering.macroexpand(x)
        x2[1]
    end
    """)
    @test JuliaLowering.include_string(test_mod, raw"""
    let foo = 1; @undo_inert(:foo); end
    """) == 1
    @test JuliaLowering.include_string(test_mod, raw"""
    let foo = 1; @undo_inert(@legacy_quote_to_syntax(:foo)); end
    """) == 1

I suspect (but haven't tested) that repeated macroexpand(st::SyntaxTree; recursive=false) is equivalent to recursive macro expansion, which would be a nice outcome.

cc @Keno I've also added a @syntax_version macro that behaves like @VERSION with zero args, or sets the version of some syntax with two args. This is finally a way of migrating to new macros that I wouldn't be ashamed of asking users to use:

julia> JuliaLowering.include_string(test_mod, raw"""
       using JuliaLowering: @syntax_version, @legacy_quote_to_syntax

       @syntax_version v"1.13" macro old_macro(args...)
           esc(Expr(:tuple, "old", typeof(args), args...))
       end

       macro new_macro(args...)
           @legacy_quote_to_syntax :("new", $(typeof(args)), $(args...),)
       end

       println(stdout, @old_macro(1, 1+1, :(1+1)))
       println(stdout, @new_macro(1, 1+1, :(1+1)))
       """)
("old", Tuple{Int64, Expr, Expr}, 1, 2, :(1 + 1))
("new", Tuple{SyntaxTree{Dict{Symbol, Dict{Int64, Any}}}, SyntaxTree{Dict{Symbol, Dict{Int64, Any}}}, SyntaxTree{Dict{Symbol, Dict{Int64, Any}}}}, 1, 2, :(1 + 1))

of course, we still need to work on the interface for creating SyntaxTree, which for now is just @legacy_quote_to_syntax or macroexpand of some pre-written macro.

[mlechu]

Also fixed the regressiosn found in #61576: I'm using _legacy_quote_to_syntax in @eval to retain provenance, but I converted nested quote, which was wrong (should be Expr). StatsBase failed because an expansion produced a keyword-arg function and I put the body method in the wrong module.

[topolarity]

@mlechu What's the stance on:

macro foo()
    :(@eval const x = :foo)
end

I don't believe the expected __module__ follows the syntax_module here

edit: Looks like SafeTestsets.jl exercises this, as well as more top-level hygiene exceptions that you may or may not be aware of

[topolarity]

One more comment: It occurs to me that this still leaves Expr vs. SyntaxTree incoherent w.r.t. their notion of self-quoting IR. I assume the intended fix here will be to make SyntaxTree a Core type and to define it to be non-self-quoting, so that both Expr and SyntaxTree require QuoteNode wrappers etc. to embed in AST as leaf values. Is that close to the mark?

[mlechu]

What problems arise if we just make SyntaxTree self-quoting in Expr?

[mlechu]

What's the stance on:

Thanks, should be fixed now. I guess @eval should just be fully-escaping. const is kind of a strange special case, since flisp does give it name-hygiene. import/using should also be fully-escaping.

I took some tests from claude's attempt at topolarity@592451a

[topolarity]

What problems arise if we just make SyntaxTree self-quoting in Expr?

I believe the main issue is anywhere we do ex isa SyntaxTree ? ex : expr_to_est(ex) - technically we can't distinguish the Expr-leaf-value vs. ST-as-actual-AST cases AFAIK

I guess ...

Sounds coherent to me (and mini-PkgEval agrees with you)

since flisp does give it name-hygiene

Nice call out. Yeah, apparently CBOOCall.jl relies on this (particularly the referent module):

julia> JuliaLowering.include_string(Main, """
       module Foo
       macro mkconst(val)
           Expr(:block, Expr(:const, Expr(:(=), :foo, esc(val))))
       end
       end
       """; expr_compat_mode=true)
julia> module Flisp; using ..Foo; Foo.@mkconst 13; end
julia> names(Flisp; all=true) .|> String .|> contains("foo") |> any
true
julia> isdefined(Foo, :foo)
false

compared to JL:

julia> JuliaLowering.include_string(Main, """
       module JL; using ..Foo; Foo.@mkconst 13; end
       """; expr_compat_mode=true)
julia> names(JL; all=true) .|> String .|> contains("foo") |> any
false
julia> isdefined(Foo, :foo)
true

as a lowering outsider, it's a bit odd that the LHS of :const acts like a GlobalRef and not an identifier for a binding in the ultimately-evaluating Module. But I assume there are good reasons for that

Anyway I guess this means more tweaks to our hygiene exception behavior? Doesn't need to be this PR

[mlechu]

I believe the main issue is anywhere we do ex isa SyntaxTree ? ex : expr_to_est(ex) - technically we can't distinguish the Expr-leaf-value vs. ST-as-actual-AST cases AFAIK

I see. I'll keep it in mind; probably best to follow what Racket does here.

Anyway I guess this means more tweaks to our hygiene exception behavior? Doesn't need to be this PR

Not sure if a "good" reason, but const x = 1 behaves mostly like x = 1 throughout both lowering implementations, so it gets full hygiene when JL makes unadorned toplevel assignments local. I mainly wanted to stop the errors from const x = 1 being counted as a local at toplevel, which was a more pressing issue, so I made it behave like global. Happy to tweak it at some point (unadorned assignments would likely also need tweaking).

[topolarity]

Various miscellaneous (minor) typos, etc. - many pointed out by Claude 🤖

First batch before a proper review from me

[topolarity]

I don't think we have much of a chance to fix this one without eagerly applying hygiene on macro boundaries, which sounds gnarly:

julia> macro oldpair(a, b)
           :(($(esc(a)), $(esc(b))))
       end
julia> JuliaLowering.include_string(Main, raw"""
       macro mix(callerx)
           JuliaLowering.@legacy_quote_to_syntax quote
               x = "macro-layer x"
               @oldpair $callerx x
           end
       end
       let x = "caller x"; @mix x end
       """)
("macro-layer x", "macro-layer x")

but wanted to get your thoughts (and maybe add the test for doc)

[mlechu]

but wanted to get your thoughts

Good question. This looks like the call_oldstyle_macro test in macros.jl, which I broke by removing Expr(:scope_layer). The problem is that oldpair needs to have callerx's new-style information pass through a, but there isn't a general way of doing that without breaking oldpair. I considered pulling an :escape node out of the layer in expr_to_est, but doing that automatically wouldn't be much less breaking than :scope_layer, and would have limitations (callerx may a mix of layers).

I think it would be reasonable to throw an error if syntax is not at the expected layer in expr_to_est and require new macros manually delete hygiene (or just do something like tmp_callerx = $callerx; @oldpair tmp_callerx x, maybe with a macro) before calling an old macro. Not sure how cumbersome that would be in practice. Let me know what you think.

[topolarity]

Mentioned to @mlechu some concerns around the semantic impacts of the JL_OLD_SYNTAX_VERSION fallback for anything coming out of old-style macros. She had several ideas - most promising might be the observation that syntactic forms are all Expr (not leaf literals) so they reliably have identity and we can potentially track their provenance with a field or side table etc. That's potentially much better than our provenance situation for identifiers etc. which do not have identity and so cannot be reliably tracked.

Separate note, but I am sure we'll run into some quirks eventually w.r.t. the lossy-ness of apply_expansion_layer but I think our co-existence troubles with old-style macros probably dominate over the scope limitations here. This seems to chart a fairly clear path for lifting those limitations later too.

Anyway this change looks good to go from my POV. A few hygiene fixes to follow-up with, but nothing that blocks these (nice) changes. Thanks!

[topolarity]

x86_64-apple-darwin build seems prone to timeouts lately - Sadly rebuilds / retries are currently broken on CI

https://buildkite.com/julialang/julia-pr/builds/260 was a full pass otherwise (the PR has not changed), so rather than waste more CI time let's merge this 👍

[aviatesk]

I merged aviatesk/JETLS.jl#799. This allows JuliaLang/JuliaLowering.jl#108 and aviatesk/JETLS.jl#108 to be closed. Once again, thank you for your work, @mlechu!