[JuliaSyntax] Allow macros enclosed in parens to treat newlines as whitespace

[MasonProtter]

Closes #52842

Supersedes JuliaLang/JuliaSyntax.jl#475. cc @c42f and @JeffBezanson who I discussed this with previously.

Macros currently have an restriction that I always found very annoying: If you want newlines in a macro call currently, you need to switch from e.g.

@info "description" x=f(...) y=g(...) z=h(...)

to

@info("description", 
      x=f(...), 
      y=g(...),
      z=h(...))

which requires ugly commas separating the arguments, and a call-like syntax for @info. I would prefer to at least have the option to write this as

(@info "description" 
   x=f(...) 
   y=g(...)
   z=h(...))

which is currently a syntax error, but now works with this PR:

julia> (@info "description"
           x = sin(1.5)
           y = 2
           z = 3)
┌ Info: description
│   x = 0.9974949866040544
│   y = 2
└   z = 3

I see this feature as directly analogous to how we allow infix function calls to span multiple lines if they're in parens:

julia> (1 + 
          2)
3

julia> (1 + sin(π)
        - 2)
-1.0

---

It took me a couple years, but I finally figured out how to make this work in a backwards compatible way without breaking cases like

julia> parsestmt(Expr, """
       (@inline function bar()
            @baz x
            y 
       end)
       """)
:(#= line 1 =# @inline function bar()
          #= line 1 =#
          #= line 2 =#
          #= line 2 =# @baz x
          #= line 3 =#
          y
      end)

and this doozy which the mocking tests generated

julia> parsestmt(Expr, """
       Ex(@a, function   @g
                 @la   
             end)
       """)
:(Ex(#= line 1 =# @a(), function #= line 1 =# @g()
          #= line 1 =#
          #= line 2 =#
          #= line 2 =# @la
      end))

but in this revision of the PR these now correctly parse, and I'm seeing all the tests pass locally at least.

---

Easter egg

A silly side-effect of this change is that you can now write lispy s-expression syntax in julia with no end in sight!:

eval(Expr(:macro, Expr(:call, :macro, :sig, :body),
          Expr(:block,
               Expr(:call,
                    :esc,
                    Expr(:quote,
                         Expr(:macro,
                              Expr(:call,
                                   Expr(:$, Expr(:ref, Expr(:., :sig, QuoteNode(:args)), 1)),
                                   Expr(:$, Expr(:...,
                                                 Expr(:ref, Expr(:., :sig, QuoteNode(:args)), Expr(:call, :(:), 2, :end)))),),
                              Expr(:block, Expr(:$,:body))))))))

using Base: isexpr

(@macro call(args...)
 esc(Expr(:call, args...)))

(@macro def(x, y)
 esc(Expr(:(=), x, y)))

(@macro var"if"(cond, yes, no=nothing)
 esc(Expr(:if, cond, yes, no)))

(@macro block(args...)
 esc(Expr(:block, args...)))

(@macro var"let"(bindings, body...)
 (@block
  argv = (@if isexpr(bindings, :tuple)
          bindings.args
          [bindings])
  esc(Expr(:let, argv..., Expr(:block, body...)))))

(@macro fn(sig, body...)
 esc(Expr(:(=), sig, Expr(:block, body...))))

(@macro var"for"(spec, body...)
 (@block
  (@def newspec (@if (spec.args[1] == :∈)
                 Expr(:(=), spec.args[2:end]...)
                 spec))
  esc(Expr(:for, newspec, Expr(:block, body...)))))


(@fn f(x::T) where {T}
   (@let (@def y x)
      T(y) + 1))

[mbaz]

This change would make long @plot commands in Gaston.jl more comfortable to use. Currently, one must write multiline commands like this:

@plot(settings,
      {title = Q"Two sinusoids", key = "columns 1", key = "box outside right top"},
      x, y,
      plotline, {title = "'sin'"})

whereas the proposed change would allow:

(@plot settings
       {title = Q"Two sinusoids", key = "columns 1", key = "box outside right top"}
       x y
       plotline {title = "'sin'"})

It's not a big difference but I think the additional convenience is worth it.

[c42f]

Hey, I think the general approach makes sense. Compatibility is tricky as always.

parse_brackets() is also used for parse_vect(), so for x[a,b] and x{a,b} and [a,b] syntax and there's some extra test cases to add and edge cases to check for sanity. Detecting indexing vs concatenation syntax within [] and {} has always been a bit fragile. Eg, after this patch the following might be an inconsistency:

# old,new: parsed as vcat?
[@foo x
y]

# old: syntax error.  new: parsed as `vect`?
[x,
@foo y
z]

It may be best to only allow macro_whitespace_newline=true inside round brackets.

[MasonProtter]

Ah good point @c42f. I've fixed that and added a test-case for it.

[MasonProtter]

friendly ping @mlechu / @c42f

[mlechu]

I'm a fan of the syntax! I'm not sure we can have macrocall parsing continue eagerly in all round parens, though, since I can find contexts where nightly treats the newline as the end of the macrocall without any parse error:

julia> Meta.show_sexpr(Base.JuliaSyntax.parsestmt(Expr,
              """
              f(a, b, @bar 1
              , 2)
              """))
(:call, :f, :a, :b, (:macrocall, Symbol("@bar"), :(#= line 1 =#), 1), 2) # nightly
(:call, :f, :a, :b, (:macrocall, Symbol("@bar"), :(#= line 1 =#), (:tuple, 1, 2))) # this PR

# similar for tuple or `@x()` parens

There's also this, but IMO this is a JuliaSyntax bug given the ambiguity (flisp produces a parse error).

julia> Meta.show_sexpr(Base.JuliaSyntax.parsestmt(Expr,
              """
              f(a, b, @bar 1
              + 2)
              """))
(:call, :f, :a, :b, (:call, :+, (:macrocall, Symbol("@bar"), :(#= line 1 =#), 1), 2)) # nightly
(:call, :f, :a, :b, (:macrocall, Symbol("@bar"), :(#= line 1 =#), (:call, :+, 1, 2))) # this PR

I see you've written a test for this case:

julia> Meta.show_sexpr(Base.JuliaSyntax.parsestmt(Expr,
                     """
                     (@foo @bar x y
                     z)
                     """))
(:macrocall, Symbol("@foo"), :(#= line 1 =#), (:macrocall, Symbol("@bar"), :(#= line 1 =#), :x, :y, :z)) # this PR

but should we consider having z be an arg of foo and not bar?

[MasonProtter]

Ah okay, good catches @mlechu. I have fixed points 1. and 3. that you raised and added tests for them.

Should the 2nd case you rasied just be its own PR, or should that be handled here?

[mlechu]

I think we also want to stop parsing the macrocall at the end of the line here, even if it would be an error in this case, since these similar expressions parse on nightly:

(1 + @foo x
 + 2)

(1 + 
 @foo x
 + 2)

# parse to (+ 1 (@foo x) 2)

Would it be reasonable to have the new behaviour only apply to macrocalls whose immediate container is K"("?

[MasonProtter]

sorry @mlechu, I wrote a fix and then forgot to commit and send it 🤦.

That seems more or less sensible to me. I've updated it to behave as you say:

julia> parsestmt(Expr, "(1 + @foo x\n + y)")
:(1 + #= line 1 =# @foo(x) + y)

and added some tests for it.