handling `let` within a pipeline step

[jariji]

It looks like this fails to expand _ inside the map call.

julia> @p begin
           rand(100)
           let m = mean(__)
               map(_ - m, __)
           end
       end
ERROR: syntax: all-underscore identifiers are write-only and their values cannot be used in expressions around

[aplavin]

I can kinda understand how you'd expect this to expand, but why write it this way?

@p let
    rand(100)
    @aside m = mean()
    map(_ - m)
end

is more straightforward IMO.

I never anticipated someone writing multi-statement let ... end as one of the pipeline steps :) Should be possible to support it if needed, but I wonder when such a style is really needed (compared to @aside)?

[jariji]

I never anticipated someone writing multi-statement let ... end as one of the pipeline steps :)

Haha, I do this all the time. It works fine:

julia> @p begin
           rand(3)
           let m = mean(__)
               map(x -> x - m, __)
           end
       end
3-element Vector{Float64}:
 -0.5591295398526472
  0.19342379352164818
  0.3657057463309993

It only fails if I use _ like in the OP example.

I can kinda understand how you'd expect this to expand, but why write it this way?

@aside "is convenient to compute intermediate results" but that's already what let is for, so I just keep using let the same way I always do, without introducing a new keyword.

[aplavin]

keep using let the same way I always do, without introducing a new keyword

Generally, I support this approach, but in these scenarios I don't really like let as it introduces yet another nesting level. Both visually and conceptually in the code.

Didn't know it "kinda works" already! "Kinda" because @p treats the whole let block as a single step (makes sense from the code structure PoV) and doesn't add __ automatically as seen in your example, and doesn't process _ into lambdas either.
I guess, you propose for @p to descend into let blocks and transform each line within them as if it was a line at the top level of @p block?

[jariji]

A let block has two sections: the initial line of comma-separated definitions, and the following procedure block.

• In the definition line, __ should expand to the value of the line before the let but it shouldn't insert __ implicitly.
• In each line of the procedure block, __ should be inserted and expanded as if it were at the top level of a @p block.
• _ should expand throughout.

[aplavin]

Actually, that's a neat idea, I like it! I just never thought of handling subparts of expressions within the pipeline.

Just a question on proposed semantics details:

In the definition line, __ should expand to the value of the line before the let but it shouldn't insert __ implicitly.

Why not insert __?

In each line of the procedure block, __ should be inserted and expanded as if it were at the top level of a @p block.

Also, why not insert it automatically?

[jariji]

In the definition line, __ should expand to the value of the line before the let but it shouldn't insert __ implicitly.

Why not insert __?

let's definition block is about creating a new scope, so it seems natural to me that it doesn't take any "arguments".

@p begin
  rand(3)
  let x = 1 * 2 * 3
    prod(__) * x
  end
end

accidentally putting the data into the first line seems unintuitive and bad.

In each line of the procedure block, __ should be inserted and expanded as if it were at the top level of a @p block.

Also, why not insert it automatically?

Yes I mean do insert __ automatically, here, taking the value from before the let block as the argument on the first proc line.

[jariji]

foo = StructArray(a=[1,2,3])
bar = StructArray(a=[10,20,30])

@p begin
    foo
    @_ @aside mbar = maximum(bar.a)
    map((;_.a, b=mbar-_.a))
end

I have two datasets and in the middle of the foo pipeline I want to calculate the maximum from bar. I can use @_ but the definitions block of let seems intuitive to me as a place for definitions that don't receive implicit __.

@p begin
    foo
    let mbar = maximum(bar.a) # no implicit __
        map((;_.a, b=mbar-_.a))  # implicit __ = foo
    end
end