DataFrameMacros.jl and DataFramesMeta.jl

[bkamins]

@jkrumbiegel and @pdeffebach - I know you are in contact and I really believe that having both https://github.com/jkrumbiegel/DataFrameMacros.jl and https://github.com/JuliaData/DataFramesMeta.jl is a good thing at this stage of ecosystem maturity that can help us all to converge to the most user-friendly designs (even maybe in some scenarios one or the other can be preferred?).

What I feel would be really great is that we could jointly come up with some guidance for the users comparing both and putting them in the documentation of DataFrames.jl in https://dataframes.juliadata.org/latest/man/querying_frameworks/ section (ideally before JuliaCon2021).

I know you both are really committed to supporting JuliaData so I hope this is something that is doable. My mindset is the following:

• DataFrames.jl syntax is clearly not very user friendly (unfortunately given the design decisions we made it has to be)
• therefore we need the Meta-package that newcomers to DataFrames.jl can use without having to go through a steep learning curve of DataFrames.jl;
• having said that - I am afraid that we should give them some guidance of DataFrameMacros.jl vs DataFramesMeta.jl as otherwise they might be confused (I do not say we should recommend one or the other - probably we should point out the differences and help them choose)

I would be obliged if you commented what you think here. Thank you!

[jkrumbiegel]

I would currently characterize the difference such that DataFramesMeta is closer to how DataFrames is built. I've made DataFrameMacros so that it's most convenient for me personally, and I don't shy away from shorthands such as @r, @c and @t, but I assume this could overlap with the requirements of a lot of people that don't do the most demanding analyses but who want simple and clear syntax in the most common cases.

Choosing row-based transforms etc. by default brings the syntax a bit closer to what people might be used to from R, as their vectorized primitives always look "scalar-like". I think because we do have fast loops it's actually a boon that we can write row-based logic which is still fast. I especially enjoy that when working with strings etc. I also like the @m flag which makes it easier to write if anything missing then missing else ... kind of logic.

The differences could change a bit in the future, as Peter said that DataFramesMeta plans to make some changes that could bring both packages closer together.

[pdeffebach]

In terms of concrete changes that should be happening soon

• :x instead of x.
• A flag for treated missing as false, preferrably recursively, see here
• A flag for passmissing, like @m
• An @astable flag, like @t, see an early attempt here
• $ instead of cols. It seems very feasible, I've explored some niche effects to consider here.

These are on the list, and I understand you have been frustrated with the slow pace of changes. Hopefully I can pick up the pace a bit to get these things merged in.

The main sticking point is row-wise by default. Would you be okay with @rtransform etc? Or do you think that as long as @transform is column-wise by default in DataFramesMeta.jl, you will always want to maintain DataFramesMacros?

EDIT: One other thing. Currently, DataFramesMeta.jl is very hard to make changes to because it was written without MacroTools. Do you think if we re-write some parts of the package using MacroTools.jl, it would lower the barrier of entry to contributing? Then it might be easier for you to help implement the changes you want.

[bkamins]

Thank you both for commenting. If I understand things correctly @pdeffebach is planning to get at almost the same design as DataFrameMacros.jl in the long term (except for some details in syntax). Is this correct?

If this is so then the question to @jkrumbiegel is - how would you prefer both packages to be promoted in DataFrames.jl documentation (or maybe you see DataFrameMacros.jl as your personal project for exploring things and of course anyone is welcome to use it, but at the same time you could support development of DataFramesMeta.jl to reach the state of the same usability as DataFrameMacros.jl?).

---

My personal opinion on what @pdeffebach plans:

:x instead of x.

agreed

A flag for treated missing as false, preferrably recursively

agreed

A flag for passmissing

agreed

An @astable flag, like @t

agreed

$ instead of cols.

agreed

---

A side question to you both. If I want a target column name as e.g. "sum x" (with space), how would that be expressed in both packages in their target design? Thank you!

[jkrumbiegel]

or maybe you see DataFrameMacros.jl as your personal project for exploring things and of course anyone is welcome to use it, but at the same time you could support development of DataFramesMeta.jl to reach the state of the same usability as DataFrameMacros.jl?

Yeah I support that statement, if DataFramesMeta ends up so similar to what I have now that there's almost no point in having two packages, that's fine by me. I can of course always use what I want, which is still nice to have. I often tend towards really short and sweet solutions, and I understand that others don't want to go that way and be more verbose, yet maybe less ambiguous.

I think that the backend design of my package isn't actually too bad and that it supports some things that DataFramesMeta didn't, so if those features are copied, also good.

About the side question, DataFrameMacros supports anything left of the = sign, this is just copied as-is to the target field. So to use a string, just do:

@transform(df, "new column" = :a * :b)

[bkamins]

This is fantastic - I really believe that with this approach we can get very far.

I understand that others don't want to go that way and be more verbose, yet maybe less ambiguous.

Indeed in DataFramesMeta.jl I think that we would probably prefer something that is easy to understand (e.g. when selecting names) so that newcomers can easily pick it up, even at the expense of some verbosity (assuming that it is still terse).

so if those features are copied, also good.

I hope that @pdeffebach would be open to both using the API ideas as well as internal design (original DataFramesMeta.jl was written very long ago so redesign of internals is probably needed anyway).

@Transform(df, "new column" = :a * :b)

👍 This is fantastic

[vjd]

Thank you all for discussing this. Personally, having tried both these packages now, in addition to using DataFrames.jl as is, I feel the DataFramesMacro.jl kind of approach would definitely lower the barrier to entry as almost everyone migrating to Julia from other ecosystems would want the rowwise thinking. Having said that, I am in complete support of merging the two solutions if feasible, with a possible rewrite of DataFramesMeta, or using the DataFramesMacros as the starting point to combine Peter's ideas, especially if a rewrite of the internals are warranted for based on Bogumil's idea. I am happy to contribute in any way I can!

[matthieugomez]

DataFrameMacros.jl looks really nice (I like the row-wise default as well as the short macro flags). Since it looks like a complete rewrite also allowed @jkrumbiegel to get a much cleaner code, why not use it as a starting point instead? Sorry if I am missing something — but what is the point of doing a series of breaking changes in DataFramesMeta.jl just to end up in the same place, with a more complicated codebase?

[pdeffebach]

Maybe I am missing something but what is the point of doing a series of breaking changes in DataFramesMeta.jl just to end up with the same place, with a more complicated code?

I think that's exactly the point. We want to keep the user-base of DataFramesMeta happy, meaning we need to have a deprecation period for changes. This means we have to keep the old (complicated) code on hand for a while.

But yeah, we recently added MacroTools as a dependency to DataFramesMeta, so once we deprecate the old stuff a re-write that uses MacroTools.postwalk is in order, which will certainly borrow from DataFrameMacros.

EDIT: DataFramesMeta also has a ton of documentation, which is more work than the code-base itself (see the recent :x on LHS PR). Updating that iteratively seems like an easier task perhaps.

[bkamins]

@pdeffebach - based on user feedback. The feature

the row-wise default

is critical. Therefore in target design of DataFramesMeta.jl we should provide this option also (I know it was discussed but I want to stress that this is the key thing based on the feedback).

@matthieugomez The challenging part of DataFrameMacros.jl is:

All macros except @combine work row-wise by default.

i.e. it is not always row-wise and the same expression passed to @combine and @select will behave differently (which seems innocent, but I fear that in the long run it might confuse users - I know the documentation of DataFrameMacros.jl is very clear about this fact, but still I feel a bit uneasy here).
I would prefer in DataFramesMeta.jl not to have such inconsistency if possible (e.g. by using some consistent macro naming scheme).

[pdeffebach]

Putting aside the row-wise discussion. Just want to note that I just merged :y = f(:x), so we are making pretty rapid progress towards other desired API changes.

[vjd]

t is not always row-wise and the same expression passed to @combine and @select will behave differently (which seems innocent, but I fear that in the long run it might confuse users

Isn't this the default and how dplyr also works that just for summarize and select, the operations are column-wise? I am not sure why it would be confusing.

Therefore in target design of DataFramesMeta.jl we should provide this option also (I know it was discussed but I want to stress that this is the key thing based on the feedback).

Absolutely agree.

but what is the point of doing a series of breaking changes in DataFramesMeta.jl just to end up in the same place, with a more complicated codebase?

I sort of agree with this comment. Just trying to understand. If it is just about documentation, which is essential and the core element of any package, perhaps the work being put into re-writing can be spent on documentation. OTOH, I do see the fact that there may be many existing users who need to be kept in mind. Difficult decision, but I think sometimes starting with a clean slate is not too bad. I have a lot of respect and appreciation to the amount of work that went in all these years into DataFramesMeta.