opam list: unexpected interaction between --latests-only and --depends-on

[kit-ty-kate]

The new --latests-only parameter added in #5375 plays in unexpected ways with "package selectors" such as --depends-on or --required-by.

Given "package selectors" and "state selectors" are separate, they have a fixed order. Currently "package selectors" are ran first then "state selectors" are applied. However i would argue that the distinction is not well defined and should be revised before the release of 2.4 to make sure we don't break the behaviour of the new parameter. EDIT: my bad the new argument was added in 2.3, not 2.4

Example

opam list --latests-only --depends-on base --all-versions --short pcre returns pcre.7.4.4 despite pcre.8.0.2 being the latest version and the position of the --latest-only parameter doesn't matter.
I would argue that the expected behaviour of such a set of parameter is to instead first filter the latest packages then to see which packages in these depend on base

[kit-ty-kate]

Notes from dev meeting: we agreed on merging the two types of selectors (though we should keep the set-order behaviour when OPAMCLI < 2.4), and change the list of parameters given by OpamArg.package_selection by passing a new ~admin parameter for whether or not it is meant to be used with opam list or opam admin list

[rjbou]

The reasoning that is that some parameters are CLI position sensitive (OpamCommand.list'state_selector) while others precedence is hardcoded (OpamArg.package_selection). It is better to harmonise both and have them all CLI position sensitive to give more ways to the user to query with opam list.
There is no emergency on having it in 2.4 as --latests-only has been introduced in 2.3, but it would be a good to have.

[arozovyk]

Just some notes after our discussion:
We can currently have vflag_all and vopt_allposition sensitive but not a mix of both opt_all + vflag_all since such combination is not allowed by Cmdliners API.
The other option would be to retreive the flag position and use that for precedence definition, but that is also not something that Cmdliner API currently provides.

[arozovyk]

https://github.com/arozovyk/cmdliner/tree/opt_vflag_all.1.3.0

To be used if we do end up forking cmdliner

[dbuenzli]

I would rather entice you to devise a usable command line. Optional arguments are not made to be ordered.

[kit-ty-kate]

As an alternative to making everything position sensitive, we could add a new opam query command which would take a selection formula (e.g. opam query 'name-match(pcre) & latests-only & depends-on(base)') and we could change the behaviour of --latests-only to a constant for a given switch, similar to the other "state selectors".

This would also be interesting to allow things that were previously impossible such as using disjunctions only in places instead of everywhere at once like what opam list --or does.

[dbuenzli]

My rule of thumb is that once you start ordering beyond a single option name (e.g. -I) things get a bit hard to understand because it's difficult to have in the head which options are ordered among each other and those that are not.

That being said I'm not even sure I actually got what the problem is because I don't even understand what this is supposed to return:

opam list --latests-only --depends-on base --all-versions --short pcre

Wouldn't the problem be solved by having a token in version numbers that represents latest versions ? I would understand these invocations:

1. opam list --depends-on base.^latest --all-versions pcre

2. opam list --depends-on base --latest-only pcre

3. opam list --depends-on base.^latest --latest-only pcre

––

1. List all versions of pcre that depend on the latest version of base
2. List the last version of pcre that depends on base
3. List the last version of pcre that depends on the last version of pcre

By understanding the semantics of opam list as follows:

1. Lists all versioned packages given on the command line or all of them if unspecified.
2. Options are filters/predicates on the elements of that list.

So an option like --latest-only only ever applies to the members of the list of packages, not on other optional predicates.

[kit-ty-kate]

That being said I'm not even sure I actually got what the problem is because I don't even understand what this is supposed to return

Sorry about that, i guess my example command uses too many options unrelated to the problem i was trying to describe.

I guess part of the confusion too, comes from the fact that --all-versions is not a filter but a flag changing the UI from a list of package names to a list of packages (<name>.<version>).

I've personally never understood why the default behaviour (package names) was ever desired and i've only ever used opam list together with both --all-versions and --short.

All that said, the issue might be better summarized to:

• opam list --latests-only --depends-on base currently (internally) returns pcre.7.4.4. If we try to describe what's happening it would be: opam list returns the list of packages that depend on base, then returns the latest version of each packages
• there are currently no way of first taking the latest version of each packages, then return only the packages that depend on base

With your proposal the current behaviour doesn't change, and the second could be written as

opam list --depends-on base '*.^latest'

Why not.

Another example is: what do we want opam list --available --latests-only to mean?

1. does that mean: amongst all the available packages, give me the latest version of each packages
2. or: amongst all the latest version of each packages, give me the ones that are available

We could actually use the same syntax. The behaviour of the command above could be the first one, and the second one could be written as:

opam list --available '*.^latest'

We need to think about it more but i think this could be a possible solution. Thanks!

[dbuenzli]

I think here is one way you could explain how opam list works.

opam list makes a list of versioned packages using positional arguments, keeps only the ones that satisfy a conjunction of predicates and then filters the result. More precisely:

1. Make a list of versioned packages (via positional arguments, none means all)
2. Only keep those that satisfy the conjunction of predicates (--available, --depends-on base.^latest)
3. Apply the filters to the result (--latest-only¹).

Now you need to make clear in the documentation which options are predicates and which ones are filters. But you don't have to rely on option ordering.

Footnotes

1. I think this name remains perhaps a bit ambiguous, it would be clearer do say --latest-result with synopsis For each package, keep only the latest version present in the list of results.. ↩

[dbuenzli]

But you don't have to rely on option ordering.

That is if all your filters are commutative. If not I would rather suggest to define them via a single --filter option which species the order via a comma separated list of tokens.