[Proposal] Add a new `opam test` subcommand and change the behaviour of the `run-test` field

[kit-ty-kate]

The issues with the current design

Testing packages in opam isn't really a separate step of action and this can cause issues for some packages.

For instance, for some packages (key packages mainly), their tests do not get defined in opam-repository because they require packages that require the package being tested (cyclic dependency). e.g. base, odoc, …
Some packages also do not define their tests for other reasons but opam cannot distinguish between "has test" and "does not have test" and thus users using opam install --with-test <pkg> won't know if <pkg> has really been tested.

The proposal

To help with those issues my proposal would be to add a new test subcommand such as opam test <pkg> would:

• Install the dependencies and test dependencies
• If <pkg> is already installed and {with-test} is used inside the build step:
  • then rebuild <pkg> without reinstalling it
  • else: do not rebuild it
• If <pkg> is not installed, install it with with-test = true
• Wait for everything to be installed (including post dependencies)
• Run the run-test phase (wasn't used in any previous step)
• If no tests has been ran, opam would print some kind of a note to notify the user of that.

The new run-test step is a change of behaviour from previous versions of opam where the run-test phase was ran after the build phase and before the install phase. Now with this proposal, this phase is never ran through opam install but only through the new opam test command.

[dra27]

This feels very like the opam build experiment - moving opam into an area it simply doesn't belong in. A little like the exception type in OCaml before extensible-variants, --with-test and --with-doc are both poor person's package variables (which are on the slate for "opam.2.soon"!). A few thoughts:

• The second "if pkg is already installed" stage can, I think, already be achieved by splitting the package into foo and foo-tests (potentially with an alteration to a package's test suite to test installed binary artefacts, rather than build ones - although with Dune I think would be transparent). Either way, it feels like packages should be bending to what opam offers, rather than the other way around!
• I'm not sure you can come up with a mechanical definition of what tests being run would entail - opam doesn't know if a build has done anything meaningful either. For example, I notice batteries defines test dependencies but doesn't actually have [make "test"] {with-test} in its build recipe. However, it's not necessarily a bug - it has a configure command and there's absolutely no way opam could know what the [make "all"] it does have would also run the testsuite if the test dependencies are installed.
• It could be worth displaying notes/warnings when --with-test is specified but packages have neither test dependencies nor any parts of a build recipe guarded with with-test

[rjbou]

This is a good candidate for a plugin, using the x-* fields.

[rjbou]

related to #4099 ?