[Logging] Add support to the Open Test Reporting format

[mpiot]

PHPUnit have a log format: OTR (Open Test Reporting) (https://github.com/ota4j-team/open-test-reporting), this one should became the JUnit replacer and the Junit format could be removed in PHPUnit 14 (ota4j-team/open-test-reporting#867 (comment)).

Currently, Paratest does not offer this format.

[Slamdunk]

Hi, I've looked into OTR and how XML are generated.

The most difficult issue to fix is going to be the id and parentId remap while merging multiple OTR xmls: the same test suite, or even the same test in --functional mode, is going to have a different id in different processes.

And the event names are not unique, so an internal composite unique name that concatenates the full hierarchy is needed, to recreate the expected combined tree.

I'll try to implement something, with no promise

[mpiot]

Hi, thanks a lot :)
Maybe the id can be replaced by a combination of: parents names + name ? (long unique string)

But... Each event have their own time, this should be difficult to merge them properly because many same parents will have different times... Generating a hierarchy format instead of event-based format should not fix it to: start are logged in both format, and duration are processed.

[sebastianbergmann]

The most difficult issue to fix is going to be the id and parentId remap while merging multiple OTR xmls: the same test suite, or even the same test in --functional mode, is going to have a different id in different processes.

The ota4j-team/open-test-reporting project has a CLI tool for converting from the event-based to the hierarchical format which is closer to what you're used to from JUnit XML.

However, I think that this tool should also support merging OTR XML logfiles. @marcphilipp, what do you think?

[marcphilipp]

The scope of an XML file is currently a single execution. In JUnit/JVM terms, that's a single JVM process running tests. In the JVM ecosystem, there are two ways of running tests in parallel: in-process using threads and multi-process by spawning multiple JVMs. While there's currently no way of merging XML files, the HTML report supports including results from an arbitrary number of XML files.

I'm not familiar with how paratest works. It sounds like it follows a multi-process approach? What's the benefit of merging the XML files? Wouldn't it be sufficient to merge when generating an HTML report?

[Slamdunk]

What's the benefit of merging the XML files? Wouldn't it be sufficient to merge when generating an HTML report?

The XML is consumed by many other tools, like a code-coverage reporting tool, or mutation testing tools that run after the main test automation framework.
The HTML may not be the final goal; it may be never needed.

From the perspective of a developer, it should be seamless to run the test suite in a single process or in multiple processes at once: hence the need for a way to run subsets of the test suite and at the end merge all the partial XML files back into a single complete XML, expected to be identical for both the types of run.