catkin run_tests should not return 0 if there are failing tests

[scpeters]

Currently catkin run_tests will return 0 even if there are failing tests (for example see this travis build that had a test failure). This behavior is probably because run_tests is an alias for catkin_make run_tests. It would be nice if catkin run_tests could return a better error code.

In the meantime, I am calling catkin_test_results after catkin run_tests.

[jbohren]

I think this is the "correct" behavior. As explained here: ros/catkin#576

You have to run catkin_test_results. In order for run_tests to run all tests even if some fail part way through it cannot return a different return code. You can use the CTest target, test, also which will return the code you are looking for but will not generate correct test XML output.

If we add a non-alias full-fledged test verb at some point, then I could see it having options to control the behavior. Until then, I feel like it's best to preserve the behavior of catkin_make in this regard.

[wjwwood]

I actually think that the test verb should default to sending returning an error code (perhaps by defaulting to running catkin_test_results at the end and returning the return code of that). The fact that the run_tests target cannot return an error code when a test fails as well as run all the tests is a technical limitation of how make targets are exposed in CMake. CTest can work around this issue by being the single point of entry for running all the tests, but this only works when building in one cmake/make cycle with catkin_make and wouldn't be a full solution for catkin build or catkin_make_isolated.

[SoftwareApe]

This is a serious problem. We can basically not trust out Jenkins pipeline because of this.
Is there a way to at least systematically obtain the error code if a test failed?

[SoftwareApe]

catkin_test_results says "Summary: 0 tests, 0 errors, 0 failures, 0 skipped", so one would need to diff the number of tests here with the number of tests found, which seems like an unreliable method as well.

[SoftwareApe]

For anybody else who's unhappy with this, gtest creates:
MISSING-gtest-*.xml test reports

This way you can temporarily work around this deficiency.