Skip to content

UnitTests

Jump to bottom
geisserf edited this page Jan 29, 2020 · 4 revisions

In this tutorial we will implement new functionality to the planner and we will employ test-driven development (TDD) during that process. Prost relies on the doctest framework for its unit tests.

In the following, we will implement a new action recommendation function. Action recommendation is the final THTS component called at the end of each search step. Given a search node, action recommendation recommends which child to take. Since children of decision nodes correspond to actions, THTS uses this method to decide which action is finally chosen.

What we want is an action recommendation function that recommends the first child if there is an even number of children, and the last child otherwise.

Let us first see what methods an action recommendation function has to support. The base implementation in src/search/recommendation_function.h provides multiple optional methods, such as an initial learning phase which is called before search starts (learn), and requires one mandatory method: recommend, which takes as input a search node and a vector in which the result is stored.

NOTE: before C++ added move semantics in C++11 it was common to pass objects which are expensive to copy, e.g. a vector, as an argument to the function and modify the passed object directly. While this is mostly outdated nowadays (for new code we recommend to construct the result inside the function and return it), for historic reasons some of the interfaces of the planner still follow this principle.

As a first step, we will write the declaration of our class. We create a new file src/search/bipolar_recommendation.h with the following content:

#ifndef BIPOLAR_RECOMMENDATION_H
#define BIPOLAR_RECOMMENDATION_H

#include "recommendation_function.h"

/**
 * This action recommendation recommends the first child if the number of
 * children is even, and the last child otherwise.
 */
class BipolarRecommendation : public RecommendationFunction {
public:
    BipolarRecommendation(THTS* thts) : RecommendationFunction(thts) {}

    void recommend(SearchNode const* rootNode,
                   std::vector<int>& bestActions) override;
};

#endif /* BIPOLAR_RECOMMENDATION_H */

To allow compilation, we will also write the skeleton of the function definition, but not the complete implementation. That is, we create a file src/search/bipolar_recommendation.cc with the following content:

#include "bipolar_recommendation.h"

void BipolarRecommendation::recommend(SearchNode const* /*rootNode*/,
                                      std::vector<int>& /*bestActions*/) {}

We add the source to src/search/CMakeLists.txt (add bipolar_recommendation to the sources following set(SEARCH-SOURCES , note that the list is alphabetical) and compile the planner to see if our declaration is well-defined.

Remember that we follow the test-driven development cycle. In TDD, once the feature specification is clear (which in C++ is conveniently expressed by providing the declaration) the first step is to write a test for the new feature.

We will create a separate test file in src/search/tests/bipolar_recommendation_test.cc with the following content:

#include "test_utils.cc"

#include "../bipolar_recommendation.h"
#include "../thts.h"

TEST_CASE_FIXTURE(ProstUnitTest, "Testing BipolarRecommendation") {
    SearchNode* node = new SearchNode(1.0, 10);
    BipolarRecommendation recommendation(NULL);
    std::vector<int> result;
    SUBCASE("If the number of children is even the first child is chosen") {
        SearchNode* child = new SearchNode(1.0, 10);
        node->children = {child, child};
        recommendation.recommend(node, result);
    }
}

and add the corresponding entry into src/search/CMakeLists.txt:

set(SEARCH_TESTS
  ../../doctest/doctest
  tests/bipolar_recommendation_test
  tests/evaluate_test
  tests/probability_distribution_test
)

This is all we have to do to activate our tests. In particular, when we add a new test file we have to include test_utils.cc, and we have to make CMake aware of our new test. We use test fixtures in Prost, since it allows to automatically reset static class members, which would otherwise be kept between unit tests and can lead to unexpected behaviour.

It is important to note that tests are only built when compiled with the debug flag.

A basic principle of TDD is to run the newly written test before implementing anything. If the test succeeds when no implementation was written either the functionality is already provided, or the test is buggy. In general, all tests are executed after the code was succesfully compiled in debug mode. This way we do not have to manually run our tests every time and we also see if our implementation broke previous functionality.

We can also run all tests explicitly by calling the prost.py script in debug mode and adding the flag -dt-e (alternatively -dt-exit) to the end: `./prost.py --debug -dt-e'. This flag makes sure that the program exits after test execution. Otherwise, if no test fails, the planner would continue its normal behaviour (in this case, print the README!). If we only want to call a specific test, we can use the parameter -dt-tc= (-dt-test-case), then only test cases that match at least one of the patterns in the comma-separated list with wildcards will get executed.

Let us execute our newly written test in vaccum: ./prost.py --debug -dt-e -dt-tc=*Bipolar* which should print the following output:

[doctest] doctest version is "2.3.5"
[doctest] run with "--help" for options
===============================================================================
[doctest] test cases:      1 |      1 passed |      0 failed |      0 skipped
[doctest] assertions:      0 |      0 passed |      0 failed |
[doctest] Status: SUCCESS!

The test succeeds! That is certainly not what we had in mind. It turns out we forgot to add the essential part of the test: checking that the result is correct. We add two lines to the test after calling recommendation.recommend:

#include "test_utils.cc"

#include "../bipolar_recommendation.h"
#include "../thts.h"

TEST_CASE_FIXTURE(ProstUnitTest, "Testing BipolarRecommendation") {
    SearchNode* node = new SearchNode(1.0, 10);
    BipolarRecommendation recommendation(NULL);
    std::vector<int> result;
    SUBCASE("If the number of children is even the first child is chosen") {
        SearchNode* child = new SearchNode(1.0, 10);
        node->children = {child, child};
        recommendation.recommend(node, result);
        CHECK(result.size() == 1);
        CHECK(result[0] == 0);
    }
}

We compile again and now get the following output:

===============================================================================
/home/prost-user/prost/src/search/tests/bipolar_recommendation_tests.cc:6:
TEST CASE:  Testing BipolarRecommendation
  If the number of children is even the first child is chosen

/home/prost-user/prost/src/search/tests/bipolar_recommendation_tests.cc:14: ERROR: CHECK( result.size() == 1 ) is NOT correct!
  values: CHECK( 0 == 1 )

/home/prost-user/prost/src/search/tests/bipolar_recommendation_tests.cc:6: FATAL ERROR: test case CRASHED: SIGSEGV - Segmentation violation signal

===============================================================================
/home/prost-user/prost/src/search/tests/bipolar_recommendation_tests.cc:6:
TEST CASE:  Testing BipolarRecommendation

===============================================================================
[doctest] test cases:      1 |      0 passed |      1 failed |      0 skipped
[doctest] assertions:      1 |      0 passed |      1 failed |
[doctest] Status: FAILURE!
Segmentation fault

We can see that the first assertion is violated, and the test crashs afterwards because we try to access an invalid index of the vector. We can actually improve our test by replacing the first CHECK with a REQUIRE: REQUIRE(result.size() == 1);. This will result in the test immediately exiting after the first assertion is violated.

#include "test_utils.cc"

#include "../bipolar_recommendation.h"
#include "../thts.h"

TEST_CASE_FIXTURE(ProstUnitTest, "Testing BipolarRecommendation") {
    SearchNode* node = new SearchNode(1.0, 10);
    BipolarRecommendation recommendation(NULL);
    std::vector<int> result;
    SUBCASE("If the number of children is even the first child is chosen") {
        SearchNode* child = new SearchNode(1.0, 10);
        node->children = {child, child};
        recommendation.recommend(node, result);
        REQUIRE(result.size() == 1);
        CHECK(result[0] == 0);
    }
}

TEST CASE:  Testing BipolarRecommendation
  If the number of children is even the first child is chosen

/home/prost-user/prost/src/search/tests/bipolar_recommendation_test.cc:14: FATAL ERROR: REQUIRE( result.size() == 1 ) is NOT correct!
  values: REQUIRE( 0 == 1 )

===============================================================================
[doctest] test cases:      1 |      0 passed |      1 failed |      0 skipped
[doctest] assertions:      1 |      0 passed |      1 failed |
[doctest] Status: FAILURE!

Before we implement our method we write the second case of our test:

#include "test_utils.cc"

#include "../bipolar_recommendation.h"
#include "../thts.h"

TEST_CASE_FIXTURE(ProstUnitTest, "Testing BipolarRecommendation") {
    SearchNode* node = new SearchNode(1.0, 10);
    BipolarRecommendation recommendation(NULL);
    std::vector<int> result;
    SUBCASE("If the number of children is even the first child is chosen") {
        SearchNode* child = new SearchNode(1.0, 10);
        node->children = {child, child};
        recommendation.recommend(node, result);
        REQUIRE(result.size() == 1);
        CHECK(result[0] == 0);
    }
    SUBCASE("If the number of children is odd the last index is chosen") {
        SearchNode* child = new SearchNode(1.0, 10);
        node->children = {child, child, child};
        recommendation.recommend(node, result);
        REQUIRE(result.size() == 1);
        CHECK(result[0] == node->children.size() - 1);
    }
}

We now implement the logic of our function. The only goal of this step is to make our test cases pass; the code does not have to be perfect or ready to be reviewed by others. Note that we have to include thts.h, otherwise the SearchNode class is incomplete.

#include "bipolar_recommendation.h"
#include "thts.h"


void BipolarRecommendation::recommend(SearchNode const* rootNode,
                                      std::vector<int>& bestActions) {
    if (rootNode->children.size() % 2 == 0) {
        bestActions.push_back(0);
    } else {
        bestActions.push_back(rootNode->children.size());
    }
}

We compile the code again, run the tests and see that our second condition fails:

TEST CASE:  Testing BipolarRecommendation
  If the number of children is odd the last index is chosen

/home/prost-user/prost/src/search/tests/bipolar_recommendation_tests.cc:22: ERROR: CHECK( result[0] == node->children.size() - 1 ) is NOT correct!
  values: CHECK( 3 == 2 )

We realize we made a common error: the index of the last child is one less than the size of the vector. Let us fix that.

#include "bipolar_recommendation.h"
#include "thts.h"

void BipolarRecommendation::recommend(SearchNode const* rootNode,
                                      std::vector<int>& bestActions) {
    if (rootNode->children.size() % 2 == 0) {
        bestActions.push_back(0);
    } else {
        bestActions.push_back(rootNode->children.size() - 1);
    }
}

Now our tests pass:

[doctest] test cases:      1 |      1 passed |      0 failed |      0 skipped
[doctest] assertions:      4 |      4 passed |      0 failed |
[doctest] Status: SUCCESS!

Great! However, we are not finished yet. Now that our test passes, the last and most crucial step in test-driven development is to refactor the code. The wikipedia entry to TDD summarizes this step in the following:

New code can be moved from where it was convenient for passing a test to where it more logically belongs. Duplication must be removed. Object, class, module, variable and method names should clearly represent their current purpose and use, as extra functionality is added. As features are added, method bodies can get longer and other objects larger. They benefit from being split and their parts carefully named to improve readability and maintainability, which will be increasingly valuable later in the software lifecycle.

In particular, when we look at our code we can switch around the order of the conditions and remove the ugly equality check:

#include "bipolar_recommendation.h"
#include "thts.h"

void BipolarRecommendation::recommend(SearchNode const* rootNode,
                                      std::vector<int>& bestActions) {

    if (rootNode->children.size() % 2) {
        bestActions.push_back(rootNode->children.size() - 1);
    } else {
        bestActions.push_back(0);
    }
}

That looks good. We compile and run the test again to see if our refactoring broke anything... doesn't look like it!

At this point we think about border cases of our function and add new test cases making sure that our implementation behaves correctly in these scenarios. One border case is if there are no children. In this case we would expect the function to not select anything, i.e. the bestActions vector is empty. Let us add this test case to our test:

    SUBCASE("If there are no children no index is chosen") {
        recommendation.recommend(node, result);
        REQUIRE(result.size() == 0);
    }

Compiling the code and running the test again reveals that we forgot about this case:

TEST CASE:  Testing BipolarRecommendation
  If there are no children no index is chosen

/home/prost-user/prost/src/search/tests/bipolar_recommendation_tests.cc:26: FATAL ERROR: REQUIRE( result.size() == 0 ) is NOT correct!
  values: REQUIRE( 1 == 0 )

We add the corresponding check to our implementation which makes the test pass:

#include "bipolar_recommendation.h"
#include "thts.h"

void BipolarRecommendation::recommend(SearchNode const* rootNode,
                                      std::vector<int>& bestActions) {
    if (rootNode->children.size() == 0) {
        return;
    }

    if (rootNode->children.size() % 2) {
        bestActions.push_back(rootNode->children.size() - 1);
    } else {
        bestActions.push_back(0);
    }
}

Great. As before, we now take a look at our code and see if we can improve it. Indeed, we can express the empty children condition more clearly.

#include "bipolar_recommendation.h"
#include "thts.h"

void BipolarRecommendation::recommend(SearchNode const* rootNode,
                                      std::vector<int>& bestActions) {
    if (rootNode->children.empty()) {
        return;
    }

    if (rootNode->children.size() % 2) {
        bestActions.push_back(rootNode->children.size() - 1);
    } else {
        bestActions.push_back(0);
    }
}

Before we conclude this tutorial we briefly mention how we usually handle function invariants: conditions we assume to be true for the function to act correctly. For example, recommendation functions usually assume that the bestActions vector is empty beforehand. Although this does not matter for the correctness of our function, we would add an assertion check at the beginning of the function: assert(bestActions.empty()). This way incorrect behaviour is at least detected in the debug build.

We could now add more tests to check border cases (what happens if the first or last action is inapplicable?), but end the tutorial here instead. For more examples and a detailed description on the abilities of the unit test framework we refer to the doctest documentation.

Clone this wiki locally