Speed up tests on Travis CI (when using DRMAA)

[jakirkham]

Currently, it is taking around 12 mins to complete. This is not terrible, but is twice as long as our current (master) test suite. Ideally, we want to take a comparable amount of time to what it does now. Admittedly, it may take a minute or so longer.

[jakirkham]

We have reverted any changes to the test settings this brings the runtime down a few minutes depending. It appears a sizable amount of time is spent in install (~4-5 min based on a couple of cases), which we can't do too much about.

When comparing the tests to running on my computer, it seems Travis seems to be a bit slower in most cases. Below is a graph of the ratio of Travis time vs my MacBook Pro from Late 2013. Our MacBook Pro compared to Travis CI has more memory 16GB vs 3GB and more cores 8 vs 2. However, both use solid state drives, Travis CI may have some overhead from the VM provided drive. Also, the disk space for Travis is shared between any other jobs running on the same node. Due to rudimentary profiling, we know our test suite uses no more and generally much less than 1GB RAM. That being said, here are the stats.

In order are the tests that are 5 times slower or worse. Asterisks are placed in front of those that are 10 times slower or worse. Note that these aren't particularly long tests so optimizing them may not be necessary. Also, note the bold ones use IO. These tend to be the longer running tests, as well.

tests.test_nanshe.test_io.test_hdf5.test_search.TestHDF5Searchers.test_get_matching_paths: 0.3991s / 0.0678s = 5.88643067847
nanshe.imp.renorm.renormalized_images: 0.0439s / 0.0065s = 6.75384615385
*tests.test_nanshe.test_io.test_hdf5.test_search.TestHDF5Searchers.test_get_matching_grouped_paths: 0.9712s / 0.0365s = 26.6082191781
nanshe.util.iters.cycle_generator: 0.0098s / 0.0012s = 8.16666666667
nanshe.util.iters.cumulative_generator: 0.0055s / 0.0010s = 5.5
*tests.test_nanshe.test_io.test_hdf5.test_search.TestHDF5Searchers.test_get_matching_grouped_paths_found: 2.0649s / 0.0691s = 29.8827785818
*nanshe.imp.segment.region_properties: 0.0677s / 0.0060s = 11.2833333333
*tests.test_nanshe.test_imp.test_registration.TestRegisterMeanOffsets.test10a: 0.3652s / 0.0493s = 7.40770791075

TL;DR In general, Travis can be taken to be about 2x slower than our MacBook. The causes of this are not entirely clear, but likely boil down to worse specs, VM overhead, and sharing resources with other jobs. The best way to improve this is to return to the container environment by solving this issue ( https://github.com/jakirkham/nanshe/issues/230 ), but is somewhat challenging. A quick improvement would be to cut down on unnecessary setup costs in testing. Following this would be optimizing the test themselves or the code. However, a fair bit of time was already spent on test and code optimization. So, this would take some time, as well.

[jakirkham]

Opened the following issues to look into getting gridengine and drmaa in the containers. ( travis-ci/travis-ci#3773 ) ( travis-ci/travis-ci#3774 )

[jakirkham]

Installing apt-get packages in the container environment can be accomplished this way ( http://docs.travis-ci.com/user/apt/ ).

[jakirkham]

Also, it is possible to cache our dependencies to speed up the build. ( http://docs.travis-ci.com/user/caching/ ). However, we must use a container to do this.

[jakirkham]

Other thoughts include separating tests into a build matrix. For instance, regression tests from unit tests using environment variables ( http://docs.travis-ci.com/user/build-configuration/#The-Build-Matrix ). See example. ( http://docs.travis-ci.com/user/speeding-up-the-build/#Parallelizing-your-builds-across-virtual-machines ) Combined with nose attributes, the environment variables could be simply which attribute(s) to run. ( http://nose.readthedocs.org/en/latest/plugins/attrib.html ) If this were done, it is possible to combine two coverage reports using coverage ( http://nedbatchelder.com/code/coverage/cmd.html#h_combining_data_files ). However, proper merging of results and correct submission hinges on a successful resolution to this issue ( travis-ci/travis-ci#929 ). A workaround might be to use the script from here ( https://github.com/dmakhno/travis_after_all ), but it requires the use of curl -Lo contrary to the README. Other fixes may be needed, as well.