Make Behat work again (quick workaround)

[kabalin]

Most moodle-plugin-ci users are experiencing issues with Behat tests which stopped working at some point last year. While PR #107 is under consideration, this is a quick workaround to make it work again:

language: php

addons:
  postgresql: "9.4"

services:
- mysql
- postgresql
- docker

cache:
  directories:
  - $HOME/.composer/cache
  - $HOME/.npm

php:
- 7.1
- 7.2
- 7.3

env:
  global:
  # Set PROFILE to either chrome or firefox depending on your preference.
  - PROFILE=chrome
  # This prevents starting selenium and php servers using moodle-plugin-ci in-built functionality.
  - MOODLE_START_BEHAT_SERVERS=NO
  matrix:
  - DB=pgsql
  - DB=mysqli

before_install:
- phpenv config-rm xdebug.ini
- nvm install 8.9
- nvm use 8.9
- cd ../..
- composer create-project -n --no-dev --prefer-dist blackboard-open-source/moodle-plugin-ci ci ^2
- export PATH="$(cd ci/bin; pwd):$(cd ci/vendor/bin; pwd):$PATH"

install:
- moodle-plugin-ci install -vvv
# Start Selenuim Standalone server using docker.
- docker run -d -p 127.0.0.1:4444:4444 --net=host -v /dev/shm:/dev/shm -v $HOME/build/moodle:$HOME/build/moodle selenium/standalone-${PROFILE}:3
# Start php built-in server.
- php -S localhost:8000 -t $HOME/build/moodle > /dev/null 2>&1 &
# Wait to make sure docker container is fully running.
- sleep 10

script:
- moodle-plugin-ci phplint
- moodle-plugin-ci phpcpd
- moodle-plugin-ci phpmd
- moodle-plugin-ci codechecker
- moodle-plugin-ci validate
- moodle-plugin-ci savepoints
- moodle-plugin-ci mustache
- moodle-plugin-ci grunt
- moodle-plugin-ci phpdoc
- moodle-plugin-ci phpunit
- moodle-plugin-ci behat --profile $PROFILE

To sum up, we use MOODLE_START_BEHAT_SERVERS=NO to disable built-in selenium and php server initialisation, and then start respective services in install stage after installation step. Selenuim server is initialised using docker image.

Please note, this requires Ubuntu Xenial build environment (currently default one in Travis).

[timhunt]

Thanks for this tip @kabalin, which almost worked to get the qtype_stack Behat tests passing again. However, it did not work: https://travis-ci.org/timhunt/moodle-qtype_stack/jobs/653818879

2 errors:

1. [WebDriver\Exception\CurlExec (-1)]
   599 Webdriver http error: 404, payload :{
   600 "sessionId": "cookie",
   601 "value": {
   602 "error": "invalid session id",
   603 "message": "No active session with ID cookie",
   604 "stacktrace": ""
   605 },
   606 "status": 6
   607 }

2. 001 Scenario: Restore the STACK demo course. # /home/travis/build/moodle/question/type/stack/tests/behat/restore_demo.feature:15
   689 And I upload "question/type/stack/samplequestions/STACK-demo.mbz" file to "Files" filemanager # /home/travis/build/moodle/question/type/stack/tests/behat/restore_demo.feature:11
   690 Exception: invalid argument: File not found : /home/travis/build/moodle/question/type/stack/samplequestions/STACK-demo.mbz

Have you seen these before? Do you know what they mean?

[kabalin]

Hello @timhunt! No, I did not see similar errors yet. Regarding the first one, I see in the CI log that you run behat test immediately following webserver and senenium container start. I think it could be that behat test is executed before Selenium container is fully ready (as you see it does not fail on the re-run)? I tried adding sleep 10 before leaving install stage (I added this to example above as well), and this seems helped:

https://travis-ci.org/kabalin/moodle-qtype_stack/jobs/653942531

Notice, I also removed some obsolete dependencies in your travis file, we do not need Behat related packages since we run it in container and also Java 9 is not required (we use default one which is version 11, mustache issue associated with this was addressed in #91). You can keep bionic requirement, I removed it only to test if this will resolve your second error.

Regarding second error, not sure why this does not work, worth trying with Firefox driver may be to confirm?

[kabalin]

001 Scenario: Restore the STACK demo course. # /home/travis/build/moodle/question/type/stack/tests/behat/restore_demo.feature:15
689 And I upload "question/type/stack/samplequestions/STACK-demo.mbz" file to "Files" filemanager # /home/travis/build/moodle/question/type/stack/tests/behat/restore_demo.feature:11
690 Exception: invalid argument: File not found : /home/travis/build/moodle/question/type/stack/samplequestions/STACK-demo.mbz

@timhunt after some trial and error this issue has been fixed by mounting Moodle directory inside selenium container, i.e.:

- docker run -d -p 4444:4444 --net=host -v /dev/shm:/dev/shm -v $HOME/build/moodle:$HOME/build/moodle selenium/standalone-chrome:3

Looks like webdriver expects file to be local when upload step is used, so when we are using docker container, it is obviously not there. Here is a successful build of qtype_stack using this change: https://travis-ci.org/github/kabalin/moodle-qtype_stack/jobs/661027012

[timhunt]

@kabalin I just wanted to say a huge "thank you" to you for working this out and sharing it. I have been making heavy use of it today, verifying my plugins for Moodle 3.9.

[kabalin]

Not at all @timhunt, glad it was helpful 😃