QA Guidelines

Keeping bugs out of our codebase is essential to reducing frustration and making sure the product stays functional in Production. All testing should be done on both development and testing servers. Testing is split into two domains; Python testing and Javascript Testing.

Before Submitting a PR

1. Manually review/read over the code to check for bugs/syntax errors.
2. Check that the product is still functional by running the Django server and testing the affected page(s).
3. Run jstest if the change is a JavaScript one.
4. Deploy and test your changes on the AWS Test environment; check the logs for potential errors.

Python Testing

All Python tests for any particular app should go in the tests.py file in that application directory. Tests/testing follow the convention of the Python unittest module. In addition, Django tests can be configured to use a test database for tests that may depend on models existing. Full details on testing Django applications can be found here: https://docs.djangoproject.com/en/dev/topics/testing/overview .

JavaScript Testing

Our JavaScript tests use an environment we've set up. To access this environment in the browser, navigate to /testing/results. To access this environment in the command-line, you can use manage.py jstest. All tests are written in Jasmine and should be run before submitting a PR or deploying a branch to production/test; this ensures that new code doesn't break old functionality.

JsTest Options

Usage: manage.py jstest [options] 

Run JavaScript tests against the JsTestDriver/Jasmine setup.

Options:
  -v VERBOSITY, --verbosity=VERBOSITY
                        Verbosity level; 0=minimal output, 1=normal output,
                        2=verbose output, 3=very verbose output
  --traceback           Print traceback on exception
  -c CONFIG, --config=CONFIG
                        Specify the config file to use (default: dev).
  -t TESTS, --tests=TESTS
                        Specify specific tests specified by regex to run or
                        "all" to run all (default: all).
  -b, --browsers          Name of browsers to use for testing (default: headless
                        webdriver).
  -r, --remote          Specify whether connecting to a remote server or not
                        (default: False).
  -m MODE, --runner-mode=MODE
                        Specify the runner mode to use; QUIET or DEBUG
                        (default: QUIET)
  -l LOG, --log=LOG     Capture responses/results in the console.
  -q, --quiet           Only print results, no names
  --version             show program's version number and exit
  -h, --help            show this help message and exit

###Typical Usage

$ python manage.py jstest --browsers safari --config=mobile
Passed (7 tests): 
  Suite: Safari_53434_Mac_OS.Setup:
      Test: setup

  Suite: Safari_53434_Mac_OS.The Basics:
      Test: intentrank_more_results
      Test: intentrank_valid_json
      Test: intentrank_loading
      Test: mediator_on_off
      Test: mediator_callback
      Test: pagesTracking_initialiazed

Failed (2 tests): 
  Suite: Safari_53434_Mac_OS.Sample:
      Test: session1
          Message: updateClickStream() method does not exist

  Suite: Safari_53434_Mac_OS.Advanced:
      Test: multiple_results
          Message: timeout: timed out after 5000 msec waiting for PAGES.loadResults

It can also be run by navigating to /testing/results in the browser, specifying the settings and hitting run. Note: There is currently no way to suppress the browser from opening in the background.

JsTestDriver

JsTestDriver is the application we use to run our tests (alongside an adapter for Jasmine). You won't need to interact much with JsTestDriver as the Python scripts take care of that, but documentation for JsTestDriver can be found here: https://code.google.com/p/js-test-driver/w/list . Configuration files are written in YAML: details on how to write a config file.

Jasmine

All of our tests are written in Jasmine. Jasmine is a javascript testing framework. A quick introduction to Jasmine can be found here: http://pivotal.github.io/jasmine/ .