#994 enabled continuous integration tests for ohm-website, but no one ever looks at them because there are hundreds of test failures and errors, even on the tip of staging:
2662 runs, 951341 assertions, 137 failures, 93 errors, 0 skips
We routinely merge PRs to staging and deploy to production with these failures, without really being able to tell whether an additional failure indicates a problem specific to the PR. We need to work through these failures to be able to actually make use of continuous integration.
The failures seem to fall into multiple buckets:
- Upstream tests that assume features we’ve disabled, such as Google authentication
- Upstream tests that assume translation strings we’ve customized
- Upstream tests that broke because of custom functionality we’ve added
None of the custom functionality we’ve added has automated tests, though there are some fragments of semiautomated tests in helper repositories like ohm-inspector. Much of the code is more or less untestable. We need to rely on manual testing until we can rewrite it or at least package it up properly: #914 #1058. Other custom functionality, such as the date suffixes in OpenHistoricalMap/ohm-website#277, is more testable but still lacks tests. We can track adding tests separately once the basic tests are working properly.
#994 enabled continuous integration tests for ohm-website, but no one ever looks at them because there are hundreds of test failures and errors, even on the tip of staging:
We routinely merge PRs to staging and deploy to production with these failures, without really being able to tell whether an additional failure indicates a problem specific to the PR. We need to work through these failures to be able to actually make use of continuous integration.
The failures seem to fall into multiple buckets:
None of the custom functionality we’ve added has automated tests, though there are some fragments of semiautomated tests in helper repositories like ohm-inspector. Much of the code is more or less untestable. We need to rely on manual testing until we can rewrite it or at least package it up properly: #914 #1058. Other custom functionality, such as the date suffixes in OpenHistoricalMap/ohm-website#277, is more testable but still lacks tests. We can track adding tests separately once the basic tests are working properly.