From 04b707fb1e4dd4ef66eb0bbfb08bc24f537bfd2c Mon Sep 17 00:00:00 2001 From: Michele Mesiti Date: Fri, 4 Sep 2026 11:41:34 +0200 Subject: [PATCH 1/4] expand keypoints in collaborative CI exercise Added keypoints about GitHub Actions and CI/CD setup. --- content/full-cycle-ci.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/content/full-cycle-ci.md b/content/full-cycle-ci.md index 6edf576a..d240dee7 100644 --- a/content/full-cycle-ci.md +++ b/content/full-cycle-ci.md @@ -342,6 +342,8 @@ Your goal: ```{keypoints} +- GitHub Actions (and other services, like GitLab CI/CD pipelines) can be used to run automatically the test suite when pushing +- Setting up is usually done with YAML files. Every platform has their own syntax, but there are templates one can use - When fixing bugs or other problems reported in issues, use the issue autoclosing mechanism when you send the pull/merge request. ``` From 8a144595efd8665231c1071753058726b4f596ce Mon Sep 17 00:00:00 2001 From: Michele Mesiti Date: Wed, 9 Sep 2026 10:59:19 +0200 Subject: [PATCH 2/4] move motivations from index, lesson descr in index Related to first comment in #245. Add lesson description in index as in the git intro https://coderefinery.github.io/git-intro/ --- content/index.rst | 43 ++++++++++++++++++++++++++++--------------- content/motivation.md | 41 +++++++++++++++++++++++++++++++++-------- 2 files changed, 61 insertions(+), 23 deletions(-) diff --git a/content/index.rst b/content/index.rst index c3f6a2fd..47370dec 100644 --- a/content/index.rst +++ b/content/index.rst @@ -1,21 +1,34 @@ Automated testing - Preventing yourself and others from breaking your functioning code ====================================================================================== -Have you ever had some of these problems?: - -- You change B and C, and suddenly A doesn't work anymore. Time - wasted trying to figure out what changed. -- There was some simple problem, systematically testing could have - found it. -- You get someone else's code and are afraid to touch it because who - knows what might break. Plot twist: it's your own code! - -People have learned that some automatic way to check problems makes -software development much easier. This lesson will talk about the -places it's useful for research code, and how easy it can be. -We will discuss why testing often needs to be part of the -software development cycle and how such a cycle can be implemented. We will -see how automated testing works and practice designing and writing tests. +In this lesson we discuss the basics of automated testing. + +We start discussing why automated testing is important. +We then show how to set up automated testing in your projects +in a few programming languages, +so that you can run a test suite conveniently on your own computer. +We will then show how to make GitHub (or GitLab) +run the test suite automatically +(typically whenever someone pushes to the repository), +and tell us when there was a problem. + +If time allows, +we might do that in a collaborative fashion, +running the test suite on a pull request +to inform the code review process. + +Writing tests can be challenging sometimes, +so we will discuss typical problems in test design, +mentioning also *Test Driven Develpment*, +and practice designing and writing tests. + +The goals of the module +is to make the learners feel comfortable +with setting up a test suite of automated tests, +feel familiar with the automation options +on Software forges (e.g. GitHub and GitLab), +and make them aware of the typical challenges +in writing automated tests. .. prereq:: diff --git a/content/motivation.md b/content/motivation.md index 2224b139..71a8bcb2 100644 --- a/content/motivation.md +++ b/content/motivation.md @@ -5,8 +5,37 @@ - Understand various benefits of testing ``` +Most scientists nowadays depend on software for research. -## Untested software can be compared to uncalibrated detectors +What can go wrong when research software has bugs? Look no further: + +- [A Scientist's Nightmare: Software Problem Leads to Five Retractions](https://science.sciencemag.org/content/314/5807/1856.summary) +- [Researchers find bug in Python script may have affected hundreds of studies](https://arstechnica.com/information-technology/2019/10/chemists-discover-cross-platform-python-scripts-not-so-cross-platform/) + +How can we avoid problems like these? + +## What are typical problems that *automated* tests can address? + +Have you ever had some of these problems? + +- You change B and C, and suddenly A doesn't work anymore. Time + wasted trying to figure out what changed. +- There was some simple problem, systematically testing could have + found it. + But testing manually takes too much time, + so nobody ever did it + with the appropriate care. +- You get someone else's code and are afraid to touch it because who + knows what might break. Plot twist: it's your own code! + +People have learned that some automatic way to check problems makes +software development much easier. This lesson will talk about the +places it's useful for research code, and how easy it can be. + +Most people test their code in some way, typically manually. + + +## Untested software can be compared to uncalibrated measurement devices *"Before relying on a new experimental device, an experimental scientist always establishes its accuracy. A new detector is calibrated when the scientist @@ -17,18 +46,14 @@ calibration are compared against the expected response."* With testing, simulations and analysis using software *can* be held to the same standards as experimental measurement devices! -What can go wrong when research software has bugs? Look no further: - -- [A Scientist's Nightmare: Software Problem Leads to Five Retractions](https://science.sciencemag.org/content/314/5807/1856.summary) -- [Researchers find bug in Python script may have affected hundreds of studies](https://arstechnica.com/information-technology/2019/10/chemists-discover-cross-platform-python-scripts-not-so-cross-platform/) - --- ## Testing in a nutshell -In software tests, expected results are compared with observed results +In the most basic form of software tests, +expected results are compared with observed results in order to establish accuracy. Why are we not comparing directly all -digits with the expected result?: +digits with the expected result? ````{tabs} ```{group-tab} Python From 41d18704e9b206daed076a5959545212a6d69eb0 Mon Sep 17 00:00:00 2001 From: Michele Mesiti Date: Wed, 9 Sep 2026 12:09:38 +0200 Subject: [PATCH 3/4] problems/solutions/for whom in motivations also modified glossary --- content/motivation.md | 40 +++++++++++++++++++++++++++++++++------- 1 file changed, 33 insertions(+), 7 deletions(-) diff --git a/content/motivation.md b/content/motivation.md index 71a8bcb2..6b693523 100644 --- a/content/motivation.md +++ b/content/motivation.md @@ -104,12 +104,34 @@ CORRECT ## What can tests help you do? -**Preserving expected functionality** -- Check old things when you add new ones +```{list-table} Problems, Solutions and who is affected? +:widths: 40 30 30 +* - Problem + - Solution + - Who is affected? +* - Breaking old functionality + when adding new features + - End-to-End tests + - Developers +* - Verify installation + - Smoke tests + - Users +* - Showing up-to-date example + - End-to-End tests + - Users +* - Improve readability and names + - Unit tests + - Developers +* - Refactor and restructure + - All tests + - Developers +* - Documentation out of date + - Executable notebooks + and [nbval](https://github.com/computationalmodelling/nbval), + End-to-End tests + - Users -**Help users of your code** -- Verify it's installed correctly and works. -- See examples of what it should do. +``` **Help other developers modify it** - Change things with confidence that nothing is breaking. @@ -152,14 +174,18 @@ Use the collaborative notes to answer these questions: ## Testing vocabulary -* Test functions one at a time - **Unit tests** +* Test functions and methods one at a time - **Unit tests** * Test how parts work together - **Integration tests** -* Test the whole thing running - **End-to-end tests** +* Test the whole thing running, checking the output - **End-to-end tests** * For example, running on sample data. +* Test that the whole thing runs in the simplest scenario possible - **Smoke Test** + * if this fails, no point in testing other things, usually. + * Check results are the same as before - **Regression tests** + * Other names for the same thing: **Acceptance Tests**, **Golden-Master Tests**, **Characterization Tests** * Write test first (the output), then write code to make test pass - **Test-driven development** From bfeefb072afd091f74e00ce818317bf992d734af Mon Sep 17 00:00:00 2001 From: Michele Mesiti Date: Wed, 9 Sep 2026 12:56:55 +0200 Subject: [PATCH 4/4] review episode titles, what is CI stress automated vs non automated testing also, add definition of CI and common meaning of CI --- content/continuous-integration.md | 48 ++++++++++++++++++++----------- content/index.rst | 2 +- content/locally.md | 5 ++-- 3 files changed, 36 insertions(+), 19 deletions(-) diff --git a/content/continuous-integration.md b/content/continuous-integration.md index c59b52c1..d5217208 100644 --- a/content/continuous-integration.md +++ b/content/continuous-integration.md @@ -1,15 +1,31 @@ -# Automated testing +# Automated testing and "Continuous Integration" ```{questions} - How can we implement automatic testing each time we push changes to the repository? - Why is it good to autoclose issues with commit messages? ``` -## Continuous integration We will now learn to set up automatic tests using either GitHub Actions or GitLab CI - you can choose which one to use and instructions are provided for both. + +```{note} + +## What is Continuous integration? + +From [Wikipedia](https://en.wikipedia.org/wiki/Continuous_integration): + +> *Continuous integration (CI) is the practice +> of integrating source code changes frequently +> and ensuring that the integrated codebase is in a workable state.* + +By extension +the expression **Continous Integration** is also commonly used +to mean the automation *tools* that facilitate the practice. + +```` + This exercise can be run in "collaborative mode" by following instead the instructions in [Full-cycle collaborative workflow](./full-cycle-ci). In the collaborative version steps C-D below are performed by a collaborator. @@ -27,7 +43,7 @@ In this exercise, we will: - **F.** Create a test to increase the code coverage of our tests. ``` -### Prerequisites +## Prerequisites If you are new to Git, you can find a step-by-step guide to setting up repositories and making commits in @@ -36,9 +52,9 @@ If you are new to pull requests / merge requests, you can learn all about them in the [Collaborative Git lesson](https://coderefinery.github.io/git-collaborative/). -### Step 1: Create a new repository on GitHub/GitLab OR fork from the example repo +## Step 1: Create a new repository on GitHub/GitLab OR fork from the example repo -#### Create a new repository +### Create a new repository - Begin by creating a repository called (for example) *example-ci*. - **Before** you create the repository, select **"Initialize this repository @@ -191,14 +207,14 @@ in the [Collaborative Git lesson](https://coderefinery.github.io/git-collaborati -#### Fork and clone an existing example repository +### Fork and clone an existing example repository - Fork the example repo. There are two options one for [Python](https://github.com/AaltoRSE/PyTestingExample) and one for [R](https://github.com/AaltoRSE/RTestingExample). - Clone your fork (`git clone git@github.com:/TestingExample.git`). -### Step 2: Run tests locally +## Step 2: Run tests locally `````{tabs} ````{group-tab} Python You can now run your tests locally with @@ -216,7 +232,7 @@ in the [Collaborative Git lesson](https://coderefinery.github.io/git-collaborati -### Step 3: Enable automated testing +## Step 3: Enable automated testing `````{tabs} ````{group-tab} GitHub-Python @@ -439,7 +455,7 @@ in the [Collaborative Git lesson](https://coderefinery.github.io/git-collaborati ````` -### Step 4: Verify that tests have been automatically run +## Step 4: Verify that tests have been automatically run `````{tabs} ````{group-tab} GitHub-Python @@ -488,7 +504,7 @@ in the [Collaborative Git lesson](https://coderefinery.github.io/git-collaborati ````` -### Step 5: Add a test which reveals a problem +## Step 5: Add a test which reveals a problem After you committed the workflow file, your GitHub/GitLab repository will be ahead of your local cloned repository. Update your local cloned repository: @@ -505,14 +521,14 @@ Verify that the test suite now fails on the "Actions" tab (GitHub) or the "CI/CD->Pipelines" tab (GitLab). -### Step 6: Open an issue on GitHub/GitLab +## Step 6: Open an issue on GitHub/GitLab Open a new issue in your repository about the broken test (click the "Issues" button on GitHub or GitLab and write a title for the issue). The plan is that we will fix the issue through a pull/merge request. -### Step 7: Fix the broken test +## Step 7: Fix the broken test Now fix the code **on a new branch**, you can call it `yourname/bugfix`. After you have fixed the code on the new branch, commit the following @@ -531,7 +547,7 @@ you try to fix issue number 1). Then push to your repository. -### Step 8: Open a pull request (GitHub)/ merge request (GitLab) +## Step 8: Open a pull request (GitHub)/ merge request (GitLab) Go back to the repository on GitHub or GitLab and open a pull/merge request. **In a collaborative setting, you could request a code @@ -544,7 +560,7 @@ can still add it to the pull/merge request: `my pull/merge request title, closes #1`. -### Step 9: Accept the pull/merge request +## Step 9: Accept the pull/merge request Observe how accepting the pull/merge request automatically closes the issue (provided the commit message or the pull/merge request contained the correct issue number). @@ -555,14 +571,14 @@ See also: Discuss whether this is a useful feature. And if it is, why do you think is it useful? -### Step 10: Increase your code coverage +## Step 10: Increase your code coverage We are currently missing several functions in our tests. Write a test for the `multiply` function in a new branch and create a pull request. On Python you can directly observe the increase in code coverage. On R you can have a look at the action (`Actions -> last run of your action -> Select a job -> Test coverage`). If you compare this with the previous run, you should see an increase once the update is in. -### Step 11 (optional): Repeat steps 5-9 for the `convert_fahrenheit_to_celsius` function: +## Step 11 (optional): Repeat steps 5-9 for the `convert_fahrenheit_to_celsius` function: Repetition helps learning, so let's do the testing again for our `convert_fahrenheit_to_celsius` function. Uncomment the test for the `convert_fahrenheit_to_celsius` function and repeat steps 5 to 9 fixing the bug this test exposes. diff --git a/content/index.rst b/content/index.rst index 47370dec..fa6fd463 100644 --- a/content/index.rst +++ b/content/index.rst @@ -1,7 +1,7 @@ Automated testing - Preventing yourself and others from breaking your functioning code ====================================================================================== -In this lesson we discuss the basics of automated testing. +In this lesson we discuss the basics of *automated* testing. We start discussing why automated testing is important. We then show how to set up automated testing in your projects diff --git a/content/locally.md b/content/locally.md index 4aae85bc..400bb9a4 100644 --- a/content/locally.md +++ b/content/locally.md @@ -1,4 +1,4 @@ -# Testing locally +# Automated Testing on your computer ```{questions} - How hard is it to set up a test suite for a first unit test? @@ -9,7 +9,8 @@ In this exercise we will make a simple function and use -one of the language specific test frameworks to test it. +one of the language specific test frameworks +to test it automatically. * This is easy to use by almost any project and doesn't rely on any other servers or services.