From be61aa47ee1073d73df6b1df8fdf9b91faa46675 Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Wed, 12 Jun 2024 18:38:55 +0200 Subject: [PATCH 1/6] initial version of test rendering CEP --- cep-recipe-tests.md | 205 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 205 insertions(+) create mode 100644 cep-recipe-tests.md diff --git a/cep-recipe-tests.md b/cep-recipe-tests.md new file mode 100644 index 00000000..293fd2bb --- /dev/null +++ b/cep-recipe-tests.md @@ -0,0 +1,205 @@ +# A new recipe format – part 5 - tests defined in the recipe + + + + + + + + + +
Title A new recipe format – part 5 - tests defined in the recipe
Status Draft
Author(s) Wolf Vollprecht <wolf@prefix.dev>
Created Jun 12, 2024
Updated Jun 12, 2024
Discussion https://github.com/conda-incubator/ceps/pull/??
Implementation https://github.com/prefix-dev/rattler-build
+ +## Abstract + +Testing is an important part of the packaging workflow - the goal is to make sure that packages +work correctly and package the files they are supposed to contain. + +In the CEP we want to clearly define how tests can be declared in the `recipe.yaml` file and how they +are stored in the final package. + +## Tests definition + +With the exception of the package content tests, this is a recap of the previous CEPs (CEP 14). + +The recipes can contain a `tests` section that is a list of individual tests that should be executed by the package build program after the package is built. + +```yaml +tests: + - script: # following the script definition in the CEP-14 + - echo "Hello World" + requirements: + build: # [MatchSpec] + - ${{ "qemu" if target_platform != build_platform }} + run: # [MatchSpec] + - python + - pytest # match spec + files: + # files to be copied to the package for test execution + source: # GlobVec + - tests/** + # Additional files from the recipe directory + # recipe: + # - ... + - python: + imports: # [string] + - foo # python imports to test + pip_check: true # bool + - package_contents: + include: # [list of globs] + - foo.h + - foo.hpp + - foo/*.hpp + - downstream: bar # string +``` + +For `script`, `python` and `downstream` tests, please refer to CEP-14. +The `package_contents` test is a new test type that runs before package creation and checks if the files listed are present (or not present) in the package. + +## Test rendering to the final package + +Tests should be shipped as part of the package in order to facilitate running tests outside of the package build execution, and as part of e.g. downstream tests. + +For this reason, tests should be rendered as part of the package. However, the rendered tests should still contain the original test definition including the un-evaluated Jinja selectors and if/else sections - in order to facilitate running tests in cross-compilation settings. + +A build tool SHOULD render the list of tests into a JSON file in the `info/` folder of the package. The file should be named `tests.json`. It should contain a list of dictionaries, where each dictionary is a test definition. + +- Any Jinja or conditionals should be kept in-tact and evaluated at runtime. (Question: should the list of tests be filtered? Should we introduce a `skip` key instead?) +- The package content tests are filtered +- The source files for the command tests are copied into the `/etc/conda/test-files///` folder and referenced from the `tests.json` file. This is done in order to keep the `info`-folder small for cases when only metadata is requested. + +```js +[ + { + script: ['echo "Hello World"'], + requirements: { + build: ["${{ 'qemu' if target_platform != build_platform }}"], + run: ["python", "pytest", { if: "win", then: "pytest-windows" }], + }, + }, + { + python: { + imports: ["foo"], + pip_check: true, + }, + }, + { + downstream: "bar", + }, +]; +``` + +### Test execution + +#### Script test + +A script test SHOULD create the `run` and `build` environment. The `build` environment SHOULD be in the architecture of the current platform. The `run` environment SHOULD be in the architecture of the target platform (ie. the platform the package is intended to run on). + +The `build` environment SHOULD be stacked on top of the `run` environment (ie. the PATH entries of the `build` environment take precedence). +The script MUST be executed with the current work dir set to a copy of of the folder of files copied for the test (ie. the `$PREFIX/etc/conda/test-files///` folder). + +## Package contents test + +A list of `globs` is used to check for the presence (or absence) of files in the package. + +The full definition of a package content test is given below: + +```yaml +package_contents: + strict: bool + include: TestGlobVec + lib: TestGlobVec + bin: TestGlobVec + files: TestGlobVec + site_packages: TestGlobVec +``` + +Where the `TestGlobVec` is defined as follows: either a single glob, a list of globs or a dictionary with `exists` and `not_exists` keys, where the values are list of globs. + +For example: + +```yaml +package_contents: + include: + - foo.h + - foo.hpp + - foo/*.hpp + lib: + exists: + - libfoo.so + not_exists: + - libbar.so +``` + +This test would check if the files `foo.h`, `foo.hpp` and any files matching `*.hpp` files are present in the package. +It would also check if `libfoo.so` is present and `libbar.so` is not present in the package. + +### The `strict` flag + +If the strict flag is set, all files included in the package MUST be matched by at least one glob. + +### The `lib` section + +The `lib` section will look in the `$PREFIX/lib` or `$PREFIX/Library/lib` (on Windows) folder. +A simple library name MUST be checked in multiple ways (e.g. `foo`) - inspired by CMake `find_library` command. + +#### Windows + +Libraries are expected to be found in both .dll and .lib formats. +Special handling is required for .dll files, which must be accompanied by a corresponding .bin file. + +If the path extension of the glob is `.dll` or `lib`, we search for + +- glob ends with `.dll`: `Library/bin/{glob}` +- glob ends with `.lib`: `Library/lib/{glob}` + +If no extension is given, the dll/lib must be search in: + +- `Library/bin/{glob}.dll` +- `Library/lib/{glob}.lib` + +#### MacOS + +On macOS the base directory that must be searched for libraries is `lib/`. +If the glob does not include `.a` or `.dylib`, the following variations are tried: + +- `lib/{,lib}{glob}.dylib` (matches `foo.dylib`, `libfoo.dylib`) +- `lib/{,lib}{glob}.*.dylib` (matches `foo.1.dylib`, `libfoo.1.dylib`) + +If the glob ends with `.a` or `.dylib`, the following variations are tried: + +- `lib/{glob}` + +#### Linux and WebAssembly + +On Linux and WebAssembly the base directory that must be searched for libraries is `lib/`. +If the glob does not end with `.a` or `.so` and does not contain `.so.`, the following variations are tried: + +- `lib/{,lib}{glob}.so` (matches `foo.so`, `libfoo.so`) +- `lib/{,lib}{glob}.so.*` (matches `foo.so.1`, `libfoo.so.1`) + +Otherwise, the following variations are tried: + +- `lib/{glob}` + +### The `bin` section + +The `bin` section will look in the `$PREFIX/bin` folder on macOS, Linux and WebAssembly. + +On Windows, the build tool must search under the following folders, with the following extensions. +If a file extension is explictly given, only that extension is searched for. + +- extensions: `.exe`, `.bat`, `.com`, `.cmd`, `.ps1` +- folders: + - `bin` + - `Scripts` + - `Library/bin` + - `Library/usr/bin` + - `Library/mingw-w64/bin` + +For example, `foo` would match `Library/bin/foo.exe`. + +### The `include` section + +Globs in the include section should match files in the `$PREFIX/include` or `$PREFIX/Library/include` (on Windows) folder. +If no file extension is supplied, `.h` and `.hpp` files must be matched. From bc1cc6fe0268507e99706c2b3920c34b29c7c55b Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Wed, 12 Jun 2024 18:50:42 +0200 Subject: [PATCH 2/6] add more info to test execution --- cep-recipe-tests.md | 149 +++++++++++++++++++++++++++++++++----------- 1 file changed, 114 insertions(+), 35 deletions(-) diff --git a/cep-recipe-tests.md b/cep-recipe-tests.md index 293fd2bb..7d46c671 100644 --- a/cep-recipe-tests.md +++ b/cep-recipe-tests.md @@ -12,17 +12,42 @@ ## Abstract -Testing is an important part of the packaging workflow - the goal is to make sure that packages -work correctly and package the files they are supposed to contain. +Testing is an important part of the packaging workflow - the goal is to make +sure that packages work correctly and package the files they are supposed to +contain. -In the CEP we want to clearly define how tests can be declared in the `recipe.yaml` file and how they -are stored in the final package. +In the CEP we want to clearly define how tests can be declared in the +`recipe.yaml` file and how they are stored in the final package. + +## History + +Conda-build has a (single) test section in the recipes. These tests are rendered +into the `info/test` folder of the package. The tests are rendered to a +`run_test.sh` or `run_test.bat` (shell) script, a `run_test.py` or `run_test.pl` +script for Python or Perl. + +An `import` test is added to the `run_test.py` script. + +Any additional test-time dependencies are added to a separata +`test_time_dependencies.json` file in the `info/test` folder. There is only a +single dependency list for all tests (making cross-compilation tests more +difficult). + +There is also only a single test definition which means it's not easy to run +multiple (independent) tests (e.g. with different sets of requirements). + +## Proposal + +The new test section should be more flexible. Test definitions should be well +separated. ## Tests definition -With the exception of the package content tests, this is a recap of the previous CEPs (CEP 14). +With the exception of the package content tests, this is a recap of the previous +CEPs (CEP 14). -The recipes can contain a `tests` section that is a list of individual tests that should be executed by the package build program after the package is built. +The recipes can contain a `tests` section that is a list of individual tests +that should be executed by the package build program after the package is built. ```yaml tests: @@ -36,7 +61,7 @@ tests: - pytest # match spec files: # files to be copied to the package for test execution - source: # GlobVec + source: # GlobVec - tests/** # Additional files from the recipe directory # recipe: @@ -53,20 +78,33 @@ tests: - downstream: bar # string ``` -For `script`, `python` and `downstream` tests, please refer to CEP-14. -The `package_contents` test is a new test type that runs before package creation and checks if the files listed are present (or not present) in the package. +For `script`, `python` and `downstream` tests, please refer to CEP-14. The +`package_contents` test is a new test type that runs before package creation and +checks if the files listed are present (or not present) in the package. ## Test rendering to the final package -Tests should be shipped as part of the package in order to facilitate running tests outside of the package build execution, and as part of e.g. downstream tests. +Tests should be shipped as part of the package in order to facilitate running +tests outside of the package build execution, and as part of e.g. downstream +tests. -For this reason, tests should be rendered as part of the package. However, the rendered tests should still contain the original test definition including the un-evaluated Jinja selectors and if/else sections - in order to facilitate running tests in cross-compilation settings. +For this reason, tests should be rendered as part of the package. However, the +rendered tests should still contain the original test definition including the +un-evaluated Jinja selectors and if/else sections - in order to facilitate +running tests in cross-compilation settings. -A build tool SHOULD render the list of tests into a JSON file in the `info/` folder of the package. The file should be named `tests.json`. It should contain a list of dictionaries, where each dictionary is a test definition. +A build tool SHOULD render the list of tests into a JSON file in the `info/` +folder of the package. The file should be named `tests.json`. It should contain +a list of dictionaries, where each dictionary is a test definition. -- Any Jinja or conditionals should be kept in-tact and evaluated at runtime. (Question: should the list of tests be filtered? Should we introduce a `skip` key instead?) +- Any Jinja or conditionals should be kept in-tact and evaluated at runtime. + (Question: should the list of tests be filtered? Should we introduce a `skip` + key instead?) - The package content tests are filtered -- The source files for the command tests are copied into the `/etc/conda/test-files///` folder and referenced from the `tests.json` file. This is done in order to keep the `info`-folder small for cases when only metadata is requested. +- The source files for the command tests are copied into the + `/etc/conda/test-files///` folder and referenced + from the `tests.json` file. This is done in order to keep the `info`-folder + small for cases when only metadata is requested. ```js [ @@ -93,14 +131,45 @@ A build tool SHOULD render the list of tests into a JSON file in the `info/` fol #### Script test -A script test SHOULD create the `run` and `build` environment. The `build` environment SHOULD be in the architecture of the current platform. The `run` environment SHOULD be in the architecture of the target platform (ie. the platform the package is intended to run on). +A script test SHOULD create the `run` and `build` environment. The `build` +environment SHOULD be in the architecture of the current platform. The `run` +environment SHOULD be in the architecture of the target platform (ie. the +platform the package is intended to run on). + +The `build` environment SHOULD be stacked on top of the `run` environment (ie. +the PATH entries of the `build` environment take precedence). The script MUST be +executed with the current work dir set to a copy of of the folder of files +copied for the test (ie. the +`$PREFIX/etc/conda/test-files///` folder). + +#### Python test + +For the Python test, a single test environment with the package should be +created. The package SHOULD depend on Python. If `pip_check` is set, `pip` +should be added as dependency to the test environment. + +The import test SHOULD be executed with the test environment activated. A file +that includes each import statement SHOULD be created and executed. The test +SHOULD succeed if the file can be executed without error. + +If `pip_check` is true, the test SHOULD also run `pip check` in the test +environment to ensure that the dependencies match the requirements of the +original Python package. + +#### Downstream test + +A downstream test SHOULD create a execute all tests of a downstream package with +the package to be tested. If dependency resolution for any of the tests fails +(e.g. due to conflicting dependencies), the test SHOULD be skipped. -The `build` environment SHOULD be stacked on top of the `run` environment (ie. the PATH entries of the `build` environment take precedence). -The script MUST be executed with the current work dir set to a copy of of the folder of files copied for the test (ie. the `$PREFIX/etc/conda/test-files///` folder). +The downstream test SHOULD then execute all tests from the downstream package in +separate test environments. The test SHOULD succeed if all tests pass. If any +test fails, the downstream test SHOULD fail. ## Package contents test -A list of `globs` is used to check for the presence (or absence) of files in the package. +A list of `globs` is used to check for the presence (or absence) of files in the +package. The full definition of a package content test is given below: @@ -114,7 +183,9 @@ package_contents: site_packages: TestGlobVec ``` -Where the `TestGlobVec` is defined as follows: either a single glob, a list of globs or a dictionary with `exists` and `not_exists` keys, where the values are list of globs. +Where the `TestGlobVec` is defined as follows: either a single glob, a list of +globs or a dictionary with `exists` and `not_exists` keys, where the values are +list of globs. For example: @@ -131,22 +202,26 @@ package_contents: - libbar.so ``` -This test would check if the files `foo.h`, `foo.hpp` and any files matching `*.hpp` files are present in the package. -It would also check if `libfoo.so` is present and `libbar.so` is not present in the package. +This test would check if the files `foo.h`, `foo.hpp` and any files matching +`*.hpp` files are present in the package. It would also check if `libfoo.so` is +present and `libbar.so` is not present in the package. ### The `strict` flag -If the strict flag is set, all files included in the package MUST be matched by at least one glob. +If the strict flag is set, all files included in the package MUST be matched by +at least one glob. ### The `lib` section -The `lib` section will look in the `$PREFIX/lib` or `$PREFIX/Library/lib` (on Windows) folder. -A simple library name MUST be checked in multiple ways (e.g. `foo`) - inspired by CMake `find_library` command. +The `lib` section will look in the `$PREFIX/lib` or `$PREFIX/Library/lib` (on +Windows) folder. A simple library name MUST be checked in multiple ways (e.g. +`foo`) - inspired by CMake `find_library` command. #### Windows -Libraries are expected to be found in both .dll and .lib formats. -Special handling is required for .dll files, which must be accompanied by a corresponding .bin file. +Libraries are expected to be found in both .dll and .lib formats. Special +handling is required for .dll files, which must be accompanied by a +corresponding .bin file. If the path extension of the glob is `.dll` or `lib`, we search for @@ -160,8 +235,8 @@ If no extension is given, the dll/lib must be search in: #### MacOS -On macOS the base directory that must be searched for libraries is `lib/`. -If the glob does not include `.a` or `.dylib`, the following variations are tried: +On macOS the base directory that must be searched for libraries is `lib/`. If +the glob does not include `.a` or `.dylib`, the following variations are tried: - `lib/{,lib}{glob}.dylib` (matches `foo.dylib`, `libfoo.dylib`) - `lib/{,lib}{glob}.*.dylib` (matches `foo.1.dylib`, `libfoo.1.dylib`) @@ -172,8 +247,9 @@ If the glob ends with `.a` or `.dylib`, the following variations are tried: #### Linux and WebAssembly -On Linux and WebAssembly the base directory that must be searched for libraries is `lib/`. -If the glob does not end with `.a` or `.so` and does not contain `.so.`, the following variations are tried: +On Linux and WebAssembly the base directory that must be searched for libraries +is `lib/`. If the glob does not end with `.a` or `.so` and does not contain +`.so.`, the following variations are tried: - `lib/{,lib}{glob}.so` (matches `foo.so`, `libfoo.so`) - `lib/{,lib}{glob}.so.*` (matches `foo.so.1`, `libfoo.so.1`) @@ -184,10 +260,12 @@ Otherwise, the following variations are tried: ### The `bin` section -The `bin` section will look in the `$PREFIX/bin` folder on macOS, Linux and WebAssembly. +The `bin` section will look in the `$PREFIX/bin` folder on macOS, Linux and +WebAssembly. -On Windows, the build tool must search under the following folders, with the following extensions. -If a file extension is explictly given, only that extension is searched for. +On Windows, the build tool must search under the following folders, with the +following extensions. If a file extension is explictly given, only that +extension is searched for. - extensions: `.exe`, `.bat`, `.com`, `.cmd`, `.ps1` - folders: @@ -201,5 +279,6 @@ For example, `foo` would match `Library/bin/foo.exe`. ### The `include` section -Globs in the include section should match files in the `$PREFIX/include` or `$PREFIX/Library/include` (on Windows) folder. -If no file extension is supplied, `.h` and `.hpp` files must be matched. +Globs in the include section should match files in the `$PREFIX/include` or +`$PREFIX/Library/include` (on Windows) folder. If no file extension is supplied, +`.h` and `.hpp` files must be matched. From ed3e27ee73a03b4e0be469c31e91a24465bd30e9 Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Thu, 13 Jun 2024 08:52:40 +0200 Subject: [PATCH 3/6] small update --- cep-recipe-tests.md | 35 +++++++++++++++++++++++++++-------- 1 file changed, 27 insertions(+), 8 deletions(-) diff --git a/cep-recipe-tests.md b/cep-recipe-tests.md index 7d46c671..5854fe08 100644 --- a/cep-recipe-tests.md +++ b/cep-recipe-tests.md @@ -97,10 +97,11 @@ A build tool SHOULD render the list of tests into a JSON file in the `info/` folder of the package. The file should be named `tests.json`. It should contain a list of dictionaries, where each dictionary is a test definition. -- Any Jinja or conditionals should be kept in-tact and evaluated at runtime. +- Any Jinja or conditionals should be kept intact and evaluated at runtime. (Question: should the list of tests be filtered? Should we introduce a `skip` key instead?) -- The package content tests are filtered +- The package content tests are filtered as they are not executed at test-time + but at build-time and we can assume that they already passed. - The source files for the command tests are copied into the `/etc/conda/test-files///` folder and referenced from the `tests.json` file. This is done in order to keep the `info`-folder @@ -138,9 +139,11 @@ platform the package is intended to run on). The `build` environment SHOULD be stacked on top of the `run` environment (ie. the PATH entries of the `build` environment take precedence). The script MUST be -executed with the current work dir set to a copy of of the folder of files -copied for the test (ie. the -`$PREFIX/etc/conda/test-files///` folder). +executed with the current work dir set to a temporary folder containing a copy +of the extra test files (from the +`$PREFIX/etc/conda/test-files///` folder). If there are +no extra test files and the folder does not exist, the current working directory +SHOULD be an empty folder. #### Python test @@ -213,9 +216,9 @@ at least one glob. ### The `lib` section -The `lib` section will look in the `$PREFIX/lib` or `$PREFIX/Library/lib` (on -Windows) folder. A simple library name MUST be checked in multiple ways (e.g. -`foo`) - inspired by CMake `find_library` command. +The `lib` section will look in the `$PREFIX/lib` folder on Unix and scan +multiple folders on Windows. A simple library name MUST be checked in multiple +ways (e.g. `foo`) - inspired by CMake `find_library` command. #### Windows @@ -233,6 +236,8 @@ If no extension is given, the dll/lib must be search in: - `Library/bin/{glob}.dll` - `Library/lib/{glob}.lib` +**Question Windows:** do we need to search in all $PATH entries for `.dll` files? + #### MacOS On macOS the base directory that must be searched for libraries is `lib/`. If @@ -277,8 +282,22 @@ extension is searched for. For example, `foo` would match `Library/bin/foo.exe`. +For `noarch` Python packages this section will also look at the +`/python-scripts` folder in the package and entry-points. + ### The `include` section Globs in the include section should match files in the `$PREFIX/include` or `$PREFIX/Library/include` (on Windows) folder. If no file extension is supplied, `.h` and `.hpp` files must be matched. + +### The `files` section + +The `files` section will look in the `$PREFIX` folder for any globs that match. + +### The site-packages section + +The site packages section will look for globs in the +`$PREFIX/lib/pythonX.X/site-packages` (on Unix) or `$PREFIX/Lib/site-packages` +(on Windows) folder. For `noarch` packages it will look into the +`/site-packages` folder. From 7f00bd28e86ebaa9fb4db2e232f20a0cced7e3ae Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Thu, 13 Jun 2024 08:55:38 +0200 Subject: [PATCH 4/6] language update --- cep-recipe-tests.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/cep-recipe-tests.md b/cep-recipe-tests.md index 5854fe08..3e922bfc 100644 --- a/cep-recipe-tests.md +++ b/cep-recipe-tests.md @@ -205,9 +205,9 @@ package_contents: - libbar.so ``` -This test would check if the files `foo.h`, `foo.hpp` and any files matching -`*.hpp` files are present in the package. It would also check if `libfoo.so` is -present and `libbar.so` is not present in the package. +This test MUST check if the files `foo.h`, `foo.hpp` and any files matching +`*.hpp` files are present in the package. It MUST also check if `libfoo.so` is +present MUST check that `libbar.so` is NOT present in the package. ### The `strict` flag From 354323176129fb567cad0d96df7a8da0360e9cf0 Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Fri, 21 Jun 2024 17:59:30 +0200 Subject: [PATCH 5/6] start list of env vars --- cep-recipe-tests.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/cep-recipe-tests.md b/cep-recipe-tests.md index 3e922bfc..2ba58de9 100644 --- a/cep-recipe-tests.md +++ b/cep-recipe-tests.md @@ -159,6 +159,17 @@ If `pip_check` is true, the test SHOULD also run `pip check` in the test environment to ensure that the dependencies match the requirements of the original Python package. +The following environment variables SHOULD be set: + +- `PKG_NAME` - the name of the package +- `PKG_VERSION` - the version of the package +- `PKG_BUILDNUM` - the build number of the package +- `PKG_BUILD_STRING` - the build number of the package +- `PREFIX` - the prefix where the package is installed +- (TODO: complete this list) + +Note: `PKG_HASH` is removed because it can't be easily extracted from `index.json` + #### Downstream test A downstream test SHOULD create a execute all tests of a downstream package with From 175a322352fbc1d9cb230f44464c7e91eceb72f4 Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Mon, 24 Jun 2024 11:15:37 +0200 Subject: [PATCH 6/6] some updates --- cep-recipe-tests.md | 65 +++++++++++++++++++++++++-------------------- 1 file changed, 36 insertions(+), 29 deletions(-) diff --git a/cep-recipe-tests.md b/cep-recipe-tests.md index 2ba58de9..8a4dc949 100644 --- a/cep-recipe-tests.md +++ b/cep-recipe-tests.md @@ -107,25 +107,27 @@ a list of dictionaries, where each dictionary is a test definition. from the `tests.json` file. This is done in order to keep the `info`-folder small for cases when only metadata is requested. -```js -[ - { - script: ['echo "Hello World"'], - requirements: { - build: ["${{ 'qemu' if target_platform != build_platform }}"], - run: ["python", "pytest", { if: "win", then: "pytest-windows" }], - }, - }, - { - python: { - imports: ["foo"], - pip_check: true, - }, - }, - { - downstream: "bar", - }, -]; +```yaml +- script: + script: + contents: + - echo "Hello World" + # this is added as the folder where the script should be executed + # (ie. where the test files were copied to) + cwd: etc/conda/test-files/// + requirements: + build: # [MatchSpec] + - ${{ "qemu" if target_platform != build_platform }} + run: # [MatchSpec] + - python + - pytest + files: + source: + - tests/** +- python: + imports: + - foo + pip_check: true ``` ### Test execution @@ -139,11 +141,12 @@ platform the package is intended to run on). The `build` environment SHOULD be stacked on top of the `run` environment (ie. the PATH entries of the `build` environment take precedence). The script MUST be -executed with the current work dir set to a temporary folder containing a copy -of the extra test files (from the -`$PREFIX/etc/conda/test-files///` folder). If there are -no extra test files and the folder does not exist, the current working directory -SHOULD be an empty folder. +executed in the `cwd` stored in the script section. If there is no `cwd` stored (there are +no extra test files and the folder does not exist), the current working directory +SHOULD be a temporary empty folder. + +> Note: should we copy the cwd folder to a temporary location to prevent editing +> the source files? #### Python test @@ -165,20 +168,24 @@ The following environment variables SHOULD be set: - `PKG_VERSION` - the version of the package - `PKG_BUILDNUM` - the build number of the package - `PKG_BUILD_STRING` - the build number of the package -- `PREFIX` - the prefix where the package is installed -- (TODO: complete this list) Note: `PKG_HASH` is removed because it can't be easily extracted from `index.json` +Some additional environment variables that MUST be set: + +- `PREFIX` - the prefix where the package is installed +- `SHLIB_EXT` - the shared library extension (e.g. `.so`, `.dylib`, `.dll`) + #### Downstream test A downstream test SHOULD create a execute all tests of a downstream package with the package to be tested. If dependency resolution for any of the tests fails (e.g. due to conflicting dependencies), the test SHOULD be skipped. -The downstream test SHOULD then execute all tests from the downstream package in -separate test environments. The test SHOULD succeed if all tests pass. If any -test fails, the downstream test SHOULD fail. +The downstream test SHOULD then execute any script and python tests from the downstream package in separate test environments. +The test SHOULD succeed if all tests pass. If any test fails, the downstream test SHOULD fail. + +Note: downstream tests of the downstream package are not executed to prevent too many tests from being executed. ## Package contents test