pytest tests in core

[gforcada]

The current status is that the following Python distributions are using pytest tests and are in core:

• plone.classicui
• plone.distribution
• plone.exportimport
• plone.volto
• Products.CMFPlacefulWorkflow

That on its own is not good or bad, the problem lies on that these Python distributions are part of Plone core and thus are expected to be tested with the regular Jenkins jobs that ensure no breakages happen across Python distributions (i.e. a change in plone.app.event breaking plone.exportimport).

This is unfortunately not possible if zope.testrunner and pytest tests are mixed.

Possible solutions

• rewrite the tests in those few distributions to zope.testrunner
• rewrite ALL other Plone Python distributions to pytest 😅
• have another separate Jenkins job that ensures those Python distributions tests work with buildout.coredev latest changes

First and second are no fun. Second is more future proof, as we might want to switch eventually to pytest, but it will not happen overnight.

So third looks like the best middle ground, and actually sets the foundation for the 2nd option to happen.

The PR jenkins jobs will be a bit more complicated, as all tests are meant to run within that job.

Opinions?!

[gforcada]

On a buildout.coredev checkout:

make install
.venv/bin/pytest src/plone.volto/tests

Seems to work, I'm not 100% sure that this is using, say the src/Products.CMFPlone git checkout though... I have to go get my kids, I will keep looking at it.

[gforcada]

Quite a late report from Bucharest sprint, and this is only the tip, I did quite some more digging into the topic, but anyway...

Move to pytest

For quite some time there has been some push to move to a more modern test runner, namely pytest.

Scaffolding generators plonecli and cookieplone-templates but also tooling like pytest-plone have been created/updated to use pytest rather than zope-testrunner.

💥 The problem

There is one problem, fortunately with a "simple" solution: tests are not shipped with released artifacts (sdists or wheels), which makes impossible to run a full test suite on jenkins (for buildout.coredev).

The idiom in pytest is to have a tests folder at the top-level alongside src and docs, but even if you include them on the sdist/wheel, they can not be found afterwards, as they are not importable.

Moving them to the expected location we are using in Plone core (i.e. src/plone/app/layout/tests) makes them work again.

And pytest supports that ✨

I wanted to write a PLIP proposal for formally moving to pytest, but if all our tooling setting the defaults to a top-level tests folder we are making our lives harder and harder to eventually do that.

Opinions?

At least I would like to hear from @mauritsvanrees @ericof @MrTango @davisagli all others are welcome, of course.

Thanks @gyst for reminding me of that with your plone/plonecli#113 😄

[davisagli]

@gforcada If it works, I think that would be a good way to go. I have a vague memory of some problem with tox + pytest when the tests are inside the package (because then the imported test modules come from the installed sdist but pytest is expecting them from the editable package). But it's worth a try.

[davisagli]

@gforcada I'm trying this with the plone.recyclebin package that I'm creating today. It seems to work with pytest --pyargs plone.recyclebin.tests {posargs} as the test command. I also had to add __init__.py inside the tests folder.

[mauritsvanrees]

I am indeed in favour of moving the tests inside the package. That seems the only possible way to be able to still run the tests of a final package, instead of a development package.

If we get that working, then we could add another GitHub Actions job on buildout.coredev to run the tests of all packages that use pytest.

[gforcada]

A draft PR on plone.exporimport to move the tests inside the distribution: plone/plone.exportimport#100

[gforcada]

Seems that --pyargs does the trick:

cd /tmp
git clone git@github.com:plone/plone.exportimport
cd plone.exportimport
git checkout move-tests
python -m build
cd ..
python -m venv wheel-tests
cd wheel-tests
cp ../plone.exportimport/dist/*.whl
pip install pytest *.whl[test]
pytest --pyargs plone.exportimport

Duplicating the --pyargs XX does the trick on running all tests at the same time (so, we have our buildout.coredev good old bin/test.

Ensuring all doctests work and plone.app.robotframework tests also are supported is a bit trickier. doctests are supported by pytest, but ensuring all combinations of them work right away with custom flags or other changes (if we think it from buildout.coredev point of view) that's a bit more complicated.

For doctests, we could convert them to plain python code (?) and seems that there are pytest plugins for robot framework tests 🤞🏾

Maybe time to write a proper PLIP? 🤔

[stevepiercy]

For doctests, we could convert them to plain python code (?) and seems that there are pytest plugins for robot framework tests 🤞🏾

icalendar uses a function to find and run doctests in source files. It might be reusable. It has some tricky bits that are parameterized, such as which timezone library is used or setting an environment.

https://github.com/stevepiercy/icalendar/blob/main/src/icalendar/tests/test_with_doctest.py#L72-L91