WIP Upgrade to Schemathesis v4

[juhoinkinen]

WIP

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[juhoinkinen]

In addition to upgrading to Schemathesis 4, this PR currently enables strict_validation in Connexion "to disallow any extra parameters that are not defined in your specification". I think this would be a good feature, because then typos in API requests parameters are noticed at once, not after debugging (with strict validation e.g. using theshold instead of threshold gives 400 Bad request error, without it the extra, nonfunctional parameter is just silently ignored).

However, Annif 1.4 introduced metadata_* parameters to suggest/, suggest_batch/ and learn/ API methods (#867), and to return a 400 error for requests with parameters not starting with metadata_ I made this: Add Custom request body validator.

But because of this, Schemathesis 4.7 (correctly, because the API schema does not define the metadata_ prefix requirement) complains:

  | - API rejected schema-compliant request
  | 
  |     Valid data should have been accepted
  |     Expected: 2xx, 401, 403, 404, 409, 5xx
  | 
  | [400] Bad Request:
  | 
  |     `{"type": "about:blank", "title": "Bad Request", "detail": "Extra formData parameter(s) x-schemathesis-additional not in spec", "status": 400}`
  | 
  | Reproduce with: 
  | 
  |     curl -X POST -H 'Content-Type: application/x-www-form-urlencoded' -d 'language=&limit=10&text=&threshold=0&x-schemathesis-additional=' http://localhost/v1/projects/dummy-fi/suggest

Schemathesis should be configurable to ignore this, but I did not manage in that for now.

@osma what do you think about enabling the strict validation and the custom validator? Those could and should be done in another PR than this, if we want start using those.

[codecov]

Codecov Report

❌ Patch coverage is 70.00000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 99.56%. Comparing base (4ab42dc) to head (87e1096).

Files with missing lines	Patch %	Lines
tests/test_openapi.py	70.00%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #879      +/-   ##
==========================================
- Coverage   99.63%   99.56%   -0.08%     
==========================================
  Files         103      103              
  Lines        8238     8253      +15     
==========================================
+ Hits         8208     8217       +9     
- Misses         30       36       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[juhoinkinen]

Schemathesis detected and failed unit test due to this Connexion bug which makes Connexion 3 crash, essentially with this message:

  | schemathesis.core.failures.FailureGroup: Schemathesis found 2 distinct failures
  | 
  | - Undocumented HTTP status code
  | 
  |     Received: 500
  |     Documented: 200, 400
  | 
  | - Server error
  | 
  | [500] Internal Server Error:
  | 
  |     `{"type": "about:blank", "title": "Internal Server Error", "detail": "The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.", "status": 500}`
  | 
  | Reproduce with:
  | 
  |     curl -X POST -H 'Content-Type: application/json' -d $'\x12�\x16��' http://localhost/my-method
  | 

I asked how to skig generating non-UTF8 content in Schemathesis discussions, because the current way of just ignoring all server errors and wrong status codes is not good.

[juhoinkinen]

Also, I removed setting the strict_validation in Connexion. That should be another PR if it is done.

(NB: Schemathesis configuration includes a setting schema.config.generation.allow_extra_parameters:

Controls whether Schemathesis produces unexpected query, header, or cookie parameters. Leave it enabled (default) to exercise additionalProperties: false; set it to false to skip generating those extras entirely.

The setting is now set to False, but if strict validation is enabled in Connexion, schema.config.generation.allow_extra_parameters should be set to True.)

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[juhoinkinen]

Schemathesis 4.11 fixed an issue regarding filtering of request body content. However, using Schemathesis 4.11 requires pytest 9, so let's postpone this PR until upgrading all dependencies.

Also, the git history had again got messy, which is why I made a force push with after clean up.