Comparisons of fuzzer

[arcuri82]

Hi all @grebnetiew @ThomasTNO @Stranger6667 @en-milie ,

writing this here to simplify the discussions, and to have it open.

At a certain point, I would like to make a "fair" and "sound" comparison of fuzzers (@Stranger6667 this is somehow related to the discussion we had on Reddit), where all compared fuzzers' authors are directly involved.
This would lead to a possible joint academic publication among us (for whom is interested).

However, what is the status of supporting WFC Authentication? If I understood correctly, WuppieFuzz now supports it. What about Schemathesis and CATS?
This would be needed to easily run experiments on WFD, by using the already existing auth info in WFC format.

This is a very busy period, unfortunately (work and personal reasons). So, don't think anything would happen any time soon before some weeks (there would be several technical details to discuss and agree among us). However, I am writing it now to test out if there is any interest in it.

Let me know.

[grebnetiew]

It would be a nice test case for our WFC auth support to start with :D

We would be particularly interested if this leads to a benchmark framework that is considered 'sound' and/or 'fair' by the broader scientific community. In the past, we have had trouble converging (with article reviewers) on benchmarks that sufficiently represent real world bug-finding performance. In the future, we would like to use such benchmarks as integration tests for new features, fixes etc.

[Stranger6667]

From the Schemathesis side I have a WIP branch with the WFC Authentication support for a few months already, but didn't get to finish it just yet. So, I am going to bump its priority - probably will take a few weeks, so I'd aim to have it implemented by the end of July :)

This would lead to a possible joint academic publication among us (for whom is interested).

You can count on me here :)

P.S. I took some inspiration from WFC Authentication in Schemathesis's own declarative dynamic auth support

[arcuri82]

@Stranger6667 nice ;)

personally i be ll off on vacation most of July... so something starting around August/September would good ;)

meanwhile, to test WFC support, I would suggest to try with the auth configuration files for the APIs needing auth in WFD.

[ThomasTNO]

August also seems to fit nicely in my schedule. 👌