as a pie-in-the-sky solution, i wonder if there might be a way to build a... matrix view of what is or is not supported for each platform. right now, you just run the suite and see what fails for a given platform, there's not an overview that says "X is supported in clj but not cljs"
Related to the discussion in #883. I meant to post this back in September and then forgot 💀
Originating Slack thread: https://clojurians.slack.com/archives/C03SRH97FDK/p1757435398485679
Details from Jeaye:
The simplest starting point is this:
- Start tracking state in our test runner for which tests are run vs skipped
- Add a
(p/skip #{:phel :jank} (deftest ...)) and track that state
- Also track skips from
when-var-exists being false
- Add a
(p/todo #{:bb} "Some explanation" (is (= ...))) and track that state
- Generate a report for each dialect, in CI, and export that as a CI artifact
This would already be helpful to have and could be merged. From there, we can:
- Use a job which runs after all dialects to combine the artifacts into a single artifact
- Turn this artifact into a helpful table and write it to
gh-pages so we can view it easily from a browser
- Hook this into the suite at a per-case level, rather than just per-test, so we can cover cases like "+ works for CLJS, but ratios aren't supported"
a. This can end up including another macro, if needed, to capture why a particular case is skipped, which ends up int he report
As an example of what I mean, Gloat has pretty much exactly this, in its test report here: https://gloathub.org/test-report/ We would just want to apply the same methodology to all dialects and ideally get some tables so that we can more easily diff parity across the board.
Related to the discussion in #883. I meant to post this back in September and then forgot 💀
Originating Slack thread: https://clojurians.slack.com/archives/C03SRH97FDK/p1757435398485679
Details from Jeaye:
The simplest starting point is this:
(p/skip #{:phel :jank} (deftest ...))and track that statewhen-var-existsbeingfalse(p/todo #{:bb} "Some explanation" (is (= ...)))and track that stateThis would already be helpful to have and could be merged. From there, we can:
gh-pagesso we can view it easily from a browsera. This can end up including another macro, if needed, to capture why a particular case is skipped, which ends up int he report
As an example of what I mean, Gloat has pretty much exactly this, in its test report here: https://gloathub.org/test-report/ We would just want to apply the same methodology to all dialects and ideally get some tables so that we can more easily diff parity across the board.