The acceptance comparison is not clean, and nothing recorded that. Raised while
fixing #396, which repaired the setup script that prepares this corpus.
What the comparison shows
scripts/acceptance/test.sh runs mdl and mado check --output-format=mdl
over the same fixtures, each under one fixed config (scripts/acceptance/.mdlrc
and the repository's mado.toml), and writes tmp/mdl.txt and tmp/mado.txt.
Diffing those two files on a corpus prepared by setup.sh:
- 36 of 52 fixtures differ.
- 103 findings mado reports and mdl does not.
- 30 findings mdl reports and mado does not.
Both files use the same path:line: MDxxx text form and absolute paths, so
these are real disagreements, not formatting.
| mado only |
|
mdl only |
|
| MD007 |
40 |
MD031 |
20 |
| MD041 |
31 |
MD007 |
3 |
| MD003 |
6 |
MD032 |
2 |
| MD025 |
5 |
MD022 |
2 |
| MD031 |
4 |
MD028 |
1 |
| MD022 |
3 |
MD024 |
1 |
| MD040, MD038, MD020, MD006, MD004, MD002 |
2 each |
MD013 |
1 |
| MD028, MD014 |
1 each |
|
|
MD007, MD041 and MD031 are all listed in .mdlrc, so this is not a
question of one tool having a rule the other was not asked for. MD031 is the
interesting one: it appears on both sides, so the two tools are finding it
in different places rather than one of them missing it.
Why it is worth a look now
The corpus these numbers come from is what setup.sh is supposed to produce:
the fixtures upstream pairs with an X_style.rb are set aside, because those
are written against a rule set of their own that test.sh cannot apply. So
this is the intended comparison, and it is the one that disagrees.
Two things to separate before reading much into the totals:
- Whether
.mdlrc and mado.toml really do ask for the same thing. They are
written in different formats and were not derived from one another.
- Which differences are mado bugs, which are mdl bugs, and which are documented
intentional divergences. Nothing in the repository currently says.
Note on how this went unnoticed
setup.sh's exclusion loop has never run in the committed script — see #396 —
so anyone cloning fresh compares all 88 fixtures instead of these 52, on a
corpus that includes documents designed for other rule sets. The numbers above
come from a checkout where the exclusion had been applied by hand before the
script was committed.
test.sh also has no diff and no assertion: it writes two files and stops.
So there has never been anything to notice, which is #400's subject from a
different angle.
The acceptance comparison is not clean, and nothing recorded that. Raised while
fixing #396, which repaired the setup script that prepares this corpus.
What the comparison shows
scripts/acceptance/test.shrunsmdlandmado check --output-format=mdlover the same fixtures, each under one fixed config (
scripts/acceptance/.mdlrcand the repository's
mado.toml), and writestmp/mdl.txtandtmp/mado.txt.Diffing those two files on a corpus prepared by
setup.sh:Both files use the same
path:line: MDxxx textform and absolute paths, sothese are real disagreements, not formatting.
MD007,MD041andMD031are all listed in.mdlrc, so this is not aquestion of one tool having a rule the other was not asked for. MD031 is the
interesting one: it appears on both sides, so the two tools are finding it
in different places rather than one of them missing it.
Why it is worth a look now
The corpus these numbers come from is what
setup.shis supposed to produce:the fixtures upstream pairs with an
X_style.rbare set aside, because thoseare written against a rule set of their own that
test.shcannot apply. Sothis is the intended comparison, and it is the one that disagrees.
Two things to separate before reading much into the totals:
.mdlrcandmado.tomlreally do ask for the same thing. They arewritten in different formats and were not derived from one another.
intentional divergences. Nothing in the repository currently says.
Note on how this went unnoticed
setup.sh's exclusion loop has never run in the committed script — see #396 —so anyone cloning fresh compares all 88 fixtures instead of these 52, on a
corpus that includes documents designed for other rule sets. The numbers above
come from a checkout where the exclusion had been applied by hand before the
script was committed.
test.shalso has nodiffand no assertion: it writes two files and stops.So there has never been anything to notice, which is #400's subject from a
different angle.