Skip to content

mado and mdl disagree on 36 of the 52 acceptance fixtures #401

Description

@akiomik

The acceptance comparison is not clean, and nothing recorded that. Raised while
fixing #396, which repaired the setup script that prepares this corpus.

What the comparison shows

scripts/acceptance/test.sh runs mdl and mado check --output-format=mdl
over the same fixtures, each under one fixed config (scripts/acceptance/.mdlrc
and the repository's mado.toml), and writes tmp/mdl.txt and tmp/mado.txt.
Diffing those two files on a corpus prepared by setup.sh:

  • 36 of 52 fixtures differ.
  • 103 findings mado reports and mdl does not.
  • 30 findings mdl reports and mado does not.

Both files use the same path:line: MDxxx text form and absolute paths, so
these are real disagreements, not formatting.

mado only mdl only
MD007 40 MD031 20
MD041 31 MD007 3
MD003 6 MD032 2
MD025 5 MD022 2
MD031 4 MD028 1
MD022 3 MD024 1
MD040, MD038, MD020, MD006, MD004, MD002 2 each MD013 1
MD028, MD014 1 each

MD007, MD041 and MD031 are all listed in .mdlrc, so this is not a
question of one tool having a rule the other was not asked for. MD031 is the
interesting one: it appears on both sides, so the two tools are finding it
in different places rather than one of them missing it.

Why it is worth a look now

The corpus these numbers come from is what setup.sh is supposed to produce:
the fixtures upstream pairs with an X_style.rb are set aside, because those
are written against a rule set of their own that test.sh cannot apply. So
this is the intended comparison, and it is the one that disagrees.

Two things to separate before reading much into the totals:

  • Whether .mdlrc and mado.toml really do ask for the same thing. They are
    written in different formats and were not derived from one another.
  • Which differences are mado bugs, which are mdl bugs, and which are documented
    intentional divergences. Nothing in the repository currently says.

Note on how this went unnoticed

setup.sh's exclusion loop has never run in the committed script — see #396
so anyone cloning fresh compares all 88 fixtures instead of these 52, on a
corpus that includes documents designed for other rule sets. The numbers above
come from a checkout where the exclusion had been applied by hand before the
script was committed.

test.sh also has no diff and no assertion: it writes two files and stops.
So there has never been anything to notice, which is #400's subject from a
different angle.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions