Is there an open-source version of the AMR checker?

[nschneid]

@uhermjakob's AMR Editor has a Checker that implements a bunch of heuristics to look for annotation issues, e.g. roles that are used in an invalid context. Is code for this (or anything similar) available?

All I could find is the resource lists at https://amr.isi.edu/download.html

[flipz357]

Do you have any idea what's important in such a tool?

This should be simple:

• invalid roles: can be checked against frame definition (for English AMR...)
• invalid senses: see above

But what's are other "annotation issues" that can happen? Introducing a new variable that is already assigned? I guess there can be arbitrary many more heuristics and issuing warnings, just from the top of my head, like matching against dep tree (if there is a neg in dep tree, maybe there's something wrong if the AMR doesn't have a polarity edge etc, and so on)

[nschneid]

My understanding is that the AMR Checker (implemented within the annotation tool) is quite extensive: it highlights words in the sentence that may have been missed, gives hints about many AMR conventions such as those listed in the Dictionary, and so on.

[nschneid]

Aha, found this documentation page: https://www.isi.edu/~ulf/amr/lib/about-checker.html

[flipz357]

Interesting! I guess in 2023 many of these background checks (and more) can also be achieved by checking against the output of a good parser (or, better, an ensemble of parsers, so as to not risk any fitting of annotation to one parser).

That said, I don't know of any of such tools. But, yeah, um, I have long thought of some things in that direction (estimating quality of AMR). Maybe it's of interest, here are some of the musings

The source is this paper, page 40 bottom right. It's about assessing quality of automatic parses, but the idea can transfer to human parses.

[flipz357]

@nschneid It seems like there's now a new editor https://arxiv.org/abs/2311.10928

Maybe it has some of the features that you want. I also see that they use parser in the loop for suggestion. (but as said above imo this could bear a risk of a bias loop between parsing/annotation)

[jheinecke]

My annotation tool provides a basic validation function (based on the propbank frames (XML)
https://github.com/Orange-OpenSource/metamorphosed