comprehensive linguistic annotation

[osherenko]

I am developing an annotation and would specify for particular words not only lemmas and POS, but also etymologic or morphological information. How should I do it?

[proycon]

For morphological information, check the respective section in the documentation:Morphological annotation

For data on etymology there is no specific element, and I'd probably need to see an example of how such an annotation looks like in your data to give the best advice. It might be added as a higher-order feature using <feat> with one of the existing types.

[osherenko]

How would you annotate the morphology of the irregular verb "went"? "go" is not a part of it. BTW, there is a broken link to the API class -- https://folia.readthedocs.io/en/latest/morphological_annotation.html doesn't exist. I want to specify etymological data within a word annotation as its definition from a dictionary, similar to the annotation of the phoneme in an utterance. <utt xml:id="example.utt.1" src="helloworld.mp3" begintime="00:00:01.000" endtime="00:00:02.000"> <ph>helˈoʊ wɝːld</ph> <w xml:id="example.utt.1.w.1" begintime="00:00:00.000" endtime="00:00:01.000"> <ph>helˈoʊ</ph> <etymology>early 19th century: variant of earlier hollo ; related to holla.</etymology> </w> <w xml:id="example.utt.1.w.2" begintime="00:00:01.000" endtime="00:00:02.000"> <ph>wɝːld</ph> </w> </utt> Moreover, I want to specify linguistic information in the sentence annotation such as dependencies and the grammatical parse. For example, for sentence annotation of "The strongest rain ever recorded in India..." <https://nlp.stanford.edu/software/lex-parser.shtml#Sample> should include the grammatical tree (ROOT (S (S (NP (NP (DT The) (JJS strongest) (NN rain)) (VP (ADVP (RB ever)) (VBN recorded) (PP (IN in) (NP (NNP India))))) and dependencies: det(rain-3, The-1) amod(rain-3, strongest-2) nsubj(shut-8, rain-3) nsubj(snapped-16, rain-3) nsubj(closed-20, rain-3) nsubj(forced-23, rain-3) advmod(recorded-5, ever-4) partmod(rain-3, recorded-5) prep_in(recorded-5, India-7) Am Mi., 9. Nov. 2022 um 20:50 Uhr schrieb Maarten van Gompel < ***@***.***>:

…

For morphological information, check the respective section in the documentation:Morphological annotation <https://folia.readthedocs.io/en/latest/morphological_annotation.html> For data on etymology there is no specific element, and I'd probably need to see an example of how such an annotation looks like in your data to give the best advice. It might be added as a higher-order feature using <feat> with one of the existing types. — Reply to this email directly, view it on GitHub <#105 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A4B2N3PYFLTITQVHL6FO4F3WHP56XANCNFSM6AAAAAARZQV3PU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[osherenko]

Probably, the etymology annotation can resemble the sense annotation https://folia.readthedocs.io/en/latest/sense_annotation.html

[proycon]

How would you annotate the morphology of the irregular verb "went"? "go" is not a part of it.

True, "go" would be the lemma, in the simplest annotation form:

<w>
  <t>went</t>
  <lemma class="go" />
</w>

If you want to express it in a morphological structure, you could do the following:

<w>
  <t>went</t>
  <lemma class="go" />
  <morphology>
    <morpheme class="stem">
       <t>went</t>
       <lemma class="go" />
       <pos class="V" />
    </morpheme>
  </morphology>
</w>

There's a bit of duplication here to show that you can choose on what level you want to express certain things (like pos/lemma/sense). Most of what you can express on the word level is also valid on the morpheme level.

Note: the classes belong to a user-defined set definition, FoLiA itself does not prescribe them.

There are some further examples on https://folia.readthedocs.io/en/latest/morphological_annotation.html , which shows that you can also do nesting and associate extra features with morphemes (such as function, or any other you can invent). This is also the place where you might want to express etymology:

<w>
  <t>went</t>
  <lemma class="go" />
  <morphology>
    <morpheme class="stem">
       <t>went</t>
       <lemma class="go" />
       <pos class="V" />
       <feat subset="etymology" class="wenden" />
    </morpheme>
  </morphology>
</w>

In this case it's an extra feature on the morpheme. Rather than using the full description I opted for a shorted 'class' here, which ideally some external database would provide a full definition of (and the set of your morphology-annotation would determine what that database is). But of course you may also just put the full definition as class.

Moreover, I want to specify linguistic information in the sentence
annotation such as dependencies and the grammatical parse.

That can be accommodated in FoLiA by respectively dependency annotation and syntactic annotation.

BTW, there is a broken link to the API class --
https://folia.readthedocs.io/en/latest/morphological_annotation.html doesn't
exist.

Oops indeed, thanks! I have corrected the mistake.

I hope this provides some more clarity?

[proycon]

One of the points arising from this issue is whether we want to introduce an explicit <etymology> element in FoLiA, It would be an inline annotation element very alike to <sense>. We could then do <etymology class="wenden" /> . Which may be nicer than using a feature structure.

[osherenko]

An explicit etymology tag is indeed nicer. Another question is about multimodal injections in annotations similar to Example 1.7.1 in the documentation. Can FoliA accept multimodal injections? You spoke about OCR corrections. My annotation should annotate texts not only lexically, but also statistically, semantically, or cognitively. It should also consider properties of the original image used by the OCR to extract the texts. Another issue is the annotation of a corpus containing several texts. In this case, each text tag should hold the source of a text like the src entity in the Utterance annotation and be saved in the output XML file. Am Mo., 14. Nov. 2022 um 14:00 Uhr schrieb Maarten van Gompel < ***@***.***>:

…

One of the points arising from this issue is whether we want to introduce an explicit <etymology> element in FoLiA, It would be an inline annotation element very alike to <sense>. We could then do <etymology class="wenden" /> . Which may be nicer than using a feature structure. — Reply to this email directly, view it on GitHub <#105 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A4B2N3OXYMRCSXIVXIN6LMTWIIZVDANCNFSM6AAAAAARZQV3PU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[osherenko]

It might be a good idea to use Metric annotation, For example:
<text src="...">
<metric class="charlength" value="4" />
</text>"

[proycon]

An explicit etymology tag is indeed nicer

I'll work on adding one.

Another issue is the annotation of a corpus containing several texts.

Just do one FoLiA document per text. It's not recommended to put the whole corpus in a single FoLiA file.

Can FoliA accept multimodal injections?

You can have multiple text layers and multiple phonological layers, yes, if that's what you mean. One limitation in FoLiA, however, is that there can only be one canonical tokenization for everything.

It might be a good idea to use Metric annotation

<text src="...">
<metric class="charlength" value="4" />
</text>

I'm not entirely sure what your exact use case is for this, but you can indeed use metric annotation for all kinds of measurements on whatever you want.

It should also consider
properties of the original image used by the OCR to extract the texts.

[proycon]

@osherenko Please check what you think of the example in the above commit d43bbc8 and the documentation in commit db5499f . That's my proposed solution for annotating etymology in FoLiA.