Since I keep receiving questions concerning the relation between the TEI guidelines and POS annotations, let me start a series of posts on the issue. As a first step, I present here the basic concepts of the underlying ISO standard: ISO 24611:2012 Language resource management — Morpho-syntactic annotation framework (MAF).
ISO 24611 provides a framework for the representation of morpho-syntactic (also referred to as part-of-speech) annotations. Such annotations correspond to a first lexical abstraction over language data (textual or spoken) and, depending on the language to be annotated, as well as the characteristics of the annotation tool or annotation scheme that is being used, can vary enormously in structure and complexity.
In order to deal with such complex issues as ambiguity and determinism in morpho-syntactic annotation, ISO 24611 introduces a meta-model that draws a clear distinction between the two levels of tokens (representing the surface segmentation of the source) and word-forms (identifying lexical abstractions associated with groups of tokens). Both levels can be represented as simple sequences and as local graphs such as multiple segmentations and ambiguous compounds. Besides, any n-to-n combination can stand between word forms and tokens.
As linguistic segments (sometimes called ‘markables’ in the literature, see for instance, Carletta et al. 1997), tokens may be embedded in the source document as inline mark-up, or they may point remotely to it by means of so-called stand-off annotations.
As linguistic abstractions, word-forms can be qualified by various linguistic features characterising the morpho-syntactic properties that are instantiated in the realisation of the lexical entry within the annotated text. Such properties may range from the simple indication of a lemma up to an explicit reference to a lexical entry in a dictionary. In most existing applications of morpho-syntactic annotation, linguistic properties are expressed by means of so-called tags; these codes refer to basic feature structures (see early examples in Monachini and Calzolari, 1994). Such codes may also provide morphological information, including its part of speech (e.g. noun, adjective or verb), and features such as number, gender, person, mood and verbal tense.
In keeping with the general modelling strategy of ISO/TC 37, MAF provides means of relating morpho-syntactic tags expressed as feature structures (compliant with ISO 24610) to the data categories available in ISOCat.org. A normative annex of this International Standard elicits a core set of data categories that can be used as reference for most current morpho-syntactic annotation tasks in a multilingual context. However, when implementers of the standard find these categories inappropriate in either coverage, scope or semantics, they are encouraged to use ISOCat to define their own categories.
Associated to the meta-model, MAF also provides a default XML serialisation. Still, as we shall see in a further post, it is possible to combine a TEI based representation with MAF compliant annotations (See also Romary and Witt, 2013).
ISO 24610-1:2006 Language resource management — Feature structures — Part 1: Feature structure representation (see also the corresponding chapter in the TEI guidelines: http://www.tei-c.org/release/doc/tei-p5-doc/en/html/FS.html)
Jean Carletta, Nils Dahlbäck, Norbert Reithinger and Marilyn A. Walker (Eds.) (1997) Standards for Dialogue Coding in Natural Language Processing, Dagsthul-Seminar Report 167; 03.02.-07.02.97 (9706)
Monachini, Monica and Nicoletta Calzolari (1994). Synopsis and Comparison of Morpho-syntactic Phenomena Encoded in Lexicon and Corpora. A Common Proposal and Applications to European Languages. Internal Document, EAGLES Lexicon Group, ILC, Università Pisa, Oct. 1994
Romary, Laurent and Andreas Witt (2013), “Data formats for phonological corpora”, in Handbook of Corpus Phonology Oxford University Press (Ed.). http://hal.inria.fr/inria-00630289