Next generation <etym> in the TEI – Putting together a proposal

After the discussion on the TEI list and the comments on the list, the post and twitter exchanges with @ttasovac, I came back to an old paper of Susanne Alt (, which provides an overview of the constraints that could apply to a comprehensive representation of etymological information in a dictionary entry.

Susanne uses the entry “Pamplemousse” from the TLF (Trésor de la Langue Française) to encode the following etymological information:

PAMPLEMOUSSE, subst. masc. […]

Empr. au néerl. pompelmoes, fém., au sens 1 a, qui est prob. pompel «gros, enlé» et de limoes «citron» (BOULAN, p.148; KÖNIG, pp.159-160). Apparaît d’abord dans des textes fr. qui le donnent comme mot néerl.: 1665 pompelmoes (J. LE CARPENTIER, L’Ambassade de la Compagnie orientale des Provinces Unies… [trad. d’un ouvrage néerl.], II, p.88 ds ARV.); 1666 pompelmous (M. THÉVENOT, Relation de divers voyages curieux… t.3 ds KÖNIG).

At the end of paper, following an in-depth discussion of etymons and links, which I advise you all to read, she suggests the following encoding:

(i.e. the lexical entry pamplemousse and its direct etymon pompelmoes) of the data structure in a TeI like format. Note that the basic building blocks for the characterization of a lexical entry (<form> and <sense>) are completely reus- able for the description of an etymon, and the language has been implemented as a standardized xml:lang attribute.

<lexicalEntry id=”LE1″>
<orth>pamplemousse</orth> …
<sense>… </sense>
<etymology> <etymon id=”LE2″>
<form> <orth xml:lang=”dutch”>pompelmoes</orth> <pos>commonNoun</pos> <gender>feminine</gender>
</form> <sense>
<glose>Citrus maxima</glose>
<note>probablement d’origine tamoule, De Vries, Nederl</note> </sense>
</etymon> <etymologicalLink source=”LE2″ target=”LE1″>
<etymologicalClass>loan word</etymologicalClass> </etymologicalLink>
</etymology> </lexicalEntry>

If we “modernize” this proposal to make it as close to a TEI compliant entry, we could think of something as follows:
<entry xml:id=”LE1″>
<orth>pamplemousse … </form>
<re type=”etymon” xml:id=”LE2″>
<orth xml:lang=”dutch”>pompelmoes</orth>
<gloss>Citrus maxima</gloss>
<note>probablement d’origine tamoule, De Vries, Nederl</note>
<link type=”loanWord” target=”#LE2 #LE1″/>

This is a little more ambitious than what was discussed in the previous post and related comments, but seems to follow what Toma Tasovac had in mind with an embedded entry in the description of the etymological content.

From print to code – multiple forms in one

An interesting discussion started on the TEI list ( about dictionary encoding, which I would like to reflect here to point to a couple of issues related to forms in dictionaries.

The post was sent by Susanna Allés, who is currently working on a Latin medieval dictionary. Here’s the initial question:

“we can’t find a satisfactory solution to encode a term (<term>) that corresponds to several languages. Here is the text:

arrencare  [cat. arrencar, arrancar (cf. oc., esp., port. arrancar), d’origen discutit | de origen discutido | of debated origin]  rompre roturar to break up: 1038 ACUrgell, Cart. I 357, f. 118, col. 2: terra quae fuit arrenchada qui est in Subuineas quae escamiauit Ermengauda a me Mirone. De una parte affrontat … in terra quae arrencauit … de IIII uero parte in ipsa fonnada de terra de Riamball que arrencauit.

The part in (square) brackets corresponds to the etymology (encoded with <etym>) and in red (cf. oc., esp., port. arrancar) corresponds to a term (the Romanic result); we want to relate this term with several languages (that it is, to Occitan, Spanish, Portuguese, in this case). Usually, if we are dealing with just a language, we encode it as <term xml:lang=”es”>arrancar</term> , but since we need to add other languages and the element term just allow one @xml:lang, we really don’t know how to handle.”

For sake of place, the original editor of the dictionary has used the same character sequence to represent three “words” from three different languages, which just happen, by chance (and linguistic proximity…), to be written in the same way.

The first line of thought, which considers the source text at face value may lead (like stated in the question) to  one could want to refer to three languages in relation to the sequence “arrancar”. Possibilities that have been alluded to on the list are:

  • putting more that one language codes as values for xml:lang, theoretically feasible, given the recent evolution of the corresponding W3C specification, not in the TEI though and probably an “interoperability nightmare” (quoting Stuart Yeates)
  • tagging the language references with <lang>, as suggested by Sabine Thuillier, in keeping to the basic constructs associated to <etym> (etymology) in the TEI guidelines (i.e. ​cf. ​<lang>oc., esp., port.</lang> <mentionned>arrancar</mentionned>​)​; Sabine rightly points to the fact that this solution is not optimal from a processing point of view.

Still, a more lexicographic analysis seems to lead to another path which consists in reconstructing the underlying logical representation, as nicely stated by Toma Tasovac:

“what we have here is typical lexicographic shorthand —  we know how lexicographers used to be stingy when it comes to space —  but that, if we really think about it, we are not (on the conceptual, modeling, even philosophical level) talking about one word being “the same” in three languages, but rather about three “different” words from three different languages that happen to have the same graphical representation.

What one could therefore also do is encode three instances of “arrancar” and each with an xml:lang for a different language.”

It is interesting to observe that several contributors, even being in line with this analysis, have actually suggested mechanisms to retain the “sameness” in the encoding (using @sameas, @exclude or <alt>; Syd Baumann actually provided quite a comprehensive set of possibilities)). No, Toma (supported by David Birnbaum) put it right:

“if one wanted to preserve the economic display in the end output, one would have to use XSLT or whatever one uses to check whether subsequent terms have the same graphical form and then display only one term, but display language ids from all of them.”

Which means that we need to produce an encoding where nothing may lead a processor to think that we have the “same” form, but it just happened that the three were printed out as one accidentally.

Finally, let me add that there is an issue with the encoding of “arrancar” as <term>. It is obviously a <form> (even better a <form>/<orth> if we anticipate that we could provide additional pronunciation information.

So my take at the encoding of this example would be as follows:

(cf. <lang>oc.</lang>, <lang>esp.</lang>, <lang>port.</lang> <form xml:lang=”oc”><orth>arrancar</orth></form><form xml:lang=”sp” ><orth>arrancar</orth></form><form xml:lang=”pt”><orth>arrancar</orth></form>)

But there is a hack here… <form> is not allowed there. Shall we change the content model of <etym>?

Morpho-syntactic annotations: the theory

Since I keep receiving questions concerning the relation between the TEI guidelines and POS annotations, let me start a series of posts on the issue. As a first step, I present here the basic concepts of the underlying ISO standard: ISO 24611:2012 Language resource management — Morpho-syntactic annotation framework (MAF).

ISO 24611 provides a framework for the representation of morpho-syntactic (also referred to as part-of-speech) annotations. Such annotations correspond to a first lexical abstraction over language data (textual or spoken) and, depending on the language to be annotated, as well as the characteristics of the annotation tool or annotation scheme that is being used, can vary enormously in structure and complexity.

In order to deal with such complex issues as ambiguity and determinism in morpho-syntactic annotation, ISO 24611 introduces a meta-model that draws a clear distinction between the two levels of tokens (representing the surface segmentation of the source) and word-forms (identifying lexical abstractions associated with groups of tokens). Both levels can be represented as simple sequences and as local graphs such as multiple segmentations and ambiguous compounds. Besides, any n-to-n combination can stand between word forms and tokens.

As linguistic segments (sometimes called ‘markables’ in the literature, see for instance, Carletta et al. 1997), tokens may be embedded in the source document as inline mark-up, or they may point remotely to it by means of so-called stand-off annotations.

As linguistic abstractions, word-forms can be qualified by various linguistic features characterising the morpho-syntactic properties that are instantiated in the realisation of the lexical entry within the annotated text. Such properties may range from the simple indication of a lemma up to an explicit reference to a lexical entry in a dictionary. In most existing applications of morpho-syntactic annotation, linguistic properties are expressed by means of so-called tags; these codes refer to basic feature structures (see early examples in Monachini and Calzolari, 1994). Such codes may also provide morphological information, including its part of speech (e.g. noun, adjective or verb), and features such as number, gender, person, mood and verbal tense.

In keeping with the general modelling strategy of ISO/TC 37, MAF provides means of relating morpho-syntactic tags expressed as feature structures (compliant with ISO 24610) to the data categories available in A normative annex of this International Standard elicits a core set of data categories that can be used as reference for most current morpho-syntactic annotation tasks in a multilingual context. However, when implementers of the standard find these categories inappropriate in either coverage, scope or semantics, they are encouraged to use ISOCat to define their own categories.

Associated to the meta-model, MAF also provides a default XML serialisation. Still, as we shall see in a further post, it is possible to combine a TEI based representation with MAF compliant annotations (See also Romary and Witt, 2013).


ISO 24610-1:2006 Language resource management — Feature structures — Part 1: Feature structure representation (see also the corresponding chapter in the TEI guidelines:

Jean Carletta, Nils Dahlbäck, Norbert Reithinger and Marilyn A. Walker (Eds.) (1997) Standards for Dialogue Coding in Natural Language Processing, Dagsthul-Seminar Report 167; 03.02.-07.02.97 (9706)

Monachini, Monica and Nicoletta Calzolari (1994). Synopsis and Comparison of Morpho-syntactic Phenomena Encoded in Lexicon and Corpora. A Common Proposal and Applications to European Languages. Internal Document, EAGLES Lexicon Group, ILC, Università Pisa, Oct. 1994

Romary, Laurent and Andreas Witt (2013), “Data formats for phonological corpora”, in Handbook of Corpus Phonology Oxford University Press (Ed.).

Scholarly work and Open Access

The whole idea of scholarship is oriented towards maximising the dissemination of research results. Carrying out a research activity is all about exploring territories, where knowing what the others are doing, what their most recent advances are, what projects are being undertaken, is essential to make sure that one’s own research actually goes beyond the state of the art and can be situated within a larger corpus of discoveries. Communicating results is thus an essential activity in one’s academic life, all the more that the assessment of such communications through peer review mechanisms impact on the capacity to get institutional recognition and thus financial means to carry out further research.

This is the context in which research organisations should be designing their own Open Access (OA) policies in order to help researchers to get access to existing publications (traditionally through journal subscriptions), publish their own results to a wide audience (by means of publication repositories[1]) and manage associated research assets (laboratory notes, observations, primary sources, databases). This has been made particularly difficult in the recent years because of the incredible increase in publication prices imposed by private publishers, which has been seen as contradicting the intuition that new technologies should indeed simplify dissemination rather than increase operational publication costs. The situation has become all the more unbearable that most of the various processes bringing a research manuscript to publication are carried out for free by researchers themselves.

Even if the OA movement is quite recent, it is possible to outline some principles and possible action lines. Among the various meetings which, in the early years 2000, contributed to stabilize the basic notions of Open Access, we can quote the core statement from the Berlin Declaration (2003) that requested a “…free, irrevocable, worldwide, right of access to, and a license to copy, use, distribute, transmit and display the work publicly and to make and distribute derivative works, in any digital medium for any responsible purpose…. ”. Not only does this statement reflect the feeling that the current landscape is inadequate to fulfil the communication needs of the research community, but it also outlines a possible ideal scheme where commercial constraints would have a lesser importance than issues related to public good and wealth.

There are basically two ways to implement such a change in the scholarly publishing environment. The first one (also called green open access) consists in letting the journal publishing industry carry out its business as is and deploy an infrastructure of publication repositories to freely disseminate authors’ versions online. Such an infrastructure may be deployed at the level of a research department, a university or cover a wider geographical or institutional spectrum. In the case of France, a national publication archive infrastructure, HAL[2], has been deployed to cover the needs of most French academic organisations. It is now part of the official Open Access policy of the French Ministry of Research and Higher Education[3]. Such publication platforms are particularly important since they support the immediate dissemination of research papers from an early drafting stage to its final publication. It is also a way for research institutions to get a global picture of all research carried out under their auspices, since it is associated with precise descriptive information (for instance, affiliations) that can be curated by librarians in a coherent way[4].

There is also an increasing demand from scholars to benefit from trusted repository environments where they can deposit their research data with the guaranty that these will be accessible and referentiable in the long-term. Recent developments in the Netherlands with the NARCIS repository[5] for example have shown that such environments can even be coupled with a traditional publication archive.

The other way to implement an OA policy (also called the gold way to OA) is to replace the current subscription-based model by other business models both allowing “fair” publishing organisations to break even in maintaining a publication framework and designing a barrier-free system for the access and reuse of research results. Even if the publishing industry has recently perverted this golden way by introducing an article-processing-charge system that only reshapes what used to be subscription costs into an author-pay model, we will see in the rest of this post that there is room for other perspectives to reform the publication landscape.

A first strategy may be to think of alternative business models that are based on an ethical support to scholarly publishing within an OA perspective. This is what has been proposed by the OpenEdition publishing infrastructure with their Freemium model. It is based on the principle that basic access (for instance in HTML) to published material should be made open for free but that additional services can be sold to libraries (pdf, ePub, cataloguing services) for a reasonable fee. The corresponding benefits are in turn given back to the journals so that they can cover their day-to-day costs. Indeed, as is the case for OpenEdition, the core infrastructure as well as the general editorial support is part of the institutionally funded infrastructure.

If we are still attached to the traditional journal editorial setting, we can observe that its core services, namely identification, certification, dissemination and long-term availability, can be easily implemented on the basis of an existing publication repository. Indeed, such a repository provides a submission environment, which identifies authors and time-stamp the document, and offers a perfect online dissemination platform, with the necessary long-term archiving facility of the hosting institution. In such a context, designing a certification environment mechanism whereby a paper deposited by an author is forwarded to an editorial committee for peer-review, is quite a straightforward endeavour. This is exactly what is now being experimented with the Episciences[6] project on top of the HAL platform. Such a platform is also interesting in that it offers new possibilities for changing our perspective on the certification process: open submission, open peer-review[7], updated versions of an article and community feedback are features that may dramatically change our views on scholarly publishing.

At specific stages of the research process, it is often not so much important to produce an in-depth scholarly than to provide short snapshots on the current developments of an experiment in hard sciences, or the analysis of a source in the humanities. This is a situation where it is more appropriate for a scholar to write small reports in the form of blog entries and publicize them on various social networks. Blogs offer a first layer of scholarly publication with both online availability and the possibility to comment on the actual scholarly content. It is also a simple way to gain a primacy for a specific result or gather observations step-by-step, for instance during an archaeological campaign. The ideal situation is when blogging occurs within a secured scholarly environment such as where researchers benefit from an editorial support as well as a wide visibility.

The various possibilities outlined so far only make sense if research institutions invest time, political energy and budget to implement such models and make them part of the daily life of their researchers. A typical best practice example can be taken from the recently published open access policy of Inria[8] which combines a deposit mandate of all publications on the HAL archive, a cautious assessment of any new models provided by the private publishing sector and the funding of the Episciences platform.

We can observe that having a not too overly conservative vision on scholarly communication opens up a whole range of possibilities to improve the way scientific ideas can be seamlessly transmitted to a wide audience. Even more, we can see that a new landscape can be outlined where the management of virtual research environments comprising research data, various types of notes and commentaries, as well as drafted documents linking these objects together could dramatically change the way scholarship will be carried out in the future. In such environments, various levels of “peer-review” are possible, from the simple feedback of known colleagues to the possibility for any member of a research community to comment on the content. Traditional peer-review is just one possible implementation of such a model where the main objective should remain to improve quality and wide accessibility for science.

As a whole, I defend a vision of scholarly communication which is entrenched in the wider notion of research infrastructure and which as such must be considered as part of the realm of public research institutions. We need to see what the consequences of such a vision are, in terms of budget shifts, investments in technological settings, but also in changing the roles of research libraries so that they can provide the necessary editorial support to such environments. The change may be drastic, but I think this is the only way to optimize tax-payers’ money at the service of science.


[1] see Romary Laurent & Chris Armbruster “Beyond Institutional Repositories”, International Journal of Digital Library Systems, 1(1), January 2010 —


[3] See

[4] Note that the quality of such descriptive information is one of the difficulties I see in such commercial platforms as or Research Gate.



[7] see Pöschl U. (2010) “Interactive open access publishing and peer review: the effectiveness and perspectives of transparency and self-regulation in scientific communication and evaluation”. LIBER Q. 19, 293–314.

[8] see

TBX goes TEI

As a first start into this blogging adventure, let me focus on a current activity which I started in the context of the recent ISO/TC 37 general meetings in Pretoria in June. The context is the ongoing revision process for ISO 300421.

Since its P4 edition, the TEI guidelines did not have a chapter on terminology any more. This is indeed missing since the current TEI dictionary chapter is covering semasiological views on lexical data (word to sense) and many applications require an onomasiological perspective (concept to term). For instance, translators, technical writers or language learners often manipulate terminologies, i.e. lexical structures are organised according to concepts, which in turn are subdivided in language sections (all words or expressions used to express the corresponding concept for this language) and then in term sections (all types of information needed to represent and qualify a given term). The underlying model is standardized  in ISO 16642 TMF2. TBX is thus a serialization of TMF.

I have now started to create an TEI/ODD specification that takes up the basic TBX structure, starting at <termEntry> level and allowing these to occur anywhere a traditional dictionary entry (<entry>) would in the TEI architecture.

To illustrate this, let me just show a typical  (yet awfully simple) example. The TBX namespace is a fake one (it is under discussion and we may want to get all this in the TEI name space).

<TEI xmlns=”″ xmlns:tbx=””

<termEntry xml:id=”c5″ xmlns=””>
<langSet xml:lang=”en”>
<langSet xml:lang=”fr”>


  1. ISO 30042:2008 Systems to manage terminology, knowledge and content — TermBase eXchange (TBX) []
  2. ISO 16642:2003 Computer applications in terminology — Terminological markup framework []

Bag of tags

L’objectif de ce carnet est de présenter différents travaux, réflexions ou synthèses sur la représentation de données numériques en sciences humaine et plus particulièrement dans le domaine des informations textuelles et linguistiques. Le carnet doit permettre de publier des entrées pédagogiques ou exploratoires permettant à un large public académique de mieux connaître les initiatives internationales portant sur la représentation de telles données, mais aussi de présenter des problèmes précis de modélisation qui peuvent accompagner, par exemple, le codage de source numériques particulières. Le champ du carnet couvre donc le domaine des corpus textuels oraux ou écrits, l’annotation de telles données à différents niveaux, ainsi que les ressources lexicales et terminologiques. Occasionnellement le carnet sera amené à couvrir des formats ou standards couvrant d’autres domaines (archives, ou domaines spécifiques dans les humanités. Concrètement, les présentations ou discussion aborderont des aspects liés aux directives de la TEI, aux travaux au sein de l’ISO (notamment du comité TC 37), mais aussi de standards tels qu’EAD, EAG ou EAC (archives).