ODD topics – a simple introduction to attribute classes in TEI

Attribute classes have been introduced in the TEI infrastructure in order to declare groups of attributes which are shared across various elements. For instance, the att.timed attribute class contains the declaration of two attributes, @start and @end (see https://www.tei-c.org/release/doc/tei-p5-doc/en/html/ref-att.timed.html): the elements whose specification states that they belong to this class will automatically get the two attributes. This is reflected in the generated documentation, as can be seen for the <u> element for instance (see https://www.tei-c.org/release/doc/tei-p5-doc/en/html/ref-u.html).

Over the years, it has been a general policy of the TEI technical council to factorise attribute declarations as classes in order to simplify the legibility and management of the guidelines.

When designing a TEI customisation in ODD, it is possible to use attribute classes to make the corresponding attributes available for additional elements. For example, you may want to say that there are several types of formulas in your document model and thus add the <formula> element to the att.typed attribute class.

In ODD specifications, adding an element to an attribute class can be achieved as exemplified below:

<elementSpec ident="formula" mode="change">
   <classes mode="change">
      <memberOf key="att.typed" mode="add"/>

The preceding declaration supplies the <formula> element with the two attributes declared in att.typed (see https://www.tei-c.org/release/doc/tei-p5-doc/en/html/ref-att.typed.html), namely @type and @subtype.

Conversely, one may want to get rid of some attributes inherited through a class on a specific element. There are actually two ways to do so:

  • Completely deselect the attribute class from the element, which takes out all attributes that were declared in the class;
  • Operates a surgical change by just taking out the intended attributes from the element.

The first method is similar to adding an attribute class to an element with the @mode attribute set to “delete” on <memberOf>. The following example removes the <s> element from the att.typed attribute class, thus depriving it of both @type and @subtype:

<elementSpec ident="s" mode="change">
   <classes mode="change">
      <memberOf key="att.typed" mode="delete"/>

The second methods elicits the deletion of the attribute from the element by using an <attDef> declaration. The following snippet deletes the @subtype attribute from the <s> element while keeping @type inherited from att.typed.

<elementSpec ident="s" mode="change">
      <attDef ident="subtype" mode="delete"/>

In some situations, when just one attribute from a class is needed, it is possible to combine a class attribution and attribute deletion declarations within the same ODD specification. For instance the following declaring makes <u> member of att.typed, while deleting @subtype:

<elementSpec ident="formula" mode="change">
   <classes mode="change">
      <memberOf key="att.typed" mode="add"/>
      <attDef ident="subtype" mode="delete"/>

Note: such a customization is closer to the spirit of the TEI guidelines, than declaring a new @type attribute on the element.

In the same way, it is possible to change the actual declaration of an attribute, in order to associate, for instance, a closed set of values to it.

ODD topics – Defining schematron constraints in ODD

Like any XML language, the TEI can be parsed with Schematron. Moreover, ODD supports natively the specification of Schematron rules,  and the TEI specification itself (for instance here: https://github.com/TEIC/TEI/blob/dev/P5/Source/Specs/att.datable.w3c.xml ) contains embedded schematron rules.

NB: This post doesn’t aim to explain how the Schematron language works. To learn more about Schematron, see W. Piez and D. Lapeyre, Introduction to Schematron, Mulberry Technologies, Inc, Nov. 2008. However, to minimally understand the following, you should know that Schematron, an ISO specification, is an XML language, based on the extensive use of XPath, that makes assertions or reports on the presence or absence of specific patterns in XML documents.

Schematron rules can be used to provide warnings without causing validation errors (i.e. “non-fatal errors”), or to add more qualitative rules to specific elements. For instance, one may want to set a character number limit for the content of a specific tag, or check the value of an attribute by means of a regular expression.

ODD makes it convenient to specify such rules at the same level of the element specification, in an element <constraintSpec>, possibly present in <attDef>, <classSpec>, <elementSpec>, or <schemaSpec> (see the TEI guidelines for more details). Inside <constraintSpec>, the <constraint> element contains assertions in the Schematron language, as shown in the following example:

<elementSpec ident="TEI" mode="change">
   <constraintSpec ident="xmlid" scheme="schematron" type="SSK">
         <sch:rule context="tei:TEI">
            <sch:let name="fileName" value="tokenize(document-uri(/), '/')[last()]"/>
            <sch:assert test="string(@xml:id)=substring-before($fileName, '.xml')" role="error">
The xml:id of the TEI element should be equal to the name of the file,
without the file extension. Currently the value of xml:id is "<sch:value-of select="@xml:id"/>" 
whilst the file name is "<sch:value-of select="$fileName"/>"</sch:assert>

Two important caveats (thanks to Syd Bauman’s XPath and Schematron for TEI Customization, https://www.wwp.northeastern.edu/outreach/seminars/_current/presentations/schematron/schematron_odd.xml):

  1. the element <sch:rule> is not mandatory. When missing (i.e., just bare sch:assert and sch:report elements), the context of the Schematron rule is the element or attribute that is being defined (<elementSpec> or <attDef>).
  2. In <constraintSpec>, you need to specify the tei: prefix for TEI elements (see example below).

When a RELAX NG schema is generated from the ODD, the schematron rule will be embedded in the schema:

<define name="tei_TEI">
   <element name="TEI">
      <pattern xmlns="http://purl.oclc.org/dsdl/schematron"
         <sch:rule xmlns="http://www.tei-c.org/ns/1.0"
            <sch:let name="fileName" value="tokenize(document-uri(/), '/')[last()]"/>
            <sch:assert test="string(@xml:id) = substring-before($fileName, '.xml')" role="error">
The xml:id of the TEI element should be equal to the name of the file, 
without the file extension. Currently the value of xml:id is "<sch:value-of select="@xml:id"/>" 
whilst the file name is "<sch:value-of select="$fileName"/>"</sch:assert>

This embedded Schematron rule is checked in addition to the RELAX NG when performing file validation, as shown in the following images:

See the full ODD example under:https://github.com/laurentromary/ODDtopics/blob/master/ODD%20examples/schematronODD.xml

ODD topics – The Encoded Archival Description (EAD) 2002 ODD specification and the EHRI use case

Maintaining an XML specification with ODD can be useful outside the TEI realm. First, an ODD specification allows for a smoother maintenance and evolution of an XML vocabulary, but it is also a great help to document project-specific customizations and practices.

A recent and fully documented use case is the EAD specification and and its customization for the EHRI project. This use case has been described in two publications : Romary, L. & Riondet, C. Arch Sci (2018) 18: 165. https://doi.org/10.1007/s10502-018-9290-y, and Romary, L. & Riondet, C. Umanistica Digitale (2019) 4: Data Sharing, Holocaust Documentation and the Digital Humanities: Best Practices, Case Studies and Benefits. https://doi.org/10.6092/issn.2532-8816/9046

This work has shown the efficiency that can be gained by the use of ODD specifications for low-constrained vocabularies.

The official EAD schema and the official EAD tag library were encoded as an ODD document (with the agreement of the Library of Congress and the Society of American Archivists), in the context of the Parthenos project. This EAD ODD is a starting point for EHRI, and was used to create an EHRI-specific EAD profile with very precise content-oriented rules, based on EHRI requirements and on the data models of the collection holding institution. ODD also allows for the creation of qualitative documentation to be served to the user of conversion and validation services provided by the EHRI project.

EHRI EAD validation process

In EHRI, the EAD specification in ODD inherits everything from the generic EAD ODD except when a more specific behaviour is required by the project. However, the philosophy is to keep the EAD schema as it is. Instead, we use another validation language: ISO Schematron, an  ISO/IEC Standard (ISO/IEC 19757-3:2016) that parses XML documents and makes “assertions about the presence or absence of patterns”. It can be used in conjunction with a large number of grammar languages such as DTD, RELAX NG, etc.

ODD topics – adapting the TEI documentation with one’s own examples

Once the decision has been taken to build up an ODD schema that documents the usages of the TEI guidelines within a given digital humanities project, a first step towards customising the guidelines can be to adapt the documentation to reflect the specific practices of the project. A good way to do so is to work specifically on the examples associated to element description, which can easily be changed in the ODD specification.

Let us imagine we create a simple specification for a dictionary project, thus integrating the modules: core, tei, header, textstructure and dictionaries. We can add a specification that changes the element <entry> as follows:

<elementSpec ident="entry" mode="change">
      <egXML xmlns="http://www.tei-c.org/ns/Examples">
               <entry xml:it="e-chat" xml:lang="fr">

What we have done here is basically two-fold:

  • Mark that we change the specification of the element by means of the @mode attribute set to “change” on the <elementSpec> element;
  • Introduce an <exemplum> element with the example we want to record for this element.

When compiled to create an XHTML output, all pre-existing examples for from the TEI guidelines are replaced by the one(s), that have been introduced that way (see screenshot below).

See the full ODD example under:


TEI Boilerplate: Displaying a facsimile beside a transcription

Charles Riondet, Inria

The TEI Boilerplate is a light framework to transform TEI P5 documents with a minimal XSLT stylesheet and present them in a browser, styled with CSS and Javascript. It has been developed by John Walsh, Grant Simpson and Saeed Moaddeli, from Indiana University.

The aim of this solution is to display directly the XML file, skipping the transformation in HTML. The usefulness of such a solution is obvious in order to have a rapid and clear human-readable version of a TEI document. However, TEI Boilerplate can also be a basis for more complex transformations and the production of extensive digital editions.

We would like to show here how we could use the TEI Boilerplate framework to visualize in the same page the edition of a manuscript and the digital image of the original document.
There are several ways to achieve the same result (or even an even more fancy result using elaborate Javascript instructions). Here is one of the simplest, which also gives a few hints of how the users could customize TEI Boilerplate according to their needs with average or even low programming skills.

The principle of TEI Boilerplate customization is very simple: the user can add any constraint in style instruction in two empty files: content/custom.xsl for XSLT processing and CSS/custom.css for styling. Here, we will use mainly the CSS file, but a minor changes in the XSLT templates can be done before.

Page breaks and facsimiles (optional)

What is follows a <pb/> corresponds to the content of a page. The image is associated to the  <pb/> element. TEI boilerplate rightly requires the use of the  @facs attribute to point to the URL of the image.

<pb facs="images/image1.png" n="1"/>

If you prefer to refer to your images in a <facsimile> element, you must modify an XSL template in the file content/teibp.xsl. But to make sure you know exactly the changes you made, I suggest that you copy the template in the file content/custom.xsl and make the change here.

The template to modify is called pb-handler :

<xsl:template name="pb-handler">
 <xsl:param name="n"/>
 <xsl:param name="facs"/>
 <xsl:param name="id"/>

 HERE, IT'S "TEI/facsimile/surface/graphic/@url"-->
 <xsl:variable name="facsUrl" select="//tei:surface[concat('#', @xml:id) = $facs]/tei:graphic/@url"/>
 <span class="-teibp-pageNum">
 <!-- <xsl:call-template name="atts"/> -->
 <span class="-teibp-pbNote">
 <xsl:value-of select="$pbNote"/>
 <xsl:value-of select="@n"/>
 <xsl:text> </xsl:text>
 <span class="-teibp-pbFacs">
 <a class="gallery-facs" rel="prettyPhoto[gallery1]">
 <xsl:attribute name="onclick">

<!-- replace the parameter $facs with the created variable $facsUrl -->

<xsl:value-of select="concat('showFacs(',$apos,$n,$apos,',',$apos,$facsUrl,$apos,',',$apos,$id,$apos,')')"/>

 <img alt="{$altTextPbFacs}" class="-teibp-thumbnail">

<!-- Replace also the value of the html attribute @src ( which is @facs), by the variable $facsUrl -->

<xsl:attribute name="src">
 <xsl:value-of select="$facsUrl"/>

CSS styling


To display side by side the transciption and the image, CSS instructions must be add in the custom.css file (CSS/custom.css)
The first widen the page.

 html {
 width: 90%;

Here, you shift the TEI boilerplate toolbox in the right corner of the page, or make it invisible

 #teibpToolbox {
 /* shift the toolbox in the right corner of the page */
 /* make the toolbox invisible */

Then, you can enlarge the image and make it float next to the text

 .-teibp-thumbnail {
 max-height: 1000px;

The following will delete the margin of the whole element containing the page number and the image, and align the tops of the transcripton and the facsimile.

 margin: 0em 0px;

Here, you can make changes to the way the page number is displayed.



These changes are very basic but efficient, and abides to the philosophy of TEI Boilerplate, that is to “largely preserve the integrity of the TEI document” (http://teiboilerplate.org/). For more elaborate CSS instructions, you can find a lot of documentation on websites such as http://www.csszengarden.com/.

Download the TEI Boilerpalte on Github : https://github.com/GrantLS/TEI-Boilerplate
TEI document kindly provided by Magdalena Porst (FHP Potsdam).
Images from :
Institut national de recherche en informatique et en automatique (INRIA, Paris FR) . (2016). Journal intime de Leonore Alt [Corpus]. Disponible sur www.ortolang.fr : http://hdl.handle.net/11403/leonore-alt

From TEI-XML to TXM, HTML and back

XSL Style Sheet – Strategy for an Import into TXM

Representations of the Other in the British, French and German Discourse on Europe: A Corpus-Based Contrastive Discursive Analysis

Naomi Truan (Université Paris-Sorbonne / Freie Universität Berlin)

The purpose of this article is to present a successful strategy to import the corpus on which my PhD project is based into the open source software TXM. If you still do not know TXM, click on the link! This software, developed at the Ecole Normale Supérieure (Lyon, France), is open source, regularly updated, and offers a wide range of textometric queries (among them cooccurrences and concordancies, but also many more).

The corpus is delivered with the TEI-XML files and two Stylesheets for the import and the conversion/visualisation in an HTML file (like new shampoos: 2-in-1!). The complete version of this document (with figures and two appendixes) is also to be found on the ORTOLANG server as a PDF document on my workspaces: French, English, German.

As a student in German Studies (focusing mainly on Literature, Philosophy, History, and Translation), I had no prior experience in the huge – but magnificent – world of Computational Linguistics. I discovered it through my PhD in Corpus Linguistics and thanks to the help of Laurent Romary and the TXM team (especially Serge Heiden, Alexey Lavrentev, and Bénédicte Pincemin). Thus, this article is not only here to show you how it works, but also to tell you: Yes, you can! Also with no idea on Computational Linguistics, you can acquire and develop your own strategies to make sense of these weird languages computer engineers work with.

My PhD thesis, entitled “Representations of the Other in the British, French and German Discourse on Europe: A Corpus-Based Contrastive Discursive Analysis”, relies on a qualitative and quantitative linguistic analysis of parliamentary debates in three European countries. You can find the corpora and their accurate description on ORTOLANG: French, English, German.

The corpus has been manually annotated according to the TEI Guidelines. If you wish to consult how the corpus was annotated, please see the document entitled “Corpus Annotation” on the ORTOLANG server (links above).

I – Import into TXM

To import the XML-TEI files into TXM, you have to follow the steps of an “Import TEI générique conservateur” and to select the Import XML/w +CSV. 

You have to put all the TEI-XML files into one folder and to add a file named import.properties (with no extension such as .doc, .pdf), in which you write ignoredelements=note|bibl. This way, the statistics of the corpus in TXM will not take into consideration (ignore) the TEI-XML tags <note> and <bibl>.

Then click on “Sélectionner le répertoire des sources” and select the corresponding folder. Please note that for your first import, the import.xml file, which is created during the import, will not be in the folder. If you happen to modify the TEI-XML files, the import.xml file will be recreated at each import.

In “Dossier des sources”, you can add information on the corpus. If you use the corpus I annotated for your own research project, I kindly ask you to refer to it in this section, for instance with following mention: Naomi Truan 2016 – CC BY 4.0.

The “Police d’affichage” depends on your personal taste and does not affect the import at all.

In the section “Langue principale”, do not forget to tick “en” for English, “de” for German, and “fr” for French if you wish to have the corpus syntactically annotated with TreeTagger (the tutorial for installing TreeTagger on TXM is here).

The “Paramètres du segmenteur lexical” do not need to be changed.

For the “Feuille XSL d’entrée”, please use the style sheet in an XSL format provided along with the TEI-XML files, freely adapted from txm-filter-teip5-xmlw-preserve.xsl.

“Editions” and “Commandes” do not need to be changed.

The import of the corpus can begin; you can now visualise the metadata of the corpus by clicking on the information icon.

Please note that the first “Propriétés des unités lexicales” (body, desc, incident, quote, seg) are not reliable; the given numbers do not correspond to anything in the corpus.

On the text level and on the utterance level, though, the information is fully accurate, so that the following metadata enable correct partitions of the corpus according to these variables: date, government, id, party, party-type, position, role, sex, who-party, who-party-type, who-position, who-role, who-sex.

II – HTML Visualisation of the Corpus with the XSL Style Sheet

I will now comment on the XSL Style Sheet, which can be used for the import into TXM (see Part I), but also to enable the visualisation of the corpus as a whole in an HTML-format. On the ORTOLANG server, you can find it under Content > UK TEI-XML Files, or here.

In the XSL Style Sheet, information in green such as <!– Corpus of British Parliamentary Debates –> does not impact on the XSL Style Sheet but simply provides information to guide the reader.

If you open the XSL Style Sheet and the XML Style Sheet together in oXygen XML Editor, and click, within the XML Style Sheet, on the red button on your right, then oXygen XML Editor will automatically run the Style Sheet and open the corpus in an HTML-format in your browser (like a webpage).

Alternatively, and if you have not proceeded to any changes in the corpus, just double-click on the “HTML file UK – Style sheet (2)”, which will automatically open in your browser as well. The procedure described above is necessary only if you wish to encode other tags or to visualise them differently (for instance, if you wish to see the <quote> tags in orange rather than in red) or if you add new tags in the corpus (for instance, if you notice a missing <quote> tag in one of the TEI-XML files of the corpus).

You can then scroll down through the corpus. It enables you a quick search (for instance through ctrl F for people not familiar with TXM, which offers much more queries in this regard) and a quick visualisation (for instance if you feel you have a better impression of the length of an utterance by seeing it on a webpage – how many lines? – rather than counting text units).

The corpus begins with general information, such as: Number of Incidents, Number of Turns, Number of Speakers, Number of Opposition Members, Number of Majority Members. In this regard, I strongly advise to rely on the statistics provided by TXM rather than on the XSL Style Sheet, which appears to be sometimes misleading. For instance, it counts seven Plaid Cymru Members but reports four names, which is inconsistent (there are, actually, fourPlaid Cymru Members):

Plaid Cymru Members: DAFIS – LLWYD – THOMAS – WIGLEY
Number of Plaid Cymru Members: 7
Number of Turns of Plaid Cymru Members: 7

This is it! Normally, every time you adjust the corpus (correct a typo, do some minor changes, rename a speaker, etc.), the HTML version will follow, enabling you to visualise the last version corpus very quickly and to search through it. At the same time, you can re-import the corpus into TXM by following the previous steps. If you do not rename the corpus, TXM will automatically ask you if you wish to replace the existing corpus. By clicking “yes”, you will update the corpus.

You now can see how to visualise (i.e. make nice!) your corpus through an XSL Style Sheet and an XML Style Sheet especially designed for the purposes of your own research. The Style Sheets can be adapted for every type of corpus following the TEI Guidelines. Thus, it should not be seen as a model, but rather as an example suitable for TXM.

Please feel free to contact me for any question regarding TXM, TEI, XML and HTML formats, but also Corpus Linguistics, Cognitive Linguistics, or Political Discourse! I cannot promise you to be able to answer every technical concern, but I will try my best (and can also forward your question(s) to people who know more than I do): Naomi.Truan@paris-sorbonne.fr. Any comment will be much appreciated!

Next generation <etym> in the TEI – Putting together a proposal

After the discussion on the TEI list and the comments on the list, the post and twitter exchanges with @ttasovac, I came back to an old paper of Susanne Alt (http://hal.archives-ouvertes.fr/hal-00110971), which provides an overview of the constraints that could apply to a comprehensive representation of etymological information in a dictionary entry.

Susanne uses the entry “Pamplemousse” from the TLF (Trésor de la Langue Française) to encode the following etymological information:

PAMPLEMOUSSE, subst. masc. […]

Empr. au néerl. pompelmoes, fém., au sens 1 a, qui est prob. comp.de pompel «gros, enlé» et de limoes «citron» (BOULAN, p.148; KÖNIG, pp.159-160). Apparaît d’abord dans des textes fr. qui le donnent comme mot néerl.: 1665 pompelmoes (J. LE CARPENTIER, L’Ambassade de la Compagnie orientale des Provinces Unies… [trad. d’un ouvrage néerl.], II, p.88 ds ARV.); 1666 pompelmous (M. THÉVENOT, Relation de divers voyages curieux… t.3 ds KÖNIG).

At the end of paper, following an in-depth discussion of etymons and links, which I advise you all to read, she suggests the following encoding:

(i.e. the lexical entry pamplemousse and its direct etymon pompelmoes) of the data structure in a TeI like format. Note that the basic building blocks for the characterization of a lexical entry (<form> and <sense>) are completely reus- able for the description of an etymon, and the language has been implemented as a standardized xml:lang attribute.

<lexicalEntry id=”LE1″>
<orth>pamplemousse</orth> …
<sense>… </sense>
<etymology> <etymon id=”LE2″>
<form> <orth xml:lang=”dutch”>pompelmoes</orth> <pos>commonNoun</pos> <gender>feminine</gender>
</form> <sense>
<glose>Citrus maxima</glose>
<note>probablement d’origine tamoule, De Vries, Nederl</note> </sense>
</etymon> <etymologicalLink source=”LE2″ target=”LE1″>
<etymologicalClass>loan word</etymologicalClass> </etymologicalLink>
</etymology> </lexicalEntry>

If we “modernize” this proposal to make it as close to a TEI compliant entry, we could think of something as follows:
<entry xml:id=”LE1″>
<orth>pamplemousse … </form>
<re type=”etymon” xml:id=”LE2″>
<orth xml:lang=”dutch”>pompelmoes</orth>
<gloss>Citrus maxima</gloss>
<note>probablement d’origine tamoule, De Vries, Nederl</note>
<link type=”loanWord” target=”#LE2 #LE1″/>

This is a little more ambitious than what was discussed in the previous post and related comments, but seems to follow what Toma Tasovac had in mind with an embedded entry in the description of the etymological content.

From print to code – multiple forms in one

An interesting discussion started on the TEI list (http://listserv.brown.edu/archives/cgi-bin/wa?A1=ind1309&L=TEI-L#14) about dictionary encoding, which I would like to reflect here to point to a couple of issues related to forms in dictionaries.

The post was sent by Susanna Allés, who is currently working on a Latin medieval dictionary. Here’s the initial question:

“we can’t find a satisfactory solution to encode a term (<term>) that corresponds to several languages. Here is the text:

arrencare  [cat. arrencar, arrancar (cf. oc., esp., port. arrancar), d’origen discutit | de origen discutido | of debated origin]  rompre roturar to break up: 1038 ACUrgell, Cart. I 357, f. 118, col. 2: terra quae fuit arrenchada qui est in Subuineas quae escamiauit Ermengauda a me Mirone. De una parte affrontat … in terra quae arrencauit … de IIII uero parte in ipsa fonnada de terra de Riamball que arrencauit.

The part in (square) brackets corresponds to the etymology (encoded with <etym>) and in red (cf. oc., esp., port. arrancar) corresponds to a term (the Romanic result); we want to relate this term with several languages (that it is, to Occitan, Spanish, Portuguese, in this case). Usually, if we are dealing with just a language, we encode it as <term xml:lang=”es”>arrancar</term> , but since we need to add other languages and the element term just allow one @xml:lang, we really don’t know how to handle.”

For sake of place, the original editor of the dictionary has used the same character sequence to represent three “words” from three different languages, which just happen, by chance (and linguistic proximity…), to be written in the same way.

The first line of thought, which considers the source text at face value may lead (like stated in the question) to  one could want to refer to three languages in relation to the sequence “arrancar”. Possibilities that have been alluded to on the list are:

  • putting more that one language codes as values for xml:lang, theoretically feasible, given the recent evolution of the corresponding W3C specification, not in the TEI though and probably an “interoperability nightmare” (quoting Stuart Yeates)
  • tagging the language references with <lang>, as suggested by Sabine Thuillier, in keeping to the basic constructs associated to <etym> (etymology) in the TEI guidelines (i.e. ​cf. ​<lang>oc., esp., port.</lang> <mentionned>arrancar</mentionned>​)​; Sabine rightly points to the fact that this solution is not optimal from a processing point of view.

Still, a more lexicographic analysis seems to lead to another path which consists in reconstructing the underlying logical representation, as nicely stated by Toma Tasovac:

“what we have here is typical lexicographic shorthand —  we know how lexicographers used to be stingy when it comes to space —  but that, if we really think about it, we are not (on the conceptual, modeling, even philosophical level) talking about one word being “the same” in three languages, but rather about three “different” words from three different languages that happen to have the same graphical representation.

What one could therefore also do is encode three instances of “arrancar” and each with an xml:lang for a different language.”

It is interesting to observe that several contributors, even being in line with this analysis, have actually suggested mechanisms to retain the “sameness” in the encoding (using @sameas, @exclude or <alt>; Syd Baumann actually provided quite a comprehensive set of possibilities)). No, Toma (supported by David Birnbaum) put it right:

“if one wanted to preserve the economic display in the end output, one would have to use XSLT or whatever one uses to check whether subsequent terms have the same graphical form and then display only one term, but display language ids from all of them.”

Which means that we need to produce an encoding where nothing may lead a processor to think that we have the “same” form, but it just happened that the three were printed out as one accidentally.

Finally, let me add that there is an issue with the encoding of “arrancar” as <term>. It is obviously a <form> (even better a <form>/<orth> if we anticipate that we could provide additional pronunciation information.

So my take at the encoding of this example would be as follows:

(cf. <lang>oc.</lang>, <lang>esp.</lang>, <lang>port.</lang> <form xml:lang=”oc”><orth>arrancar</orth></form><form xml:lang=”sp” ><orth>arrancar</orth></form><form xml:lang=”pt”><orth>arrancar</orth></form>)

But there is a hack here… <form> is not allowed there. Shall we change the content model of <etym>?

Morpho-syntactic annotations: the theory

Since I keep receiving questions concerning the relation between the TEI guidelines and POS annotations, let me start a series of posts on the issue. As a first step, I present here the basic concepts of the underlying ISO standard: ISO 24611:2012 Language resource management — Morpho-syntactic annotation framework (MAF).

ISO 24611 provides a framework for the representation of morpho-syntactic (also referred to as part-of-speech) annotations. Such annotations correspond to a first lexical abstraction over language data (textual or spoken) and, depending on the language to be annotated, as well as the characteristics of the annotation tool or annotation scheme that is being used, can vary enormously in structure and complexity.

In order to deal with such complex issues as ambiguity and determinism in morpho-syntactic annotation, ISO 24611 introduces a meta-model that draws a clear distinction between the two levels of tokens (representing the surface segmentation of the source) and word-forms (identifying lexical abstractions associated with groups of tokens). Both levels can be represented as simple sequences and as local graphs such as multiple segmentations and ambiguous compounds. Besides, any n-to-n combination can stand between word forms and tokens.

As linguistic segments (sometimes called ‘markables’ in the literature, see for instance, Carletta et al. 1997), tokens may be embedded in the source document as inline mark-up, or they may point remotely to it by means of so-called stand-off annotations.

As linguistic abstractions, word-forms can be qualified by various linguistic features characterising the morpho-syntactic properties that are instantiated in the realisation of the lexical entry within the annotated text. Such properties may range from the simple indication of a lemma up to an explicit reference to a lexical entry in a dictionary. In most existing applications of morpho-syntactic annotation, linguistic properties are expressed by means of so-called tags; these codes refer to basic feature structures (see early examples in Monachini and Calzolari, 1994). Such codes may also provide morphological information, including its part of speech (e.g. noun, adjective or verb), and features such as number, gender, person, mood and verbal tense.

In keeping with the general modelling strategy of ISO/TC 37, MAF provides means of relating morpho-syntactic tags expressed as feature structures (compliant with ISO 24610) to the data categories available in ISOCat.org. A normative annex of this International Standard elicits a core set of data categories that can be used as reference for most current morpho-syntactic annotation tasks in a multilingual context. However, when implementers of the standard find these categories inappropriate in either coverage, scope or semantics, they are encouraged to use ISOCat to define their own categories.

Associated to the meta-model, MAF also provides a default XML serialisation. Still, as we shall see in a further post, it is possible to combine a TEI based representation with MAF compliant annotations (See also Romary and Witt, 2013).


ISO 24610-1:2006 Language resource management — Feature structures — Part 1: Feature structure representation (see also the corresponding chapter in the TEI guidelines: http://www.tei-c.org/release/doc/tei-p5-doc/en/html/FS.html)

Jean Carletta, Nils Dahlbäck, Norbert Reithinger and Marilyn A. Walker (Eds.) (1997) Standards for Dialogue Coding in Natural Language Processing, Dagsthul-Seminar Report 167; 03.02.-07.02.97 (9706)

Monachini, Monica and Nicoletta Calzolari (1994). Synopsis and Comparison of Morpho-syntactic Phenomena Encoded in Lexicon and Corpora. A Common Proposal and Applications to European Languages. Internal Document, EAGLES Lexicon Group, ILC, Università Pisa, Oct. 1994

Romary, Laurent and Andreas Witt (2013), “Data formats for phonological corpora”, in Handbook of Corpus Phonology Oxford University Press (Ed.). http://hal.inria.fr/inria-00630289

Scholarly work and Open Access

The whole idea of scholarship is oriented towards maximising the dissemination of research results. Carrying out a research activity is all about exploring territories, where knowing what the others are doing, what their most recent advances are, what projects are being undertaken, is essential to make sure that one’s own research actually goes beyond the state of the art and can be situated within a larger corpus of discoveries. Communicating results is thus an essential activity in one’s academic life, all the more that the assessment of such communications through peer review mechanisms impact on the capacity to get institutional recognition and thus financial means to carry out further research.

This is the context in which research organisations should be designing their own Open Access (OA) policies in order to help researchers to get access to existing publications (traditionally through journal subscriptions), publish their own results to a wide audience (by means of publication repositories[1]) and manage associated research assets (laboratory notes, observations, primary sources, databases). This has been made particularly difficult in the recent years because of the incredible increase in publication prices imposed by private publishers, which has been seen as contradicting the intuition that new technologies should indeed simplify dissemination rather than increase operational publication costs. The situation has become all the more unbearable that most of the various processes bringing a research manuscript to publication are carried out for free by researchers themselves.

Even if the OA movement is quite recent, it is possible to outline some principles and possible action lines. Among the various meetings which, in the early years 2000, contributed to stabilize the basic notions of Open Access, we can quote the core statement from the Berlin Declaration (2003) that requested a “…free, irrevocable, worldwide, right of access to, and a license to copy, use, distribute, transmit and display the work publicly and to make and distribute derivative works, in any digital medium for any responsible purpose…. ”. Not only does this statement reflect the feeling that the current landscape is inadequate to fulfil the communication needs of the research community, but it also outlines a possible ideal scheme where commercial constraints would have a lesser importance than issues related to public good and wealth.

There are basically two ways to implement such a change in the scholarly publishing environment. The first one (also called green open access) consists in letting the journal publishing industry carry out its business as is and deploy an infrastructure of publication repositories to freely disseminate authors’ versions online. Such an infrastructure may be deployed at the level of a research department, a university or cover a wider geographical or institutional spectrum. In the case of France, a national publication archive infrastructure, HAL[2], has been deployed to cover the needs of most French academic organisations. It is now part of the official Open Access policy of the French Ministry of Research and Higher Education[3]. Such publication platforms are particularly important since they support the immediate dissemination of research papers from an early drafting stage to its final publication. It is also a way for research institutions to get a global picture of all research carried out under their auspices, since it is associated with precise descriptive information (for instance, affiliations) that can be curated by librarians in a coherent way[4].

There is also an increasing demand from scholars to benefit from trusted repository environments where they can deposit their research data with the guaranty that these will be accessible and referentiable in the long-term. Recent developments in the Netherlands with the NARCIS repository[5] for example have shown that such environments can even be coupled with a traditional publication archive.

The other way to implement an OA policy (also called the gold way to OA) is to replace the current subscription-based model by other business models both allowing “fair” publishing organisations to break even in maintaining a publication framework and designing a barrier-free system for the access and reuse of research results. Even if the publishing industry has recently perverted this golden way by introducing an article-processing-charge system that only reshapes what used to be subscription costs into an author-pay model, we will see in the rest of this post that there is room for other perspectives to reform the publication landscape.

A first strategy may be to think of alternative business models that are based on an ethical support to scholarly publishing within an OA perspective. This is what has been proposed by the OpenEdition publishing infrastructure with their Freemium model. It is based on the principle that basic access (for instance in HTML) to published material should be made open for free but that additional services can be sold to libraries (pdf, ePub, cataloguing services) for a reasonable fee. The corresponding benefits are in turn given back to the journals so that they can cover their day-to-day costs. Indeed, as is the case for OpenEdition, the core infrastructure as well as the general editorial support is part of the institutionally funded infrastructure.

If we are still attached to the traditional journal editorial setting, we can observe that its core services, namely identification, certification, dissemination and long-term availability, can be easily implemented on the basis of an existing publication repository. Indeed, such a repository provides a submission environment, which identifies authors and time-stamp the document, and offers a perfect online dissemination platform, with the necessary long-term archiving facility of the hosting institution. In such a context, designing a certification environment mechanism whereby a paper deposited by an author is forwarded to an editorial committee for peer-review, is quite a straightforward endeavour. This is exactly what is now being experimented with the Episciences[6] project on top of the HAL platform. Such a platform is also interesting in that it offers new possibilities for changing our perspective on the certification process: open submission, open peer-review[7], updated versions of an article and community feedback are features that may dramatically change our views on scholarly publishing.

At specific stages of the research process, it is often not so much important to produce an in-depth scholarly than to provide short snapshots on the current developments of an experiment in hard sciences, or the analysis of a source in the humanities. This is a situation where it is more appropriate for a scholar to write small reports in the form of blog entries and publicize them on various social networks. Blogs offer a first layer of scholarly publication with both online availability and the possibility to comment on the actual scholarly content. It is also a simple way to gain a primacy for a specific result or gather observations step-by-step, for instance during an archaeological campaign. The ideal situation is when blogging occurs within a secured scholarly environment such as Hypotheses.org where researchers benefit from an editorial support as well as a wide visibility.

The various possibilities outlined so far only make sense if research institutions invest time, political energy and budget to implement such models and make them part of the daily life of their researchers. A typical best practice example can be taken from the recently published open access policy of Inria[8] which combines a deposit mandate of all publications on the HAL archive, a cautious assessment of any new models provided by the private publishing sector and the funding of the Episciences platform.

We can observe that having a not too overly conservative vision on scholarly communication opens up a whole range of possibilities to improve the way scientific ideas can be seamlessly transmitted to a wide audience. Even more, we can see that a new landscape can be outlined where the management of virtual research environments comprising research data, various types of notes and commentaries, as well as drafted documents linking these objects together could dramatically change the way scholarship will be carried out in the future. In such environments, various levels of “peer-review” are possible, from the simple feedback of known colleagues to the possibility for any member of a research community to comment on the content. Traditional peer-review is just one possible implementation of such a model where the main objective should remain to improve quality and wide accessibility for science.

As a whole, I defend a vision of scholarly communication which is entrenched in the wider notion of research infrastructure and which as such must be considered as part of the realm of public research institutions. We need to see what the consequences of such a vision are, in terms of budget shifts, investments in technological settings, but also in changing the roles of research libraries so that they can provide the necessary editorial support to such environments. The change may be drastic, but I think this is the only way to optimize tax-payers’ money at the service of science.


[1] see Romary Laurent & Chris Armbruster “Beyond Institutional Repositories”, International Journal of Digital Library Systems, 1(1), January 2010 — http://hal.inria.fr/hal-00399881/

[2] hal.archives-ouvertes.fr

[3] See http://www.enseignementsup-recherche.gouv.fr/cid71277/partenariat-en-faveur-des-archives-ouvertes-plateforme-mutualisee-hal.html

[4] Note that the quality of such descriptive information is one of the difficulties I see in such commercial platforms as Academia.edu or Research Gate.

[5] http://www.narcis.nl

[6] http://www.nature.com/news/mathematicians-aim-to-take-publishers-out-of-publishing-1.12243

[7] see Pöschl U. (2010) “Interactive open access publishing and peer review: the effectiveness and perspectives of transparency and self-regulation in scientific communication and evaluation”. LIBER Q. 19, 293–314.

[8] see http://tonyhey.net/2013/06/03/a-global-view-of-open-access-part-1/