From TEI-XML to TXM, HTML and back

XSL Style Sheet – Strategy for an Import into TXM

Representations of the Other in the British, French and German Discourse on Europe: A Corpus-Based Contrastive Discursive Analysis

Naomi Truan (Université Paris-Sorbonne / Freie Universität Berlin)

The purpose of this article is to present a successful strategy to import the corpus on which my PhD project is based into the open source software TXM. If you still do not know TXM, click on the link! This software, developed at the Ecole Normale Supérieure (Lyon, France), is open source, regularly updated, and offers a wide range of textometric queries (among them cooccurrences and concordancies, but also many more).

The corpus is delivered with the TEI-XML files and two Stylesheets for the import and the conversion/visualisation in an HTML file (like new shampoos: 2-in-1!). The complete version of this document (with figures and two appendixes) is also to be found on the ORTOLANG server as a PDF document on my workspaces: French, English, German.

As a student in German Studies (focussing mainly on Literature, Philosophy, History, and Translation), I had no prior experience in the huge – but magnificent – world of Computational Linguistics. I discovered it through my PhD in Corpus Linguistics and thanks to the help of Laurent Romary and the TXM team (especially Serge Heiden, Alexey Lavrentev, and Bénédicte Pincemin). Thus, this article is not only here to show you how it works, but also to tell you: Yes, you can! Also with no idea on Computational Linguistics, you can acquire and develop your own strategies to make sense of these weird languages computer engineers work with.

My PhD thesis, entitled “Representations of the Other in the British, French and German Discourse on Europe: A Corpus-Based Contrastive Discursive Analysis”, relies on a qualitative and quantitative linguistic analysis of parliamentary debates in three European countries. You can find the corpora and their accurate description on ORTOLANG: French, English, German.

The corpus has been manually annotated according to the TEI Guidelines. If you wish to consult how the corpus was annotated, please see the document entitled “Corpus Annotation” on the ORTOLANG server (links above).

I – Import into TXM

To import the XML-TEI files into TXM, you have to follow the steps of an “Import TEI générique conservateur” and to select the Import XML/w +CSV. 

You have to put all the TEI-XML files into one folder and to add a file named import.properties (with no extension such as .doc, .pdf), in which you write ignoredelements=note|bibl. This way, the statistics of the corpus in TXM will not take into consideration (ignore) the TEI-XML tags <note> and <bibl>.

Then click on “Sélectionner le répertoire des sources” and select the corresponding folder. Please note that for your first import, the import.xml file, which is created during the import, will not be in the folder. If you happen to modify the TEI-XML files, the import.xml file will be recreated at each import.

In “Dossier des sources”, you can add information on the corpus. If you use the corpus I annotated for you own research project, I kindly ask you to refer to it in this section, for instance with following mention: Naomi Truan 2016 – CC BY 4.0.

The “Police d’affichage” depends on your personal taste and does not affect the import at all.

In the section “Langue principale”, do not forget to tick “en” for English, “de” for German, and “fr” for French if you wish to have the corpus syntactically annotated with TreeTagger (the tutorial for installing TreeTagger on TXM is here).

The “Paramètres du segmenteur lexical” do not need to be changed.

For the “Feuille XSL d’entrée”, please use the style sheet in an XSL format provided along with the TEI-XML files, freely adapted from txm-filter-teip5-xmlw-preserve.xsl.

“Editions” and “Commandes” do not need to be changed.

The import of the corpus can begin; you can now visualise the metadata of the corpus by clicking on the information icon.

Please note that the first “Propriétés des unités lexicales” (body, desc, incident, quote, seg) are not reliable; the given numbers do not correspond to anything in the corpus.

On the text level and on the utterance level, though, the information is fully accurate, so that the following metadata enable correct partitions of the corpus according to these variables: date, government, id, party, party-type, position, role, sex, who-party, who-party-type, who-position, who-role, who-sex.

II – HTML Visualisation of the Corpus with the XSL Style Sheet

I will now comment on the XSL Style Sheet, which can be used for the import into TXM (see Part I), but also to enable the visualisation of the corpus as a whole in an HTML-format. On the ORTOLANG server, you can find it under Content > UK TEI-XML Files, or here.

In the XSL Style Sheet, information in green such as <!– Corpus of British Parliamentary Debates –> does not impact on the XSL Style Sheet but simply provides information to guide the reader.

If you open the XSL Style Sheet and the XML Style Sheet together in oXygen XML Editor, and click, within the XML Style Sheet, on the red button on your right, then oXygen XML Editor will automatically run the Style Sheet and open the corpus in an HTML-format in your browser (like a webpage).

Alternatively, and if you have not proceeded to any changes in the corpus, just double-click on the “HTML file UK – Style sheet (2)”, which will automatically open in your browser as well. The procedure described above is necessary only if you wish to encode other tags or to visualise them differently (for instance, if you wish to see the <quote> tags in orange rather than in red) or if you add new tags in the corpus (for instance, if you notice a missing <quote> tag in one of the TEI-XML files of the corpus).

You can then scroll down through the corpus. It enables you a quick search (for instance through ctrl F for people not familiar with TXM, which offers much more queries in this regard) and a quick visualisation (for instance if you feel you have a better impression of the length of an utterance by seeing it on a webpage – how many lines? – rather than counting text units).

The corpus begins with general information, such as: Number of Incidents, Number of Turns, Number of Speakers, Number of Opposition Members, Number of Majority Members. In this regard, I strongly advise to rely on the statistics provided by TXM rather than on the XSL Style Sheet, which appears to be sometimes misleading. For instance, it counts seven Plaid Cymru Members but reports four names, which is inconsistent (there are, actually, fourPlaid Cymru Members):

Plaid Cymru Members: DAFIS – LLWYD – THOMAS – WIGLEY
Number of Plaid Cymru Members: 7
Number of Turns of Plaid Cymru Members: 7

This is it! Normally, every time you adjust the corpus (correct a typo, do some minor changes, rename a speaker, etc.), the HTML version will follow, enabling you to visualise the last version corpus very quickly and to search through it. At the same time, you can re-import the corpus into TXM by following the previous steps. If you do not rename the corpus, TXM will automatically ask you if you wish to replace the existing corpus. By clicking “yes”, you will update the corpus.

You now can see how to visualise (i.e. make nice!) your corpus through an XSL Style Sheet and an XML Style Sheet especially designed for the purposes of your own research. The Style Sheets can be adapted for every type of corpus following the TEI Guidelines. Thus, it should not be seen as a model, but rather as an example suitable for TXM.

Please feel free to contact me for any question regarding TXM, TEI, XML and HTML formats, but also Corpus Linguistics, Cognitive Linguistics, or Political Discourse! I cannot promise you to be able to answer every technical concern, but I will try my best (and can also forward your question(s) to people who know more than I do): Naomi.Truan@paris-sorbonne.fr. Any comment will be much appreciated!

Next generation <etym> in the TEI – Putting together a proposal

After the discussion on the TEI list and the comments on the list, the post and twitter exchanges with @ttasovac, I came back to an old paper of Susanne Alt (http://hal.archives-ouvertes.fr/hal-00110971), which provides an overview of the constraints that could apply to a comprehensive representation of etymological information in a dictionary entry.

Susanne uses the entry “Pamplemousse” from the TLF (Trésor de la Langue Française) to encode the following etymological information:

PAMPLEMOUSSE, subst. masc. […]

Empr. au néerl. pompelmoes, fém., au sens 1 a, qui est prob. comp.de pompel «gros, enlé» et de limoes «citron» (BOULAN, p.148; KÖNIG, pp.159-160). Apparaît d’abord dans des textes fr. qui le donnent comme mot néerl.: 1665 pompelmoes (J. LE CARPENTIER, L’Ambassade de la Compagnie orientale des Provinces Unies… [trad. d’un ouvrage néerl.], II, p.88 ds ARV.); 1666 pompelmous (M. THÉVENOT, Relation de divers voyages curieux… t.3 ds KÖNIG).

At the end of paper, following an in-depth discussion of etymons and links, which I advise you all to read, she suggests the following encoding:

(i.e. the lexical entry pamplemousse and its direct etymon pompelmoes) of the data structure in a TeI like format. Note that the basic building blocks for the characterization of a lexical entry (<form> and <sense>) are completely reus- able for the description of an etymon, and the language has been implemented as a standardized xml:lang attribute.

<lexicalEntry id=”LE1″>
<form>
<orth>pamplemousse</orth> …
</form>
<sense>… </sense>
<etymology> <etymon id=”LE2″>
<form> <orth xml:lang=”dutch”>pompelmoes</orth> <pos>commonNoun</pos> <gender>feminine</gender>
</form> <sense>
<glose>Citrus maxima</glose>
<note>probablement d’origine tamoule, De Vries, Nederl</note> </sense>
</etymon> <etymologicalLink source=”LE2″ target=”LE1″>
<etymologicalClass>loan word</etymologicalClass> </etymologicalLink>
</etymology> </lexicalEntry>

If we “modernize” this proposal to make it as close to a TEI compliant entry, we could think of something as follows:
<entry xml:id=”LE1″>
<form>
<orth>pamplemousse … </form>
<sense>…
<etym>
<re type=”etymon” xml:id=”LE2″>
<form>
<orth xml:lang=”dutch”>pompelmoes</orth>
<gramGrp>
<pos>commonNoun</pos>
<gen>feminine</gen>
</gramGrp>
</form>
<sense>
<gloss>Citrus maxima</gloss>
<note>probablement d’origine tamoule, De Vries, Nederl</note>
</sense>
</re>
<link type=”loanWord” target=”#LE2 #LE1″/>
</etym>
</entry>

This is a little more ambitious than what was discussed in the previous post and related comments, but seems to follow what Toma Tasovac had in mind with an embedded entry in the description of the etymological content.

From print to code – multiple forms in one

An interesting discussion started on the TEI list (http://listserv.brown.edu/archives/cgi-bin/wa?A1=ind1309&L=TEI-L#14) about dictionary encoding, which I would like to reflect here to point to a couple of issues related to forms in dictionaries.

The post was sent by Susanna Allés, who is currently working on a Latin medieval dictionary. Here’s the initial question:

“we can’t find a satisfactory solution to encode a term (<term>) that corresponds to several languages. Here is the text:

arrencare  [cat. arrencar, arrancar (cf. oc., esp., port. arrancar), d’origen discutit | de origen discutido | of debated origin]  rompre roturar to break up: 1038 ACUrgell, Cart. I 357, f. 118, col. 2: terra quae fuit arrenchada qui est in Subuineas quae escamiauit Ermengauda a me Mirone. De una parte affrontat … in terra quae arrencauit … de IIII uero parte in ipsa fonnada de terra de Riamball que arrencauit.

The part in (square) brackets corresponds to the etymology (encoded with <etym>) and in red (cf. oc., esp., port. arrancar) corresponds to a term (the Romanic result); we want to relate this term with several languages (that it is, to Occitan, Spanish, Portuguese, in this case). Usually, if we are dealing with just a language, we encode it as <term xml:lang=”es”>arrancar</term> , but since we need to add other languages and the element term just allow one @xml:lang, we really don’t know how to handle.”

For sake of place, the original editor of the dictionary has used the same character sequence to represent three “words” from three different languages, which just happen, by chance (and linguistic proximity…), to be written in the same way.

The first line of thought, which considers the source text at face value may lead (like stated in the question) to  one could want to refer to three languages in relation to the sequence “arrancar”. Possibilities that have been alluded to on the list are:

  • putting more that one language codes as values for xml:lang, theoretically feasible, given the recent evolution of the corresponding W3C specification, not in the TEI though and probably an “interoperability nightmare” (quoting Stuart Yeates)
  • tagging the language references with <lang>, as suggested by Sabine Thuillier, in keeping to the basic constructs associated to <etym> (etymology) in the TEI guidelines (i.e. ​cf. ​<lang>oc., esp., port.</lang> <mentionned>arrancar</mentionned>​)​; Sabine rightly points to the fact that this solution is not optimal from a processing point of view.

Still, a more lexicographic analysis seems to lead to another path which consists in reconstructing the underlying logical representation, as nicely stated by Toma Tasovac:

“what we have here is typical lexicographic shorthand —  we know how lexicographers used to be stingy when it comes to space —  but that, if we really think about it, we are not (on the conceptual, modeling, even philosophical level) talking about one word being “the same” in three languages, but rather about three “different” words from three different languages that happen to have the same graphical representation.

What one could therefore also do is encode three instances of “arrancar” and each with an xml:lang for a different language.”

It is interesting to observe that several contributors, even being in line with this analysis, have actually suggested mechanisms to retain the “sameness” in the encoding (using @sameas, @exclude or <alt>; Syd Baumann actually provided quite a comprehensive set of possibilities)). No, Toma (supported by David Birnbaum) put it right:

“if one wanted to preserve the economic display in the end output, one would have to use XSLT or whatever one uses to check whether subsequent terms have the same graphical form and then display only one term, but display language ids from all of them.”

Which means that we need to produce an encoding where nothing may lead a processor to think that we have the “same” form, but it just happened that the three were printed out as one accidentally.

Finally, let me add that there is an issue with the encoding of “arrancar” as <term>. It is obviously a <form> (even better a <form>/<orth> if we anticipate that we could provide additional pronunciation information.

So my take at the encoding of this example would be as follows:

(cf. <lang>oc.</lang>, <lang>esp.</lang>, <lang>port.</lang> <form xml:lang=”oc”><orth>arrancar</orth></form><form xml:lang=”sp” ><orth>arrancar</orth></form><form xml:lang=”pt”><orth>arrancar</orth></form>)

But there is a hack here… <form> is not allowed there. Shall we change the content model of <etym>?

TBX goes TEI

As a first start into this blogging adventure, let me focus on a current activity which I started in the context of the recent ISO/TC 37 general meetings in Pretoria in June. The context is the ongoing revision process for ISO 300421.

Since its P4 edition, the TEI guidelines did not have a chapter on terminology any more. This is indeed missing since the current TEI dictionary chapter is covering semasiological views on lexical data (word to sense) and many applications require an onomasiological perspective (concept to term). For instance, translators, technical writers or language learners often manipulate terminologies, i.e. lexical structures are organised according to concepts, which in turn are subdivided in language sections (all words or expressions used to express the corresponding concept for this language) and then in term sections (all types of information needed to represent and qualify a given term). The underlying model is standardized  in ISO 16642 TMF2. TBX is thus a serialization of TMF.

I have now started to create an TEI/ODD specification that takes up the basic TBX structure, starting at <termEntry> level and allowing these to occur anywhere a traditional dictionary entry (<entry>) would in the TEI architecture.

To illustrate this, let me just show a typical  (yet awfully simple) example. The TBX namespace is a fake one (it is under discussion and we may want to get all this in the TEI name space).

<TEI xmlns=”http://www.tei-c.org/ns/1.0″ xmlns:tbx=”http://www.tbx.org”
xmlns:tei=”http://www.tei-c.org/ns/1.0″>
<teiHeader>…</teiHeader>
<text>
<body>
<div>

<termEntry xml:id=”c5″ xmlns=”http://www.tbx.org”>
<langSet xml:lang=”en”>
<tig>
<term>e-mail</term>
</tig>
</langSet>
<langSet xml:lang=”fr”>
<tig>
<term>courriel</term>
</tig>
</langSet>
</termEntry>

</div>
</body>
</text>
</TEI>

  1. ISO 30042:2008 Systems to manage terminology, knowledge and content — TermBase eXchange (TBX) []
  2. ISO 16642:2003 Computer applications in terminology — Terminological markup framework []