In-depth quality assessment of machine (and human) interpreting is one thing – turning it into an added value for our customers is quite another. This year, we decided to have a go at TH Köln, and the experience was really worth it.
Collaborating with the Oskar Killinger foundation
Everything started with the Oskar Killinger foundation creating a free online educational series called “explainEpilepsy”, designed to make high-quality, evidence-based medical information accessible to people living with epilepsy as well as healthcare professionals. The event was held in German and “interpreted” into English, French and Spanish by AI – in this case Microsoft Teams. So obviously I registered for the event, just to listen in (btw, thanks to AIIC France for bringing my attention to the event!). But luckily, Iris Killinger noticed my job title and contacted me, at first just out of mere curiosity. We met for a chat and it quickly became clear that we had a lot to offer to each other.
The Oskar Killinger Foundation is a small non-profit organisation dedicated to raising awareness of Sudden Death in Epilepsy (SUDEP), a significant yet rarely discussed risk associated with epilepsy. The foundation was established by Oskar’s parents after losing their 14-year-old son to SUDEP in 2019. Much of what the foundation has achieved ever since is based on volunteers’ work, and leading experts contribute their time and expertise to the educational series on a pro bono basis. So who could blame them for resorting to machine interpreting to bring their life-changing and possibly life-saving message out to the world in more than just one language?
What started as a casual chat between Iris Killinger and myself soon turned into a multi-project collaboration. We integrated one of the foundation’s next seminars into our practice course, so that our students could provide human interpretation. One student will be using the same webinar as a case study on lay-expert communication. Discussions are ongoing about providing human subtitling within the SAAM project (thanks, Rebecca Bremer and Carlos de Pablos-Ortega). And – to my researcher’s mind – the best bit: The foundation gave us the recordings of the machine interpretation of one of their online seminars from German into English, French and Spanish for further analysis.
Assessing machine interpreting quality
So my students learned how to use the Assessment Metrics for Interpreting (AMI) and conducted a detailed, sentence-by-sentence analysis of this sample. This process entails identifying deviations between the source and the target text and assigning them to different categories, for example semantic accuracy, omissions, additions, numbers, terminology, syntax, pronunciation etc. In AMI, all deviations are rated as positive or negative deviations, which means errors or improvements. Furthermore, they are assigned degrees of severity: critical, major or minor.
After my students had finished their meticulous work of quality assessment, we finally met with a group of people from the foundation to discuss the results.
We started off by asking our client which of the six high-level deviation categories (Terminology, Content, Form, Style, Context, Delivery) they considered the most important. The answer was quick and straightforward: terminology and content were the issues that mattered most to them.
So we concentrated on those two aspects when presenting the deviation statistics for our three language combinations (German into English/French/Spanish). For the sake of simplicity, in this blog post I will focus on content only, and just the critical negative deviations, i.e. errors that produced sustantive problems in communication.
Content deviations
For all three language combinations, the numbers showed most negative deviations clustering in the sub-categories of semantic accuracy and semantic cohesion.
For German into Spanish, e.g., the quotients were 0.009 and 0.002 respectively (39 and 10 in half an hour). This means that there were 0.9, or almost one, critical cases of incorrect content and/or 0.22 of incorrect logical links for every 100 words or every 7 segments (one segment roughly corresponds to one sentence, or several short ones, and in the present case contains 14 words on average).
Too literal translations, omissions and additions were other sub-categories where problems occured.
To quote one interesting example that caused problems in both English and French (but slightly less so in Spanish) is the following:
German: Also, ist das ein gutartiger Hirntumor? Ist es eine Vernarbung, eine subtile Aufbaustörung, die sogenannte fokale kortikale Dysplasie?
English translation: So, ist ist a benign brain tumor? Is it scarred tissue, a subtle disorder of cortical formation known as focal cortical dysplasia?
English: So is the malignant brain tumor a finnabour [artefact of the German “Vernarbung” – artefact], a subtle structure disorder, the so-called focal-cortical dysplasia, …
French: Alors, la tumeur cérébrale maligne, est-elle un finabeau [artefact of the German “Vernarbung”], un trouble subtil de la structure, la dysplasie corticale focale ?
It is hard to explain why the benign tumor is turned into a malignant one in English and French (but not in Spanish), as this is by no means an ambiguous expression. Maybe it has to to with the statistical probability of the combination malign+tumor?
Another interesting phenomenon was this one:
German source text: Und welche Rollen spielen Hilfsmittel wie Wearables?
English human translation of source: And which role do aids like wearables play?
English machine target: And what role do aids play? It is important to know that wearables are not.
French machine target: Et quel rôle joue le SIDA ?
Spanish machine target: ¿Y qué papel juegan el SIDA? Cómo juegan los dispositivos portátiles?
It is almost impossible to understand why the German Hilfsmittel (English aids) turn into SIDA, i.e. AIDS, the disease, in French and Spanish, without looking at the English version. Obviously, the English aids were “misunderstood” as the disease AIDS, which then became SIDA in French and Spanish. So apparently, the AI does not interpret directly from German into French or Spanish, but uses an English “relay”.
Correlations
To get an idea of how different kinds of deviations correlate, or one might also be the cause of the other, it can be interesting to filter for one specific deviation category and see which other kinds of deviations have ocurred in the same segments. In the example below, segments containing prosody issues were filtered out. It can be seen that many of them correlate with other, content-related issues.
One interesting example that shows how one deviation can cause a cascade of other issues is this one:
German: Und welche Rollen spielen Hilfsmittel wie Wearables? Wichtig zu wissen ist, dass diese Fortbildung keine individuelle ärztliche Beratung ersetzt.
English translation: And which role do aids like wearables play? It is important to know that this seminar does not replace indivdual medical advice.
English machine interpreting: And what role do aids play? It is important to know that wearables are not.
Apart from the fact that the complete disclaimer is omitted, what’s interesting is how bad sentence segmentation distorts the meaning.
Where it worked well
But of course, we didn’t want to get lost in AI bashing. While on the one hand, my students and I felt the responsibility of showing where the critical issues and risks are, we do recognise that machine interpreting can be an interesting option for those who do not have the means to hire professional interpreters.
So on the positive side, we also looked at those instances where machine interpreting worked well or at least was acceptable. Those were often short sentences with simple structures.
| Hier können auch Videos sehr helfen. | Videos can be very helpful here. |
| Der nächste Baustein in der Diagnostik, das ist die genetische Diagnostik. | The next building block in diagnostics is genetic diagnostics. |
| Wichtig ist die Anfalls-Beschreibung. Sie hilft uns, diese Anfälle richtig zu klassifizieren. | The seizure description is important. It helps us to classify these seizures correctly. |
| Eine psychologische Beratung und Mitbehandlung von Anfang an ist ein wichtiger Punkt. | Psychological counseling and co-treatment from the beginning is an important point, |
| Und zu beachten ist, dass es kritische Zeitfenster gibt. | and it should be noted that there are critical time windows. |
| Damit bedanke ich mich sehr für Ihre Aufmerksamkeit. | Thank you very much for your attention. |
| und wenn ich jetzt sage, jemand hat eine Epilepsie, dann habe ich noch nicht so viel gesagt. Für Menschen, die sich die ganze Zeit damit beschäftigt, damit beschäftigen, natürlich. | And if I now say someone has an epilepsy, then I haven’t really said so much yet for people who deal with it all the time, of course. |
| Medikamente sind eigentlich die Basistherapie. Das bekommen alle, wird auch von allen Fachgesellschaften so empfohlen. | Medications are actually the basic therapy that everyone gets and this is also recommended by all professional societies |
| Interessant ist, dass es mittlerweile 27 verschiedene Medikamente gibt. | Interesting is that there are now 27 different drugs. |
| Wie sucht man die aus? Das hängt davon ab, wie alt der Mensch ist mit Epilepsie, welches Geschlecht hat, in welcher Lebenssituation sich jemand befindet, möchte vielleicht eine junge Frau oder eine Frau Kinder bekommen und so weiter. | And how do you choose them? That depends on how old the person is with epilepsy, what gender they have, in what life situation someone is in. Maybe a young woman or a woman wants to have children and so on. |
Conclusion
The idea of this endeavour was to provide our client with enough insights to enable them to judge whether a machine-based interpretation is fit for their purpose or not. We tried to find the right balance between pinpointing the critical bits (like a malignant tumour being turned into a benign tumor) and explaining in which instances machine-interpreting worked well (usually short and simple sentences). After all, as professionals we have a responsibility to raise awareness of the risks whilst at the same time illustrating in how far machine-generated interpretation can do a reasonable job.
It was a bit to my surprise that in the final round of questions, the client asked “Well, that was very insightful. But could you please tell us now: Would it be a good idea to use machine-interpreting for our purpose or not?” Obviously, providing hard facts and detailed analysis still have to go hand in hand with human expert judgement to be of use.
About the author:
Anja Rütten has specialised in tech, information and terminology management since the mid-1990s. She holds a professorship in interpreting studies and Computer-Aided Interpreting at the Cologne University of Applied Sciences.
Disclaimer:
Views or opinions expressed are solely my own and do not express the views or opinions of my employer.

Leave a Reply