Marking Translations Positively

 From time to time we delve down into the archive of this blog in order to revive a post that deserves not to be forgotten amidst the mass of hundreds. Here is one such post.

FRIDAY, AUGUST 3, 2012

Marking Positively: How to Score Natural Translations

 

This post is addressed particularly to researchers, but it's relevant too for teachers of translation. Note that Natural Translation (NT) is used here as a cover term for both Natural Translation and Native Translation.

At the Forli conference in May (enter forli in the Search box), I noticed that some people are still using the old subtractive scoring method to rate NT.

What is the subtractive method? It means starting from 100 points and knocking off a point, or several points, for each mistake of any kind; typically a point or two for minor errors of content or expression and up to five points for major ones. The 'pass mark' is usually expressed as a positive percentage, but it's really a 'failure score'. That's how students' written translations are marked, and likewise the examinations of the professional associations like the Canadian one to which I belong. It can also be used for interpretations, especially if they're transcribed.

Two objections can be raised. The first is a didactic one: that the approach is negative and therefore discouraging. True, mathematically speaking, -30% of mistakes is equivalent to +70% correct, but the psychological effect is different. Anyway, it's not so important as the second objection, which is that the approach reinforces 'nit-picking' by the markers, because small details are allowed to affect the score significantly. I still squirm at a sequence in an old film about an interpretation exercise for European Commission interpreters (see References) in which a student is berated in front of the other students for his translation of a single word.

When evaluating NT, we need to take the opposite approach. Although mistakes are of great interest insofar as they reveal the limitations and the 'pathology' of NT, in NT research our primary interest should be in what subjects can translate and not in what they can't. A score of only 40% because of numerous distortions and omissions would probably entail failure for an Expert or Professional translator or a translation school student; but for a Natural Translator it represents a non-negligible translating ability and we should focus on it and analyse what that 40% consists of.

How can we build a positive scoring method?

In the 1990s I became involved in the design of tests for candidates who wanted to work as community interpreters for public services in Ontario, Canada. These became known as the CILISAT tests and are still in use. The Government of Ontario funded the necessary research. The candidates were almost always Native Interpreters, because the pay was too low to attract Professional Experts and because the languages were not taught in Canada. We decided we needed a test instrument that would be better suited to Native, i.e. untrained, Interpreters than those used by the translation schools and in the profession. So we turned to a method called propositional analysis. It's used by psychologists among others, and in fact I'd been introduced to it by the late David Gerver, who was one of the pioneer researchers on interpreters and was also a clinical psychologist. The form of it we used it can be described this way:

"To analyze the text, propositional analysis – a description of the text in terms of its semantic content – is used. The units of analysis are propositions, or units of meaning containing one verbal element plus one or more nouns. The corresponding units are then selected on the basis of meaning rather than structure."

In practice this meant that we broke down the scripts for the interpretation tests into simple, single-clause sentences representing propositions and then awarded points according to whether the meaning of each proposition as a whole was conveyed in translation: zero points for an omission or a meaning contrary to that of the proposition; 1 point for a meaning conveyed but not clearly or not completely; 2 points for a complete and true rendering. There was a weighting that distinguished between important and unimportant propositions. This scale was solely for meaning. Other factors, for example correct language, were scored separately and globally, not proposition by proposition.

For example, the statement, "At around 6 o'clock I saw a blue sports car waiting on the other side of the road," might be broken down into:

The time was approximately 6 pm

I saw a car.

The car was blue.

The car was a sports car.

The car was waiting.

The car was on the other side of the road.

A paraphrase like, "I seed a sport car stopping at the kerb of our street before supper" would score 7 points for informational meaning before being weighted for importance. (Work it out! 1+2+0+2+1+1.)  The maximum possible points varied with each script. Small language mistakes like "seed" were relegated to a separate evaluation.

References
Guadalupe Barrera Valdes and Manuel Rosalinda Cardenas. Constructing matching tests in two languages: the application of propositional analysis. NABE: The Journal for the National Association for Bilingual Education, vol. 9 no. 1, pp. 3-19. 1984. There’s an abstract here.

Roda P. Roberts. Interpreter assessment tools for different settings. In R. P. Roberts et al. (eds.), The Critical Link 2: Interpreters in the Community, Amsterdam, Benjamins, 1999. Most of it is here.

David Gerver. A psychological approach to simultaneous interpretation'. Meta, vol. 20, no. 2, pp. 119-128, 1975. "A slightly altered version of a paper presented at the 18th International Congress of Applied Psychology in Montreal in July 1974". The text is here.

André Delvaux (director). Les Interprètes. Brussels: Commission of the European Communities. c1975. 16 mm film. c15 mins.

Comments

The post drew comments. Here are a couple of them.

To those of you who have commented on the post about positive marking...

I ought to have acknowledged that even before I heard about propositional analysis from David Gerver, I'd learnt about positive marking from Daniel Gouadec, a well-known French translation teacher who came to teach for a couple of years at the University of Ottawa in the late seventies (see References). He was working at the time on a marking system for the Canadian government Translation Bureau's quality assessment section, but I don't know whether they ever used it.

In reply to SEO Translator: the deductive method is usually applied to short texts, say 300-500 words. For purposes of comparison, texts of about the same length as one another are used; and also, obviously, of the same level of difficulty. The 'pass mark' varies according to the expectations of the markers or examiners, taking account of the purpose of the exercise (professional examination, translation school assignment, etc.), the institution, the difficulty of the text, the level of the examinees, and so on. I've seen pass marks of 60% to 90%. Logically, tests for Expert Translators should have a high pass mark.

In the CILISAT tests, using positive scoring, we actually had two pass marks: one for 'ready to work' and a lower one for 'shows promise but needs training'. As I recall, they were 80 and 60 respectively, but that was after combining with the separate assessment for quality of target language. I haven't thought about automating these or other scorings. Possibly.

Theodor Herzl's Palestine Interpreter and British Translator

 

“The past is never dead. It’s not even past.” ─ William Faulkner

Few people are aware of how far the current tragic conflict in the area once called Palestine or the Holy Land stretches back. Its seeds were sown long before the state of Israel even existed, in fact to the late nineteenth century when it was part of the Ottoman Turkish empire.

Theodor Herzl, the Austro-Hungarian writer who was the father of modern Zionism, arrived from Vienna by rail and sea at Jaffa, on the Palestine coast,  on October 8, 1898. He had come somewhat reluctantly, because the choice of Palestine as a new homeland for Jews was not his but that of the delegates at the meeting of the first Zionist Congress. However, the German Kaiser Wilhelm II was scheduled to pay a show pilgrimage visit to the country and Herzl didn’t want to miss an opportunity of meeting him on favourable ground and pressing his case. His plan was to proceed by train on the railway to Jerusalem that had been constructed recently by a French company.

Meanwhile he went on a side trip with his four travelling companions to the prospering settlement of Rishon LeZion.  Today it’s the fourth largest city in Israel, eight kilometres south of Tel Aviv and hence not far from Jaffa, but in 1898 it was an agricultural settlement with a population of about 350. It had been founded in 1882 by Jewish immigrants from Ukraine. By 1889,under the guidance of agronomists sent by Baron Edmond James de Rothschild, it had become “a pretty little hamlet surrounded by vineyards and orange groves.” Its citizens were encouraged to speak Hebrew. In 1886 the Haviv elementary school was established as the first modern school to teach exclusively in Hebrew. So Herzl went there and prepared a speech. And then he struck a problem. His speech was written in German  and he couldn’t speak any Hebrew. Once again interpreting came to the rescue: “He delivered his speech in German and one of the villagers stood next to him and translated as he spoke.” We know nothing more about that interpreter and therefore cannot say that he (or she?) was a natural translator, but it’s certainly a case of an NPIT interpreter.

 Sources

Amy Dockser Marcus. Jerusalem 1913. New York: Penguin Viking, 2007. 225 p. Available from the usual booksellers. This is a ‘must read’ for anyone interested in the origins of the Arab-Israeli conflict.

Theodor Herzl, LL. D.. A Jewish State: An Attempt at a Modern Solution of the Jewish Question, translation of Der Judenstaat, Leipzig and Vienna, 1896, by Sylvie d’Avigdor, London: David Nutt, also 1896. Modern re-editions available from booksellers and on Kindle. A copy of the 1896 edition was recently on offer at $5,000. There are other translations.

Sylvie d'Avigdor, born 1872 at Nice, was the scion of a distinguished French and British Jewish family and was an NPIT Native Translator who, according to the British Library catalogue, didn't publish any other translations. Where did she learn such good German and how did she read the book so soon? David Nutt was a prominent publishing house but also a bookseller of foreign books, so maybe she obtained she book from them. Nutt himself had died years earlier. Unlike Herzl, she lived to see the Jewish state established. Notice she changed the opening German definite article Der to the more tentative English indefinite A. Noteworthy too is that the translator appears prominently on the title page, something exceptional in her time and unusual even today; perhaps her family’s prestige had something to do with it.

Rishon LeZion. Wikipedia, 2021.

Child Language Brokering in the Pandemic

 

 A good deal has been written about child language brokering, some of it on this blog. (Enter brokering in the Search box on the right.) But now the covid pandemic adds another page to the story. So an excellent article that has just appeared on a Canadian website is particularly timely. It’s too long to reproduce in extenso, but here’s an extract. There are interesting sidelights on the broker's progress from age five and use of the dictionary. I urge you to read the full article (see below).

"It was Mimi Nguyen’s older sister Kim who first modelled the expectation that the children of newcomers should step in as the family’s translators.

"Their parents settled in East Vancouver, by a stretch of Kingsway that many other Vietnamese refugees have called home since the late-1970s. Growing up, Kim helped these families too, translating teacher’s notices so that parents could keep up with their kids’ school progress.

"Because Mimi was educated in Canada, it was common for her generational peers to translate for their families as soon as they learned English.

"Kim once helped their mother with an English question at just five years old, with the help of a cousin on the phone and a Vietnamese-English dictionary in front of her. The well-thumbed volume is still in the family’s possession.

"It wasn’t until Mimi entered her preteens that she took over helping her parents, translating at in-person appointments and interpreting documents like bank slips.

“'Sometimes it would take the whole community to translate bits and pieces of a document, calling one person and another to verify words,' she said. 'Nobody in our network was fluent enough to translate everything confidently, so oftentimes, people felt like they were shooting in the dark.'

"Language barriers are an age-old problem for immigrants and refugees, affecting everything from housing to health care, education to employment.

"But the pandemic has meant there’s more to translate than ever — and there have been dire consequences for those who can’t read the vital information.

"Even for people who do speak English in B.C., it’s hard to keep up with official sources and sort out the bad ones. But the 'infodemic' weighs more heavily on families like Nguyen’s, who don’t get translations of government information as quickly or completely compared to official languages, if at all.

"Nguyen is now 25, and with her sister living out of town, she is her parents’ primary translator. Because they do essential in-person work — her father at a warehouse, her mother at a food packaging facility — their understanding of their rights and public health messaging is vital to their safety.

"According to Statistics Canada, newcomers are over-represented in high-risk jobs on the frontlines, and the hardest-hit industries like food and hospitality… It’s a privilege to be able to access information about the pandemic, and Nguyen worries about those who don’t have the language, time or know-how. 'Every single day, those inequalities are heightened even further,' she said."

 

Source

Christopher Cheung.  The Translator Kids. The Tycee, 30 April 2021.

 Image

Mimi Nguyen. Photo by Christopher Cheung.