A Chatbot Experiment

 

In the previous post we lauded some of the wonders of which the new generation of chatbots is capable. In this post it we will look at some of the drawbacks.

 

To be fair, It must be said that the home screen of ChatGPT displays a whole list of caveats: not up to date by several years, may occasionally produce biased content, may give information that is plain wrong, etc. The internet is buzzing with examples. Rather than make an invidious selection, I decided to conduct my own experiment. Here’s a foretaste.

 

There’s is a long-established Irish-Spanish family in the city of Valencia, Spain, where I Iive, which played a prominent role in its modernisation in the nineteenth and early twentieth centuries. The family name is Trenor. I’d heard of them but wanted to know more. To my consternation, the answer came back from ChatGPT that it had no information about the family. The answer is all the harder to understand because there’s a long article about the Trenors in Wikipedia! True ChatGPT adds, “I cannot access up-to-date information on specific families or individuals without violating their privacy;” but I didn’t specify “up to date.”

 

Undaunted, I decided to try a second experiment, this time on a topic that should be more familiar to regular readers of this blog and so make it easier for them to judge the result.. The question was, “Who first proposed the natural translation hypothesis.” This time I did get a meaningful response – only it was a wrong one. It was that the inventor was Eugene Nida. Nida was an immensely influential translatologist; there was a time, when I was beginning in translation studies, that no thesis was complete  without a reference to him. And it’s true that Nida recommended the use of natural-sounding target language. However, he did not formulate a natural translation hypothesis and he used natural in quite a different sense. Why then did ChatGPT make this mistake? We can only speculate, because chatbots are black boxes whose workings are not revealed by their designers if they even understand them themselves. It could not be for lack of data, since a cursory search with Google finds lots of references to the hypothesis. So a more likely explanation is failure to apply the longest match principle. The principle states that when seeking to match strings of characters or words, only the longest match is acceptable. Could it be that ChatGBT mistakenly stopped seeking after translation and missed hypothesis?

 

The next thing that happened was my mistake. I pressed a wrong keyboard key and as a result the previous question was repeated. Again I soon got an answer – but it was not the same one! It was this:

“The Natural Translation Hypothesis was proposed by Gideon Toury, an Israeli scholar and translation theorist, in his influential book "Descriptive Translation Studies and Beyond" published in 1995. Toury's Natural Translation Hypothesis suggests that translations are influenced by various factors such as the translator's language competence, the target culture, and the translation norms of the time, which shape the translation process and result in translations that are shaped by the target language and culture, rather than being an exact replica of the source text. Toury's work has had a significant impact on the field of translation studies, and the Natural Translation Hypothesis is considered a fundamental concept in translation theory.”

This is much better. In fact Toury was very sympathetic to the hypothesis, as I know from personal correspondence with him. However, he had his own model of how the translation ability develops and he had his own term for it, which was native translation. And he never called it a hypothesis.

Notwithstanding the disappointments, I decided to have one last try. By this time there were already imitators and competitors of ChatbotGPT on the market, so I turned to one of them, BingAI. Here’s its equally fast response:

"Brian Harris is credited with proposing the Natural Translation Hypothesis. According to Harris, natural translation is a way of translating that occurs when a person translates without any formal training or knowledge of translation theory. The hypothesis suggests that natural translation is a more effective way of translating than traditional methods. Harris' work has been influential in the field of translation studies and has led to further research into natural translation."

This is a correct answer to the query, though the hypothesis doesn't suggest that natural translation is “more effective.” It seems that BingAI is capable of going beyond its data and making things up into what are now called hallucinations.

A single example isn’t enough to declare one chatbot superior to another. However, if there’s a lesson we can learn from it it's not to rely on only one chatbot. If there is the slightest doubt, try another.

 

 

Sources

ChatGPT. https://chat.openai.com

BingAI. https://bing.com/chat.