Comparing MT engines, which MT should I choose?
Autor de la hebra: Philippe Locquet

Philippe Locquet  Identity Verified
Portugal
Local time: 17:25
Miembro 2013
inglés al francés
+ ...
Aug 10, 2020

I have been browsing and I could not find a comparison for MT output. So, here’s a thread about it.
When a translator or LSP considers Machine Translation, it is likely that, among other things, they goal is to reduce their effort and speed up a little their workflow (maintaining quality through human language skills off-course).
Since there are now many, many options, the big question becomes:
Which MT should I choose?
This seems like a good question, but which criter
... See more
I have been browsing and I could not find a comparison for MT output. So, here’s a thread about it.
When a translator or LSP considers Machine Translation, it is likely that, among other things, they goal is to reduce their effort and speed up a little their workflow (maintaining quality through human language skills off-course).
Since there are now many, many options, the big question becomes:
Which MT should I choose?
This seems like a good question, but which criteria should be used to discriminate MTs?
Here I propose a method, feel free to test it and comment/add your results.

Method:
These are the criteria you need to look at. In my opinion, price should be last, because only when you have tested and compared you will then have an idea if using an MT will really help you and how much will help you. Then you will be able to work out the value of the service for your specific needs.
Here are the criterias:
A/Language pair (not all MT perform the same and not all language pairs are supported)
B/Type of documents (not all MT perform the same in legal, news, medical, marketing etc.)
C/Access method (how you want to use the MT: pretranslate a document outside a CAT, pretranslate an xliff, use it inside your CAT etc. Not all MTs can connect equally)
D/Price (and method of payment: yearly, monthly, mileage (per character consumed) etc.)

Procedure for testing:
1/When starting your comparison, you need to determine for which language pair you want to use the MT most. Then pick up a news article about a common topic and talking about very famous people. This is the easiest for most MTs. Create a table worksheet and start copy-pasting the outputs of all the MTs you are comparing. Review and then, select 3 or 4 that performed the best.
2/Now you need to decide what is the document type you will use the MT most for and in which domain (automotive, legal, medical etc.). Now you can make a new table comparing the 3 or 4 engines that came ahead in the previous test. Feed a type of text represents the type of doc + domain you want to use the MT for. Pick the 2 winners.
3/Using criterias C and D decide who is your winner. (Or another conclusion could be that in your language pair + domain, using MT will waste you time instead of helping.)

In the next posts, I will upload 2 comparisons made from the same source text. These will be in EN2FR and EN2CZ. EN2FR is a language pair where MT perform well since they have a lot of resources available for training and EN2CZ has far less resources.
Then moving from English as a source, results differ a lot. For example, in DE2FR usually DeepL performs better.
Feel free to upload your findings. If you want to compare with the same text, it’s from https://www.bbc.com/news/business-52504187
I will put it below so you can copy-paste. This text has different levels of difficulty in each paragraph so it’s a very interesting sample.
My bests

Source text:
Tesla boss Elon Musk wiped $14bn (£11bn) off the carmaker's value after tweeting its share price was too high.

It also knocked $3bn off Mr Musk's own stake in Tesla as investors promptly bailed out of the company.

"Tesla stock price too high imo," he said in one of several tweets that included a vow to sell his possessions.

In other tweets, he said his girlfriend was mad at him, while another simply read: "Rage, rage against the dying of the light of consciousness."
Collapse


Monica Carneiro
Morano El-Kholy
 

Philippe Locquet  Identity Verified
Portugal
Local time: 17:25
Miembro 2013
inglés al francés
+ ...
PERSONA QUE INICIÓ LA HEBRA
EN2FR Aug 10, 2020

Here goes the snapshot.
Difficult to have 10 columns in the same picture... I had to remove manually some returns so it could fit
9 engines tested here.



My overall winner IMO is "ModernMT"

Engines give different results some are better than others in different aspects.


 

Philippe Locquet  Identity Verified
Portugal
Local time: 17:25
Miembro 2013
inglés al francés
+ ...
PERSONA QUE INICIÓ LA HEBRA
EN2CZ Aug 10, 2020

So here this language pair is more taxing on MTs and as you see the 9 MT providers are a bit different since not all MT providers provide services for the same language pairs.



I don't speak Czech, but a colleague told me that "ModernMT" came on top here again.

Let me know what you think and who's your winner in your languape pair + field.

... See more
So here this language pair is more taxing on MTs and as you see the 9 MT providers are a bit different since not all MT providers provide services for the same language pairs.



I don't speak Czech, but a colleague told me that "ModernMT" came on top here again.

Let me know what you think and who's your winner in your languape pair + field.

My bests
Collapse


 

TranslateWithMe
Polonia
Local time: 18:25
inglés al polaco
+ ...
My opinion Aug 10, 2020

Hi,

I have used some MTs over the years and I would also choose MTModern as the best. A few months ago, it was offered for free due to lockdowns, so I used it at that time, and it is really good.

But, if you need an equally good MT for free, you can use Google Translate API. When you register in Google Cloud (Console) for the first time, you get 300 USD of credit to be used for their services, including their MT, for one year, which is usually enough for one year (I hav
... See more
Hi,

I have used some MTs over the years and I would also choose MTModern as the best. A few months ago, it was offered for free due to lockdowns, so I used it at that time, and it is really good.

But, if you need an equally good MT for free, you can use Google Translate API. When you register in Google Cloud (Console) for the first time, you get 300 USD of credit to be used for their services, including their MT, for one year, which is usually enough for one year (I haven't paid a penny for it).

https://cloud.google.com/free

The information how to get API for Google Translate can be found online.

Cheers
K.
Collapse


Philippe Locquet
 

Philippe Locquet  Identity Verified
Portugal
Local time: 17:25
Miembro 2013
inglés al francés
+ ...
PERSONA QUE INICIÓ LA HEBRA
Switching MTs ? Aug 12, 2020

TranslateWithMe wrote:

Hi,

I have used some MTs over the years and I would also choose MTModern as the best. A few months ago, it was offered for free due to lockdowns, so I used it at that time, and it is really good.

But, if you need an equally good MT for free, you can use Google Translate API. When you register in Google Cloud (Console) for the first time, you get 300 USD of credit to be used for their services, including their MT, for one year, which is usually enough for one year (I haven't paid a penny for it).

https://cloud.google.com/free

The information how to get API for Google Translate can be found online.

Cheers
K.


While thinking about this, it is a fact that because of language pair or translation domain, one MT is better than the other.

Fort such reason do you subscribe to two or more MTs and switch between them or is it not something to consider because of cost issues?

My bests 😊


 

Philippe Locquet  Identity Verified
Portugal
Local time: 17:25
Miembro 2013
inglés al francés
+ ...
PERSONA QUE INICIÓ LA HEBRA
Comparative MT study available! Aug 20, 2020

Hi all, a comparative study report has just been published by Intento.

https://try.inten.to/mt_report_2020?

They did a comparative test study between 15 MT engines across 14 language pairs. They made it specific too to 16 industry sectors.

If you are considering getting an MT engine, I strongly suggest you check it out! Once you spot which is best for you, make
... See more
Hi all, a comparative study report has just been published by Intento.

https://try.inten.to/mt_report_2020?

They did a comparative test study between 15 MT engines across 14 language pairs. They made it specific too to 16 industry sectors.

If you are considering getting an MT engine, I strongly suggest you check it out! Once you spot which is best for you, make a few tests of your own with that engine and compare it to two or three others. That will make for a very educated purchase!!!

They explain their methodology, scoring etc. (BERT, BLEURT…). This can be foreign concept to some translators, so I suggest you jump to these pages:

P 25 for ranking by industry sectors
P 29 for ranking by content types

There is also a call for volunteers rewarded with some free access.
I’ll open a different thread about this.

Hope this is useful!
Collapse


Morano El-Kholy
 

Samuel Murray  Identity Verified
Países Bajos
Local time: 18:25
Miembro 2006
inglés al afrikaans
+ ...
The report is a KEY file Aug 21, 2020

Philippe Locquet wrote:
https://try.inten.to/mt_report_2020?


FWIW, the report is in KEY format. This converter worked for me:
https://cloudconvert.com/key-to-pdf


Philippe Locquet
Mina Chen
 

TranslateWithMe
Polonia
Local time: 18:25
inglés al polaco
+ ...
I use one MT Aug 22, 2020

Philippe Locquet wrote:

TranslateWithMe wrote:

Hi,

I have used some MTs over the years and I would also choose MTModern as the best. A few months ago, it was offered for free due to lockdowns, so I used it at that time, and it is really good.

But, if you need an equally good MT for free, you can use Google Translate API. When you register in Google Cloud (Console) for the first time, you get 300 USD of credit to be used for their services, including their MT, for one year, which is usually enough for one year (I haven't paid a penny for it).

https://cloud.google.com/free

The information how to get API for Google Translate can be found online.

Cheers
K.


While thinking about this, it is a fact that because of language pair or translation domain, one MT is better than the other.

Fort such reason do you subscribe to two or more MTs and switch between them or is it not something to consider because of cost issues?

My bests 😊


From my experience, I use only Google Translate API now, as it is simply the best in both my language pairs: EN-PL, PL-EN. It does not matter if the text is very simple or technical. With technical texts, there is more "proofreading".


Philippe Locquet
 

Philippe Locquet  Identity Verified
Portugal
Local time: 17:25
Miembro 2013
inglés al francés
+ ...
PERSONA QUE INICIÓ LA HEBRA
Comparing and switching at translator-friendly rates Dec 2, 2020

Hi,
If you’ve been following this thread, you are probably using MT and you have probably found your MT of choice and decided to stick with it.
Like discussed earlier, things can change when we translate in a different language pair or in a different domain, some MTs are poor with IT content, some are excellent at legal etc.
I mentioned before Intento and I think it’s very useful for this. They have a platform that allows you to compare MTs and connectors to switch the MT y
... See more
Hi,
If you’ve been following this thread, you are probably using MT and you have probably found your MT of choice and decided to stick with it.
Like discussed earlier, things can change when we translate in a different language pair or in a different domain, some MTs are poor with IT content, some are excellent at legal etc.
I mentioned before Intento and I think it’s very useful for this. They have a platform that allows you to compare MTs and connectors to switch the MT you want directly in your CAT. They have sped pretranslation processes too.
So instead of having to stick to a single MT subscription, you can just use this and witch between over 30 MTs (depending on language pairs). Their cheapest plan starts at 8.99 and then whichever MT you’re using form there is charged by the mileage (characters you’ve used for each MT).
I can easily see this making things a lot simpler for many translators.

Since it’s a new concept, I made a video describing how it works, you can watch it here: https://youtu.be/5K6raLjZZ3A
Hope you’ll find this useful!
Collapse


 

Stepan Konev  Identity Verified
Federación Rusa
Local time: 20:25
inglés al ruso
Just a tool Dec 2, 2020

MT is just a tool that saves your time by typing words for you. Does it really matter what the analysis say about them. Choose one and enjoy. For En>Ru, DeepL works best.

If anybody speaks Russian here, you can assess the result:

Source: Circuit breakers shall be accessible from the front, by opening the relevant compartment door.
DeepL: Доступ к автоматическим выключателям должен обеспечиваться спереди
... See more
MT is just a tool that saves your time by typing words for you. Does it really matter what the analysis say about them. Choose one and enjoy. For En>Ru, DeepL works best.

If anybody speaks Russian here, you can assess the result:

Source: Circuit breakers shall be accessible from the front, by opening the relevant compartment door.
DeepL: Доступ к автоматическим выключателям должен обеспечиваться спереди путем открытия двери соответствующего отсека.
ModernMT: Выключатели должны быть доступны спереди, открыв соответствующую дверь отсека.

In this particular case, DeepL sounds perfect, humanly and professional. I don't know what to amend there.
ModernMT, on the contrary, needs to be deleted and translated from scratch.
*However, ModernMT may work better for other pairs, right.

[Edited at 2020-12-02 14:00 GMT]
Collapse


Tomás Cano Binder, BA, CT
Philippe Locquet
 

Philippe Locquet  Identity Verified
Portugal
Local time: 17:25
Miembro 2013
inglés al francés
+ ...
PERSONA QUE INICIÓ LA HEBRA
Domain vs language pair vs MT Dec 3, 2020

Stepan Konev wrote:
Choose one and enjoy.


Thanks Stepan! Indeed this approach of "find the one that fits you and stick to it" seems to be the preferred approach among translators.
However, LSPs use a different approach, and it begins to make sense that translators start too to make choices based on best performance/project (what requires less editing effort).
The reason is that MTs are constantly changing and the training data and algorithms change. So the output quality is constantly evolving. However translators do not feel this as much because MT connectors available in their CAT tools are often connecting to a version of the MT that is one or two versions behind.

In the examples I'll list below, some results are from Microsoft v3 that cannot be accessed in any CAT tool at the moment (AFAIK). The only way is through Intento. Results are from DeepL are from v2.

Thanks for your post, it motivated me to do some testing!
I don't speak Russian so I reversed the pair, I did Russian to English.

First I took your best output from ModernMT and translated it to English. Then I went and fished some KudoZ questions: one in legal and one in medical. I don't speak russian, but it seems to me that for your test and legal, DeepL comes on top. However, this seems to change in medical. So, to me, it makes sense to take 5 to 10 mins to compare outputs on a sample from a big document we are starting on to make sure we have the best MT connected.
Here goes the results (following post).


 

Philippe Locquet  Identity Verified
Portugal
Local time: 17:25
Miembro 2013
inglés al francés
+ ...
PERSONA QUE INICIÓ LA HEBRA
my test results Dec 3, 2020

Stepan Konev wrote:

If anybody speaks Russian here, you can assess the result:


Baidu The automatic switch must open the corresponding door from the front
DeepL v2 Circuit breakers shall be accessed from the front by opening the door of the corresponding compartment.
Google v3 The circuit breakers must be accessible from the front by opening the door of the respective compartment
Microsoft v3 Access to automatic switches should be provided from the front by opening the door of the corresponding compartment
ProMT Access to circuit breakers shall be provided from the front by opening the door of the corresponding compartment
Yandex Access to the circuit breakers must be provided from the front by opening the door of the corresponding compartment

https://www.proz.com/kudoz/english-to-russian/law-contracts/6898784-sustaining-audits.html
Периодические/плановые проверки качества
DeepL Periodic/scheduled quality inspections
ProMT Periodic/scheduled quality checks
Yandex Periodic / scheduled quality checks
Microsoft Periodic/planned quality checks

https://www.proz.com/kudoz/english-to-russian/medical-cardiology/6781087-lead-maximum-guidewire-insertion-force.html
максимальное усилие введения проводника в просвет электрода
DeepL maximum insertion force of the conductor into the electrode lumen
Google maximum effort of insertion of the conductor into the electrode lumen
Microsoft maximum force of inserting a conductor into the electrode's lumen
ModernMT maximum force of conductor insertion into the electrode lumen
ProMT maximum conductor insertion force into electrode lumen
Yandex maximum force of insertion of the conductor into the lumen of the electrode


 

Tomás Cano Binder, BA, CT  Identity Verified
España
Local time: 18:25
Miembro 2005
inglés al español
+ ...
My experience with two providers Dec 3, 2020

Even if this is only tangential to the conversation, I would like to briefly summarise my experience with two providers in my main language pairs (EN>ES and DE>ES), namely Deepl and ModernMT.

- Deepl:
On the translation results: while not an integral part of my daily work and only used in certain cases, I have had a Deepl account for a couple of years. Because it cannot be trained and I cannot even add glossaries in my language pairs, Deepl has this rare ability
... See more
Even if this is only tangential to the conversation, I would like to briefly summarise my experience with two providers in my main language pairs (EN>ES and DE>ES), namely Deepl and ModernMT.

- Deepl:
On the translation results: while not an integral part of my daily work and only used in certain cases, I have had a Deepl account for a couple of years. Because it cannot be trained and I cannot even add glossaries in my language pairs, Deepl has this rare ability to translate the same term in ten different ways in a relatively small translation, as well as to mix different registers, understand things the other way around, and all the consequences of not having any context.

To fight the inherent variability, you can resort to translating the whole project in one go, look for mistranslations/bad terms and spend some time straightening them, and the build an auxiliary TM for your job. In total, a helluvalot of rather unproductive time!

In terms of interaction with Deepl the company, despite my repeated suggestions and requests to fix endemic issues in spelling in Spanish, I have never seen any result or even had a single response back from Deepl. Despite being a paying client, I do not count in their customer relations efforts at all. Sending me their monthly invoice is all my communications with them.

- ModernMT:
ModernMT can be easily trained with your TMs (in memoQ, also creating different settings per client to easily enable the right "engine"), learns quite a bit as you translate interactively, and is fast.

Every time I have had a question or suggestion to the ModernMT team over the last weeks, they have replied right away, have shown interest, and have satisfied my curiosity about the technology to a certain extent, so kudos to ModernMT in this area.

After thinking that Deepl was just fair enough as a simple solution for my occasional internal MT projects, I find myself being awfully pleased with ModernMT. In terms of money, a ModernMT account costs me very much the same and is very affordable. I am beginning to feel ready to cancel my Deepl account!
Collapse


Philippe Locquet
 

Philippe Locquet  Identity Verified
Portugal
Local time: 17:25
Miembro 2013
inglés al francés
+ ...
PERSONA QUE INICIÓ LA HEBRA
My point too, you’re right! Dec 3, 2020

Tomás Cano Binder, BA, CT wrote:
In terms of money, a ModernMT account costs me very much the same and is very affordable. I am beginning to feel ready to cancel my Deepl account!


I understand how you feel, MTs change over time, so, in 3 months the MT that was best for a translator’s specific work could now be 2nd or 3rd place for the same needs.
That’s why I suggest looking into solutions like Intento: you pay 8.90 USD and just pay the number of characters you consume by MT provider. So it can end up costing the same or 10 USD more than 1 MT subscription and you can choose between 30 MTs on the fly.

I show how it works in this video: https://youtu.be/5K6raLjZZ3A

I confirm that ModernMT team is super nice and approachable. They’ve been very helpful when I made the video about their adaptive processes.

My bests


Tomás Cano Binder, BA, CT
 

Philippe Locquet  Identity Verified
Portugal
Local time: 17:25
Miembro 2013
inglés al francés
+ ...
PERSONA QUE INICIÓ LA HEBRA
MS Word action Dec 7, 2020

Hi all,

Playing with the tools we discussed here, I just discovered that Intento have an add-in for MS Word, so you can translate directly in Word and change your MTs from there too.

To take this one step further, I decided to see if it worked alongside Wordfast Classic, and it does!

So than can be a productivity boost for those using Classic or wanted to work directly in the Word file.

I made a video demonstrating this, you can watch it here i
... See more
Hi all,

Playing with the tools we discussed here, I just discovered that Intento have an add-in for MS Word, so you can translate directly in Word and change your MTs from there too.

To take this one step further, I decided to see if it worked alongside Wordfast Classic, and it does!

So than can be a productivity boost for those using Classic or wanted to work directly in the Word file.

I made a video demonstrating this, you can watch it here if you're curious:
https://youtu.be/IffZ4oJ5HDc

Have a lovely week translating and stay safe!
Collapse


 


To report site rules violations or get help, contact a site moderator:


You can also contact site staff by submitting a support request »

Comparing MT engines, which MT should I choose?

Advanced search






TM-Town
Manage your TMs and Terms ... and boost your translation business

Are you ready for something fresh in the industry? TM-Town is a unique new site for you -- the freelance translator -- to store, manage and share translation memories (TMs) and glossaries...and potentially meet new clients on the basis of your prior work.

More info »
Anycount & Translation Office 3000
Translation Office 3000

Translation Office 3000 is an advanced accounting tool for freelance translators and small agencies. TO3000 easily and seamlessly integrates with the business life of professional freelance translators.

More info »



Forums
  • All of ProZ.com
  • Búsqueda de términos
  • Trabajos
  • Foros
  • Multiple search