https://www.duolingo.com/Luis

Testing a new translation format. Thoughts?

Based on your feedback, we just started testing a new way to do real-world translations in which the grading is done entirely by other users. The exact changes are as follows:

  • Instead of comparing the percentage agreement with others, we will have each sentence graded by at least three fellow users. Because there are many ways of correctly translating each sentence, we think this will give more accurate feedback.

  • There will still be a very lenient filter to make sure your translation is not gibberish, but we will no longer take points away because you don't agree with other people. As long as your translation passes our initial filter, you will be able to move on in your learning while you wait for others to grade your sentence.

  • Once enough users have graded your translation, you will get an email with a grade report, showing what others thought about your translation, along with the current best translation of the sentence.

  • Because we need more human ratings per sentence to make this happen, some of the real-world exercises in the skill page path will only require you to rate (instead of both rating and translating).

  • To encourage truthful rating (and also make it more fun!), you will get points only when your rating agrees with the ratings of others. If you're the first person to rate a sentence, you will get points no matter what.

This change is being tested on 50% of our users. Let us know what you think!

November 1, 2012

30 Comments


https://www.duolingo.com/Luis

I should mention that the system combines multiple grades into a single email, so you won't get an email for every translation you do.

November 1, 2012

https://www.duolingo.com/anomalocaris

I like the grade report idea and think it'll be really helpful, but I don't really want to get lots of emails...it would be great if you guys implemented a private messaging system, but if that's really not in the pipeline, maybe send the grade reports as a daily digest?

November 1, 2012

https://www.duolingo.com/Romatik

Well, in general I agree with this system, but some of the points are not valid, in my opinion: "To encourage truthful rating (and also make it more fun!), you will get points only when your rating agrees with the ratings of others. If you're the first person to rate a sentence, you will get points no matter what."

This means that now I will have to guess the rating that this sentence received before me. For example, what if there is only one rating (by one person) and this person think that this sentence is perfect (green rating). I come along and I disagree with this and put in yellow rating. I might have my reasons to think that my rating is more accurate and in fact, I may be right, but I will be penalized since I won't get any points for my rating since: "Hmm, looks like you disagree with others". Notice that this is plural, not singular (otherS), so I think, that before you penalize someone for disagreeing, you should wait for 3-4-5 ratings (and give points for each of those ratings) before you penalize someone for disagreeing. In general, I really don't think that this system is "fun" right now.

"There will still be a very lenient filter to make sure your translation is not gibberish, but we will no longer take points away because you don't agree with other people. As long as your translation passes our initial filter, you will be able to move on in your learning while you wait for others to grade your sentence."

This may be my personal preference, but I like to translate up until I get 100 points each day. In other words, I would translate only that many sentences. With this system I will have to wait to get my score which is a little bit disheartening (at least to me). It takes the sense of achievement that I would get once I hit 100 points.

November 1, 2012

https://www.duolingo.com/bf2010
  • 1799

I find one aspect especially helpful (compared to the initial changes proposed in the blog) Luis writes "As long as your translation passes our initial filter, you will be able to move on in your learning", -> good point; what I would appreciate is to have an example how "the initial filter" works...
Still, considering that all translation exercises have got the most flak in Duolingo; Duolingo really tries to include as many user suggestions as possible - so lets find out if these changes work for most of us... and thanks to the Duolingo team for making this possible..

November 1, 2012

https://www.duolingo.com/SamFen

I like this in general (fairly new user here, so I've only seen minimal changes to what I was used to), but I would like to just see a summary of the recent translations that were graded when I come back to Duolingo the next day, instead of having to read emails about it.

I also like Romatik's point above a lot. Currently, the first rater gets points, and the second rater only gets points if he agrees with the first, whether or not later users eventually decide that rater #2 was actually correct. Why not wait until n ratings are collected, then assign points to everyone at once? If the problem is motivation for users who won't see their points right away (and so may not bother with ratings), award everyone a point for participating, then award the proper points later.

November 2, 2012

https://www.duolingo.com/kelvinma13

@Luis

Then that takes away the instant gratification, and with it, the motivation to translate. Plus, most of the full articles available(That haven't been 90% taken) are the obscure ones, where it's unlikely that enough people will come after you to rate your sentence.

November 3, 2012

https://www.duolingo.com/Kbenco

I agree with rocko. At this point, I find the translations are often quite difficult - they include many words and tenses I have not covered in the lessons and idiomatic expressions with which I am not familiar. I am often pretty sure my translation is rubbish, and don't need a grade report to tell me this, so I turned it off. I feel I learn more from looking at the "best answer" and even with my poor French, I can tell if the translation makes sense in English- which is often not the case.

November 4, 2012

https://www.duolingo.com/Kbenco

This is a great idea. I am finding that the use of Australian English does not always agree with the American English "best" translation - maybe it might be possible to show the country of origin of the raters at some point. Eg:Just because doctors can be called physicians in the USA doesn't make doctor a wrong word!

November 1, 2012

https://www.duolingo.com/pjreads

I agree with Romantik that points should not be lost for disagreeing with only the first person who rated a translation. I also agree that it's disheartening to only get one point for translating a sentence. Translating is more difficult than rating and should be awarded more points. Award a base level of points for a translation that passes the filter (3 points for easy sentences, 5 points for average sentences, 7 points for hard sentences), then award bonus points for high scores from raters.

November 3, 2012

https://www.duolingo.com/Luis

@pjreads: we do award a lot more points for translating than rating, but you just don't get them immediately.

November 3, 2012

https://www.duolingo.com/monzac

@Romatik I agree with you about problems arising from not conforming to earlier rating/s of translation. Also, I'm not sure how this requirement to conform keeps us honest.

November 4, 2012

https://www.duolingo.com/flinda

Sounds excellent .

November 1, 2012

https://www.duolingo.com/arc4oiris

My most recent translation experience is to rate or edit sentences.

What is my purpose? You have two different objectives going here. Is it part of finalizing a sentence for readability or rating the sentence provided; is it to help the initial translator of the sentence with typical English usage or to finalize the sentence?

Will they be provided with the suggested edits along with the ratings? It is confusing to rate them and then provide edits.

I suggest a separate system. Either have us rate a sentence or have us provide our best edits - one or the other.

November 1, 2012

https://www.duolingo.com/Luis

@arc4oiris: You don't have to do both.

November 1, 2012

https://www.duolingo.com/arc4oiris

thank you - I now see that tucked in one of your bulleted items.

November 1, 2012

https://www.duolingo.com/rweba

I like the new system, it feels more logical than getting a percentage which is often misleading. I also like getting the report card since I can see which of my were translations were good.

My only suggestion is to provide even more tools for users to learn from their previous translations as they could presumably learn a lot from that..

November 2, 2012

https://www.duolingo.com/flinda

I agree with Samfen

November 4, 2012

https://www.duolingo.com/Romatik

@Luis May be it's just a bug, but i don't really get the points even if i have passed the "3 perfect ratings" filter. Also, I agree completely with pjreads that you should award some points for passing the filter (computerized) and you get bonus points for passing "human filter". For example, I have translated 10 sentences one day but I only passed "human filter" once with the sentence which was literally: "No!". I know, that my translations are not perfect, but I don't know French enough to translate every single sentence perfectly. That is why I learn it. And learning by definition is to make mistakes. With this system you punish learners pretty severely if translation is not perfect.

November 4, 2012

https://www.duolingo.com/grey_turtle

I do like the new system although I notice I get fewer points this way. However, one problem I've noticed is that on some categories, it won't let me continue. It just has me translate or rate sentence after sentence. I know I'm not great at Spanish, but some of the sentences are simple and I'm pretty sure they should be accurate enough to pass the filter (I even checked a few with Google translate and one with a Spanish speaking friend so it seems to legitimately be stuck in a loop of some sort). I can't go on in those categories. When I give up and go back to the menu it has the translation I just worked on (translating sentence after sentence) as not complete and when I try to do it again, it gets stuck in the same neverending loop of translation. Anyone else notice this problem?

November 4, 2012

https://www.duolingo.com/Luis

@grey_turtle: Can you send us some screenshots of what you're seeing?

November 4, 2012

https://www.duolingo.com/grey_turtle

@Luis. For example, right now it's stuck on Skill: Arts, Real World translation 5. I'm not sure how to send you screen shots, but whenever I translate a sentence, it says "Thank you for your contribution" or it if I rate a sentence, it says "nice" or whatever and then it just keeps giving me more to do. I never get the "finish" challenge button.

November 4, 2012

https://www.duolingo.com/Luis

@grey_turtle: Thanks we'll look into this.

November 4, 2012

https://www.duolingo.com/grey_turtle

@ Luis Just to let you know, I'm now also stuck in V: Inf 1, Abs. obj 1 and Sports with the same problem. Just wants to me to keep translating instead of ever giving me the finish challenge button. I've gotten extra practice though :).

November 4, 2012

https://www.duolingo.com/DShanky

I have had the same issue as grey_turtle - I have done a few translations / gradings but not had the 'Finish Challenge' button. I have had to restart the same lesson a few times and eventually had an option to finish the challenge.

I like the idea of the new marking system. However in my first report card I received 2 'Perfect' ratings and 1 'Some Mistakes' rating for a translation which meant that my translation failed with "not enough good ratings" - this seems a bit harsh!

November 4, 2012

https://www.duolingo.com/grey_turtle

Now also stuck in V: future and Politics. Sometimes I can restart and finally get the finish challenge buttons, but sometimes I get the same problem over and over even after restarting :(

November 4, 2012

https://www.duolingo.com/arc4oiris

So, has the number of points that one can receive, changed for each acceptable sentence, regardless of the difficulty as used to be the case?

November 4, 2012

https://www.duolingo.com/Luis

The number of points has not changed.

November 4, 2012

https://www.duolingo.com/rocko2012

The email grade report is not a useful feature IMO. I think when a person does a translation they have a very good idea how good it is. No one is going to accidentally translate something right. And if it sounds off it probably is off. Duolingo really needs to address what a user is suppose to do when they do not think their translation is very good. Do you simply give up and do not submit it? Oh I see now you can turn off that email report.

November 4, 2012

https://www.duolingo.com/arc4oiris

The new system is frustrating. You can sign on to an article that had not been added more than an hour or two before, and there is nothing to translate. The sentences are either all at 100% or there are only ratings of others available. I get more value translating an entire article; doing one sentence per article or rating others does not add to one's expertise. Why not compare translations to your dictionary or machine translation to determine if the sentence "passes". You can still have people rate sentences AFTER they have submitted their translations to see if there is a smoother way of translating a sentence.

November 6, 2012

https://www.duolingo.com/maurac73

POINTS FOR HARDER SENTENCES? People translate all of the easy sentences first, and leave the difficult ones for others. We need extra points for translating the last sentences left that no one wants to translate. These are typically strewn throughout the article and are very difficult indeed. I'm learning quite a lot by tackling these juggernauts. It takes a lot of time and I only get a point or two for the extra effort. No one wants to translate the references either. Maura

January 1, 2016
Learn a language in just 5 minutes a day. For free.