Fine-tuning Immersion Incentives
Has anyone else working in Immersion noticed that many more articles are posted than translated, and that many more translated articles are ignored than proofread/edited and finished? My experience has been with the Spanish to English, and over the last couple of months it seems like the number of unfinished articles has dramatically increased. And while it's good to have more options on what to translate, it's also hard to find any specific article, and it seems less likely that someone else will choose the article you've worked on to help edit or complete it.
I might have a few compulsive tendencies, but seeing so many untranslated articles, and even more translated but not proofread articles (and seeing them languish for days and weeks without advancing much if at all) is kind of disheartening. Sometimes I think to myself, "I'll be the one who proofreads these," but then the immensity of the task, and the seemingly exponential way it grows is too much and I give up.
Mostly, though, I'm interested in finding a better balance between untranslated, un-proofread, and finished articles because it seems to me that if we're going to translate the whole web, then we have to first be able to translate a whole article, instead of just 10 of 17 sentences. This is especially true if there are ever going to be any Spanish to English articles that someone pays to have translated.
Two ways I can think of to ensure more completion of translations would be to limit the number of articles that one user can post to be translated. Once they reach that limit, one of their articles will have to be finished before they could post another one. This would give the posters of articles incentive to help finish what they've started, or alternatively, limit people who are posting things that no one wants to translate from posting even more things that no one wants to translate.
The other way I can think of would be to tinker with the points/lingots/levels that can be obtained by posting, translating, and editing articles, so that finishing articles is as much a priority to the community as starting them.
To be very clear, and because it can't be said enough, Duolingo is a fantastic site and I'm very grateful to the people who work on it for giving me the chance to expand my horizons. This isn't meant as a complaint, but as an observation of a possible problem and a few possible solutions. I'm interested in learning what others (that means you!) think about it.
I worked as an editor on salary at a major university for six years. Our institute edited manuscripts of interviews for publication in the university archives. Once an interview series hit my desk, it had to be edited and proofread from start to finish. I quickly developed a written order of operations so that manuscripts didn't languish on my desk.
Our backlog of interviews was considerable. I reasoned (rationalized?) that one day technology would improve to such a degree that recorded interviews could be transcribed, even edited, by computer programs. In the meantime, I was paid to complete what the institute considered the most valuable series. The institute's motto: Accuracy over speed.
In the immersion area on Duolingo, I pick one article to edit. Right now I'm translating an article on Immanuel Kant. I may translate one or two sentences from articles I want to do next so that they'll be easily accessible in my profile. But I'll work on the Kant article exclusively until it's complete. I'm leading by example. That's the best I can do.
It would be helpful for Duolingo to write an FAQ to explain the editing and proofreading process. Perhaps they could limit the number of articles that someone can post or offer greater rewards to those who finish articles. In doing so, they would not only train linguists but also teach editors discipline. Sure, many people are here to learn a language for pleasure--translating is like a game. But editing and proofreading pays well: I earn $25 an hour freelance editing and proofing academic articles for colleagues of mine in the university. In fact, today I'm finishing up an article one of my colleagues is preparing to send out tomorrow morning. So there's an incentive to becoming a disciplined editor.
Still, I have questions about proofreading: According to whose rules of punctuation does one proof an article? Is British or American spelling preferred? Criticise or criticize? Favour or favor? Single quotation marks or double? Periods outside the final punctuation or inside? Those kinds of questions aren't going to be resolved in the comments section through "translation wars." In editing houses, those kinds of questions are answered by one or two proofreaders in the final edit.
Ultimately, my clients tell me whose punctuation governs the text. What do Duolingo's clients say? What does Duolingo say? Some guidance on those questions will help proofreaders to complete the copy efficiently and by the book.
Thanks for starting this thread! I'm wondering if one problem is that translators want upvotes - and if they work on an "older" article, nobody else will see their work and upvote them.
Personally, I've been trying to go back to some of the articles I first worked on months ago and proofread them all the way through. I get a "congratulations you finished the article!" notice, but then the article still doesn't disappear from the list. However, I consider myself finished with it at that point. I think it is up to the original uploader to remove it.
The version of DL I have rewards me with lingots for helping translate an article, whether it's finished or not. Perhaps the reward system could be jiggered to ONLY reward when a translation is finished?
I like the idea of limiting uploads to a specific number. Maybe linked to the translation tier? Or possibly having a once-a-month moratorium on uploading and focus on housekeeping instead? Interesting thoughts! - Ricaloca
I remember reading on another thread someone else's suggestion that older articles could be listed in a "Featured" tab. If Duolingo is interested in clearing out some of those old articles, those articles need to be easier to find! I'm guessing there are enough helpful/obsessive people who would want to do Duo a favor by cleaning house even without tinkering with the incentive structure.
As long as sentences are being uploaded faster than they're being translated & checked then there will be unfinished documents. That's just how the math works. So there are basically three options: get people to translate & check more, get them to upload less, and/or accept that some documents won't get finished and prioritize finishing some over others. There are a number of different options for doing each of those.
Things users can do:
- Translate and check more (obviously ;-).
- Upload less.
- Use the filters to the right of the Immersion document listing to help find the documents you think are most worthy of your attention or that you think you can be the most benefit to.
- Post links to documents you think are worthy of completion in the relevant language discussion areas and ask for help finishing them.
- Report documents that you don't think the user has the rights to upload.
Things Duolingo staff can do:
- Release Immersion for mobile. (They've said this is planned.)
- Get website users to translate & check more using your magical A/B testing sorcery. ;-) (I'm sure they're already working on this, but they have to make sure that an increase in quantity doesn't come with an undue decrease in quality.)
- Charge lingots for uploads (or at least uploads past a certain daily/weekly limit).
- Be more pro-active about removing documents the uploader doesn't have the rights to upload.
I suspect the document reporting/removal idea will be one of the more controversial of these options, but it would probably also be one of the more effective at increasing the percentage of documents completed, so I figured I'd throw it out there.
As for paid translations, the ones currently in Immersion for Spanish, Portuguese etc. speakers seem to be be being completed pretty quickly and consistently, most likely because people know that they're helping Duolingo by translating them, so I don't think there's really a problem with those at the moment.
Anyway, since we're on the topic, here are a few Spanish -> English Wikipedia article translations that could use some help being completed:
There was an appeal yesterday asking people to spend some time proof-reading and checking instead of just more translating. I was thinking last night that it might be helpful to have something like Clean-Up Sunday when people commit to spending even half an hour helping to finish translating marooned articles, or proofreading fully translated but not fully checked ones. But then, it probably doesn't really matter if these practice pieces aren't finished anyway.
Is anyone willing to create a sort of team/translation group? That way the whole group could work on the same article at once and get it translated, proofread, and checked in quite a fast amount of time (And as critically as possible: this wouldn't be a "give me exp, I want exp, hurr durr" thing :P )
I've pretty much done all I can on the La Lajita article stecchetto linked to, and I'm starting on the Pabellon Argentino article. I see you're working on the Malaysia Airlines article...Give me a holler if you find that it's languishing, unchecked.
I've gone through the La Lajita piece but it looks as though there are still a couple of sentences that need to be approved by a further person.
I saw that. I will finish the Malaysia Airlines one and go straight to check on La Lajita. Maybe we should make a new post just for this though
I'm willing to be part of the clean up crew! I did start working on one of the articles stecchetto linked to.
I'll give it a go. Have a lingot for volunteering to co-ordinate it :)
I didn't originally mean it as "the clean-up crew", but just making sure that we start and finish the same article, then move on to the next when that is done. We could start with theones that no one wants and actually clean up though =P
This thread is making me feel guilty because I am certainly one of those people who spends time translating but not proofing. I think a lot of people might be like me, wanting to practice but not feeling comfortable enough to revise what other people have done unless it is blatantly wrong. And sometimes I do upload things that I don't finish translating because parts of the article turns out to be over my skill level. I do try to upload things I think will engage other people, but sometimes I find an article that fascinates me apparently does not fascinate anyone else. Rewarding completion of an article is a good idea, but I don't think rewarding checking of an article will motivate me until I get more comfortable proofing.
Just from this post, it seems like your English is strong enough to do what I do. I look for translations where the English doesn't sound as smooth as it could and I tweak those. I find it pretty easy to pick out the sentences where the content is correct but the English sounds unnatural. And in the process, I get exposed to Spanish phrases and sentence structures that don't translate word for word. I admit, it helps that I have the option to share credit, but I don't know if that's been rolled out to everyone.
So how about, instead of the current system of rewards that gives no great incentive to help with proofreading, the rewards for proofreading were tokens that allowed one to submit a certain number of translated words or sentences. This would ensure that everyone who uses Immersion contributes equally to translating and checking and, by tuning the ratio of tokens to translation access, it should be possible to guarantee adequate proofreading of all completed articles.
Interesting idea. If I'm understanding you correctly, people would need to check/edit already translated sentences in order to earn the right to translate fresh, previously untranslated sentences? How do you maintain the quality of the votes/edits though? Perhaps some people might be so anxious to do the new sentences (to rack up the XP) that they would just randomly do some upvoting without really checking the translations. Or am I being too cynical?
No, I don't think you are being cynical. It seems obvious to me that there are people gaming immersion who are probably racking up the upvotes (and hence rising through the tiers) by swooping on newly uploaded documents and translating as many of the single words, short image captions or subheadings, or copying numbers, as possible but often not staying around to translate actual sentences.
You're correct, Luyematsu.
I'm off into the realms of fantasy here, but anyway:
Access tokens would be used to purchase the exclusive right to translate an article, or part of an article, for a certain period. If submitted with a specified minimum percentage translated within that time, it would then pass through three phases of proofreading:
1) Checking. During the checking period it would be available to users at the same or higher translation tier, to rate each sentence using a traffic light system - green = good translation, amber = needs work, red = needs re-translation. The Checking period would last until a certain minimum number of votes had been cast for each sentence, at which point the majority vote on each sentence would be accepted; points awarded to the translator accordingly (eg. 2 pts for a green, 1 for amber, 0 for red, for a certain number of words) and tokens awarded for each sentence only to those checkers who voted with the majority. At the end of the Checking period the amber and red sentences would be available for Correction. Articles not receiving the minimum votes within a certain time would be made available for purchase again and a proportion of tokens refunded to the translator.
2) Correction. The correction phase would last for some fixed period. The original version of each sentence would be displayed with a count of alternative translations submitted. Selecting the sentence would allow you to add your own version to the list.
3) Voting. Users would vote for their preferred version of each sentence, the decision going to whichever version first achieved a certain numerical lead over any other. Points would then be awarded to the winning translator depending on sentence length, its 'traffic light' rating, and the position of their translation in the list; submitting the best translation as a first attempt would earn more points than winning the vote with a slight amendment to three or four earlier attempts by other translators.
So, cherry picking headings and captions would not meet the minimum percentage of translation required to submit work for checking. Tokens could not be earned by maliciously down-voting others work nor, necessarily, by up-voting poor translations. Both translators and checkers would be rewarded for the quality of their work.
There would be separate systems for rewarding the efforts of the original translator and those making amendments, with no reward for pointless amendments or opportunity for endlessly reverting to overly literal translation.
Must dash... I'm hungry and I still need to have two more good ideas before breakfast :D
Indeed, but short words aren't gaining them any tier points since they must translate at least 5 words to gain a point. One way to stop compulsive upvoters has already been implemented by Duolingo. It warns you if you are upvoting too quickly. They could make this better by preventing any further upvotes if users make 15 upvotes within 2 minutes, especially for long sentences.
Another way to ensure quality is to either operate under good faith or use something like iCaptcha that I suggested.
Thank you. I wasn't aware of this, but then I wouldn't game the system. Is it 5 words in total, or all in one sentence?
According to the Immersion page :
- For each sentence translation you do, there is a limit on the number of votes that can affect your progress. The limit is one vote for every five words in the original sentence (so for a 10 word sentence, only the first two votes are counted).
Though I'm not sure how it works, because it either means that only <4 words = 0 Tier points, or it means that 1-5 words(in a sentence)= 1 Tier point at most.
Do you know, I think that may not be entirely applicable?
Overnight GregHullender kindly undertook some checking on the article on silver that I posted. In the "show votes" link on my immersion activity page I am seeing upvotes from things as simple as "translations" of "Ag" (the chemical abbreviation for silver) and "47" (its atomic weight). At the bottom of that window it says "Note: only votes that contribute to your progress within tier 3 are shown."
So it appears that someone simply copying across a number or translating a single word does get a credit or qualifying upvote for doing so.
Looking back down my upvote list, I also seem to have been credited for several other single words not requiring substantial translation (eg "entitad"), or indeed any (such as "jazz").
If this is the case, it seems to explain why some people do swoop on headings and suchlike in newly posted articles, and to suggest that one could reach very high translation tiers without ever actually doing any translation.
Or have I completely lost the plot?
Well, all that means is that even a one word sentence can only give you 1 tier point. However, unlike before where users could get at most 3 points for each sentence. Now they are restricted, for example:
- "A memorable night" = 1 tier at most(if upvoted);
- "Duolingo is a very good (1 point)/ program for learning a new (1 point)/ language". = max points are 2;
So it doesn't matter if 20 people upvote sentence 1, that user can only get one point, and 2 points at most for sentence 2.
So what I meant is that if they only translate 1 word "sentences", they will need to do 100 sentences to get to the next tier.
I don't think the problem is with the articles uploading. It is probably that people don't care to check a big document.
So I think one of the best ways to improve the system is linking proofreading to the tier system, for example:
- A Proofreading based system - To graduate to the next tier the user would have to proofread X % of sentences (e.g. Tier level 1, 5%, T2 -10%, etc), and translate new sentences.
- Article completion based - To graduate to the next tier the user must proofread a whole article, for instance Tier 1 proofreads 2 articles, T2- 4, T3-6, and so on.