Should there be a minimum skill level to report errors?
With the growing popularity of Duolingo, the current system of "error reporting" seems to create a chokepoint for effective handling of complaints. Huge numbers of complaints from new learners about 150 ways to interpret "Ça va", etc., create a noise level that slows to a crawl the reactions from Duolingo staff, the support team, and the volunteer moderators who are working hard to make genuine improvements. Here's a thought for consideration. What if there were restrictions that users must have achieved a minimum skill level before being allowed to register "errors" or request "My answer should be accepted"? This is not intended to disenfranchise new learners but to allow them to gain some experience before starting to bring in all the struggles, misinformation (my teacher told me....), etc., into the domain of program staff and moderators. I would think it would be fairly simple to program Duolingo to check the user level in the language. Even at Disneyland, there are signs that say "You must be this tall to get on this ride." I think there must be a way to implement some conditions on "error reporting" which will bring some balance into the system and remove obstacles from staff to deal with issues that will improve the program for everyone. Your thoughts?
Nope. Because I have a collection of e-mails thanking me for my contribution and changing entries from when I was a little newbie and making my way up those first branches. And if the newbies, who really pay attention to things, don't get to report when they think there is an error, or bad French, or bad English, or bad German, etc. Duolingo won't improve.
Now, if you are talking about the discussion section under each question, that is another problem. I found that very helpful as I went through - because people ask the same questions and so I often found the answer I was looking for, right away. BUT I wouldn't mind if some kind intelligent person weeded out all the questions and left the good answers...but wait, sometimes there were two good answers... but wait, sometimes I learned more from the questions... Well, perhaps just get rid of the older things so there aren't ten pages to wade through - but wait, sometimes the real answer was at the beginning - at least the one about grammar - the rest is about whether we would ever say such and such in English - a clear indication that the sentence has some problems. Hmmm. Guess I would leave the whole thing alone.
BTW, as I made my way up the tree, I did not find that the questions or the answers were of any greater quality - but they were of significantly lower quantity.
I am assuming that that is a measure of people falling off the tree. From frustration? Lack of information? The examples were too hard? The discussions give you clues. Too many 'why is this' and 'I don't understand' means that the material is to blame, not the student. I had a friend who was proud of spending hours after class tutoring students so they understood his math lectures, he actually remained my friend after I pointed out that had he taught well in the first place he wouldn't have left them confused. But he did start using his tutoring technique in the classroom.
So, now I am back to supporting having someone go into the discussions after they reach a certain level and write a cogent summary of the discussion - de vs des after negative, etc. and put it at the top where people can see it -- and having long, long, long, discussions triggering a review of the phrase by the team - to see if there is something wrong with either the French or the English. That is what crowd sourcing should do for us, create better information.
Well put! =) Let me give you my viewpoint as a course editor. More reports are never a bad thing. They just give us a better idea of what people are having trouble with. They do create some noise, especially before the first checkpoint, but we've adapted to this. We also have a system that automatically rejects certain reports and sends back canned explanations.
As for the discussion section, you've hit the nail on the proverbial head. Some of the earlier discussions have reply counts running into the hundreds! This is indeed noise and takes a lot of work to clean up, but we're working on grammar notes to try to cut down on repetitive questions.
We don't necessarily convene the Council of Elders whenever there's a discussion with a lot of comments, but we do spend a lot of time going back and forth about what the best translation for a certain sentence is. However, there are thousands of sentences, millions of users, and only a few of us, so it's kind of like 300...but with fewer spears.
Oh Ye Honored Council of Elders: I will now think of you guys as having beards and smoking clay pipes while sitting on ... well, buffalo robes.
Though a lot of reports aren't necessarily a bad thing. Reducing the noise is undoubtedly an important thing. If the system was well adjusted, I think that the courses could reach phase 3, and greatly reduce the number of errors within a short time frame. I can think of at least two approaches that will probably reduce the number of reports:
- Grammar checking - Automatically grammar checking every single report, and setting items with inaccurate grammar as low priority may reduce the errors;
- Spelling checking - would work exactly like grammar checking but would check all words for correct spelling. Those sentences that contain errors will automatically be given a low priority too, or alternatively rejected until the user provides good spelling.
Another approach could be for the system to automatically google/linguee the sentence to check how many hits it gets (if the sentence gets a lot of hits set it as medium priority), or alternatively check an internal corpus to to see if there are any hits.
As for sentence discussion, I think the easiest solution to that is simply preventing a user from posting something there, until they've read one of the tips and notes. Kind of like those annoying license agreements users have to read and tick before they can install an application.
Alternatively, Duolingo could always re-captcha them with some nice sentences from immersion containing the word. :)
Though I agree with this in principle, I think it will be harder to implement. Duolingo's skill levels are pretty much useless, and only indicate effort, not knowledge. One alternative could be something like the Proficiency test from the test center. However, rather than the full test, Duolingo could allow a smaller and free test that will quickly determine more or less how well you've mastered the language.
Thereafter, all reports sent by those who have completed that test are automatically prioritized, and help the moderators clear up simple errors. I also think that the first couple of lessons should be heavily reviewed by the course contributors, with ample explanations, and alternative translations. Once that's done, the error reporting for those lessons, should simply be disabled.
I mean, there are only so many ways to translate "La femme", it is probably not productive to keep receiving reports of basics 1 after about 1 year.
No, there should not be a minimum skill level to report errors. A number of errors occur in the English translations, poor audio quality, missing or inaccurate hints, etc..., and require no knowledge of the other language to notice. English synonyms and spelling variations are sometimes marked incorrect. If there is a problem with error reporting, perhaps explanations should be required with all error reports and some kind of metric could be applied behind the scenes to weed out the bad error reports.
I would suggest a few things - behind the scenes, you can make certain users or certain keywords move up or down in the queue of problem reports. For instance, if moderators have accepted five (or fifty) alternative translations from a particular user and not rejected any, the person could float to the top of the queue (or be asked to join the moderators?). If, on the other hand, five (or fifty) have been rejected as clueless, let the person sink further to the bottom of the queue. Ditto, certain keywords in the comments might be used for promotion or demotion.
It's clear to me as a user that there is also no tracking of the type, wait, not a single user gets this item right, could something be wrong with this question? For instance, Italian offers the wrong choice of articles for 'the foot', and has been doing so for quite some time now.
I'd rather see them overhaul the reporting system. Having it give feedback would be highly useful.
I've seen some of the mods mention they go through and find many of the same mistakes being reported as correct every day. There is no reason for a mod to have to deal with these at all. If they tracked previously rejected reports they could write a quick blurb about why it isn't wrong and then all future reports with that error could be answered automatically and the blurb shown to the reporter when they file the report.
Of course this isn't a viable approach for all reports but would certainly save a lot of time on those common ones that get reported hundreds of times. Not only would this free up mod time but it actually helps people learn since they don't go away thinking Duolingo is wrong and they just have to wait for them to fix it.
It would also be great if there was some way to track reports you made. The ability to go over them and see why it was rejected or possibly cancel the report if you realize after the fact that you were mistaken. Yet again, I realize asking them to explain everything would be time consuming so that has to remain optional and used only on the most common reports.
I agree that error reporting is a problem, however I am not certain that using the Duolingo skill level is the best way to address the issue.
A person may have a higher Duolingo skill level simply because they have been powering through the exercises longer than the intermediate- advanced French speaker who started the program a few weeks ago. In fact, I am sometimes surprised at the basic questions/complaints made by people at higher levels.
Perhaps some sort of area in the "profile" section where people can register their certification level received for tests such as DELF, TCF, DAEFLE, etc. The errors reported by people with such qualifications, could be 'weighed' heavier than say beginners with no certification registered.
I would also like to take the opportunity to thank all you amazing volunteer moderators. I know it is frustrating at times, but know that there are many who appreciate the work you do to make Duolingo a fantastic learning resource!
This is just an aside to your point, but something that I have thought about. I go back and redo my lessons in order to keep my French Tree golden and it is funny to see my old questions from a year ago in the Discussion section and see a 25 as the level and my streak next to it and no date. I have wondered if people have seen that and thought "Wow! She's really kind of slow. She's still asking basic questions and she's at a high level." I wish Duolingo would allow you to know how old the responses are.
mort de rire!
Yes you are being judged, however as Salihua stated above, the questions you had are likely still useful to someone encountering the lesson for the first time.
There are a few things that should be implemented into duolingo atm, like no date next to comments, or when you are translating it does not automatically update with latest updates, or not notifying you when someone replies to a message on your stream. When you find one of these problems you should make a discussion about it or email duolingo, etc, so the moderators and admins can see.
You can either delete your comment (if no one had replied to it) or add a reply yourself explaining it. That will help others who had the same misunderstanding.
Two issues with this:
1) Some people don't have a certification letter/level from a test. I, for example, have Grade 13 French from High School which technically should make me fluent (because you needed to take French in school starting in grade 3 so 10 years of French). But I'm taking the French as a refresher because I have not used much French in the intervening years. I'm sure there are others who are taking Duo as a quick refresher course.
2) If it's self-reporting. This means that anyone can say they did xxx and while it will cut down on some reports from the honest, on an online system with pseudonyms, people sometimes glorify their accomplishments.
I think another poster said it best when he/she stated that Duolingo skill level is more of a measure of effort in learning French, (in this forum), rather than overall knowledge of French.
Yes some people will not have certification - Native French speakers for example, would not have any certification of their proficiency.
In the system N6zs is proposing, you as someone with 13 years of French, would not be allowed to report errors until you reached a certain Duolingo Skill Level, while the person who has no previous knowledge of French, but spent several months working through the exercises, could.
My suggestion is some sort of way to capture background knowledge of the language (certification, classes, native speaker etc) and work it into an algorithm that prioritises/weighs/determines which error report gets looked at first.
I see capturing background knowledge of the target language as one aspect of an overall strategy to reduce the 'noise' of error reporting.
I would agree with this if the standard was high to begin with, but it was almost impossible to complete a single French lesson without losing all your hearts in the early days, due to frequent and persistent errors. How do you establish a "track record" when the material is wrong to start with?
OK, so French is out of Beta now, but you also need to measure TWO skill levels for any given language. For French from English you need to measure someone's proficiency in both English and French. How will that happen?
(This isn't my original userid by the way, so you can't judge my level of French from what you see above.)
I agree wholeheartedly with the need to measure two skill levels, but prob very difficult to do. I have seen quite a few reports about incorrect English from non-native speakers, indignant that their answer has been marked as incorrect, whereas they have simply not understood the nuances of the English phrase. Even for Anglophones it's difficult because what is common in one region, (or even one generation) is unheard of in another. :) Tricky business.
I guess one could hope that most queries come through "sentence discussions" and get answered without tying up moderator time.
I think it's a great idea because, just like in life, people should have certain credentials if they want to teach or instruct somebody. Certainly, it isn't fair that anybody can correct other people's sentences, downvote comments, report errors, etc, without having even reached level 1. Therefore, it's important to have a tool that could help us see if the correction comes from a "reputable source", or if the error being reported is an actual error and not just a misinterpretation of a rule. I propose to establish level 10, as a minimun skill level to have "expert rights", or a long test to check if the student is a native speaker. Since I'm not a native English speaker, I would appreciate any corrections to this paragraph and your opinions on the matter at hand. Thank you
And if Duolingo's team already has applied all of what you spoke about ? :) Recently, in a discussion, I don't remember its title, someone was talking about a tab, which shows you the vocabulary you have learned. And this section is not accessed by everbody. Personally, I don't have it, though I like it very much. :) And I know someone who does have it. And I think I know which kind of people have it. :)
Maybe Duolingo's team has already used some kind of selection of these errors, with the same method used to give you or not give you the vocabulary section ? :) Who knows ?
It is useless to remind you, isn't it, that Duolingo has a lot of information concerning each member : the member's level, his or her performance, if he or she has completed a tree, and in addition to his or her level, Duolingo knows exactly in which section of a tree he or she is...
So, I really think that in the algorithm of Duolingo's Program :), it has already applied, with all this information put at its disposal, one of them to make all the selections it wants, including the selection of errors we are talking about.
If there are any errors in my comments, and I'm sure there are, please correct me. You are welcome.
Most of my issues with the French course are about the English. It often makes no sense at all. Not for any English dialect. Perhaps native English-speakers might flag up their proficiency somewhere when reporting jibberish?
The tricky part is that some people really believe their English is perfect. That includes the moderators of the course. I recall one instance where I reported some obvious jibberish in a thread only to have the moderator insist it was correct. Perhaps people could identify their one native language?
That's the disadvantage of translating. It is an art not a science, and just about any nonsense can be justified when one calls it an art. The best way to have a discussion with a moderator or any intelligent being for that matter is to offer evidence in terms of references.
Duolingo needs a proper method of certifying proficiency, not native language because as you said being a native speaker doesn't make one an expert in writing or even speaking for that matter. Whether it tests proficiency deliberately or behind the scenes, it could rate the submitters using some intelligent algorithm, and then use that to prioritize the report queue.
One possibility is the language tree itself, people with an above average success rate in the lessons could automatically be given higher priority. Another possibility is a hidden test, one that appears like the lessons, but every so often pops material that can't be incorrect, e.g. "correct a sentence". This would work like re-captcha, where people would think they are doing only a regular lesson, but the system is also testing them and using that to prioritize the reports.
I just came acrosss "I will do an attempt" as the correct solution to a translation. There might be others which are accepted but that's the one placed at the top on the discussion. It is not correct English anywhere (according to people in that discussion). How could I provide a link to show evidence that this is poor English? .That the thread grew so long, with native speakers from around the world saying it is not correct English, just illustrates that those with the power to correct things truly believe that their English is right. Those who wrote the course did a fantastic job but now need help from native English speakers, and perhaps also need help to understand that they need help.
I'm not even a L1 English speaker and even I can find many ways to invalidate the claims of any moderators who believe it is right. First of all, a simple search for a quote of "do an attempt" returns fewer hits vs "make an attempt". So even if the phrase was correct, it shouldn't be one of the first shown.
Secondly, apparently according to this paper (http://www.academia.edu/1689598/Language_Structure), that way of writing is wrong, and it apparently references this (http://applij.oxfordjournals.org/content/19/1/24.short) paper that discusses it further.
So really, empty statements indicating that "native" speakers don't say something are pretty much useless in a debate about language, there are many things that native speakers don't say, but they do in fact write and are grammatically correct (e.g. https://www.duolingo.com/comment/3982790).
Alternatively, it could be that they are volunteers who already put in a lot of effort and who are now overwhelmed by the amount of discussions and problem reports generated by the million users of the course. I get frustrated when things get counted wrong that are actually right, but I can also see how hard it is to anticipate, and then keep supplementing, the many variants of ok answers that people might give. So I try, not always successfully, to shrug it off:-) I do remember one time in a fit of .. something, accusing the system of male chauvinist behavior when a female version of "I am such and such" (professor, director,..) was not accepted:-)
Well, in truth, I'm not sure how Duolingo checks these answers but depending on how it is programmed, this is something that wouldn't necessarily require any interaction from the course contributors. A simple dictionary check could immediately determine that there are many ways of referring to a professional/worker,e.g.:
- User answer: Je suis professeure
- Stored answer : Je suis professeur (wrong)
- Dictionary check : Professeur ~= Professeure (Correct).
Feedback : 3 or 2 →Correct answer !
The real issue is that once you change the gramatical gender the sentence may need extensive changes to be correct once again.
Hadn't realized this started as a thread in French; I originally made the comment about Italian, I think :-) Not so sure professeure would be ok en français.. And dictionaries can be misleading. Just because English plant can be both usine (factory) and plante (green thing) in French, you can't use them as alternatives in sentences. But I don't enjoy being dinged for 'must' when the 'correct' answer has 'have to' etc.
Surely the moderators can filter the "my answer should be accepted" requests by the user's level at the time the request was submitted? SELECT * FROM requests WHERE level > 5 ... is SQL a Duolingo language yet? ;)
Oh god, I'm imagining coming up with SQL commands in a timed exercise.. select * from requests where user in (select user from users where....) => but my version is wrong. level should be associated with every request because before you know it, that same user is now level 12. However, user might also have been designated clueless trouble maker, which might be a permanent characteristic: where level > 5 and user not in (select user from quarantinetable where reason is 'clueless')..
Yes, if you need a simple test but filter them invisibly behind the scenes. We need a way to blow off steam when we feel some idiosyncrasy has done us an injustice! THANKS, LOVE DUOLINGO!
I think it is fine how it is. The task is on Duolingo to filter through the noise, for example they could simply filter out the lower levels of proficiency according to their own ranking system.
A variant on your suggestion:
Leave the "Report a Problem" button open to all, but for technical problems only. As others have said, if sound quality is poor or some other Duolingo lesson facility is broken, you shouldn't need to be experienced in the language to report a problem that stops you using the site as intended. Nobody else may yet be having the issue because of operating system or configuration differences.
If someone has a problem with the language (regardless of skill level) they should open a new or review the existing "Discuss sentence" entry. If, having read this, they still think there is a problem, post to the thread. If their post is upvoted by enough other Duolinguists (2-3?), it gets automatically reported.
Maybe users with sufficient "skill" could have have a report now button on the "Discuss sentence" thread but, even then, they cannot report their own posts. This is what is sometimes termed "four-eyes" verificiation; at least one other person has to agree there is a problem before it gets reported.
I don't entirely agree. What you're proposing is to restrict fluent English speakers from reporting errors in the English translation because they haven't attained a requisite French level, and conversely, allowing fluent French speakers to report English errors, even though their English may be relatively poor.
There are four categories for complaints / errors:
French sentence error
Perhaps implementing a "skills" level requirement prior to reporting the first error (French sentence error) would reduce the number of "how to interpret X" comments from beginner French students.
In the second (dictionary), and third (audio) options - I assume the moderators probably already know about this. The biggest issue here is lack of feedback from Duo. If I notice the audio is bad for a particular question, I have no way of knowing if 2, 30 or 500 people have already reported it, and I don't know if it is in the process of being amended.
Could a space be allowed for the moderator to make a red/bold comment directly above the "Report a Problem" button? They could then disable the dictionary or audio check-box to prevent people selecting it, or replace this option with a red note (eg: "we are working on fixing the audio").
The fourth (other) option encompasses everything else, and I'm unsure what the typical problems in this section cover. I do know there are problems with English definitions, grammar etc.
Why not include another check-button above "Other", labelled "Error with the English sentence"? You don't have to speak fluent French to assess those errors, because I assume the same problems occur on the German etc. portals as well.
Conversely, for the "French sentence error", there's no reason why the moderator assessing these complaints have to be bilingual, either.
Perhaps Duo could have (using the French/English courses as an example, but assuming it would apply to all the languages Duo offers) bilingual (French/English) moderators, French-only moderators, and English-only moderators. The latter two would surely be able to deal with a large number of those issues which don't require special bilingual skills, and they could be used to "filter" requests to the smaller number of bilingual moderators, if necessary. An English moderator, therefore, would receive errors from the French, German, Italian portals, etc. This would alleviate some of the pressure on the bilingual moderators, and would allow more people to be able to volunteer to contribute.
Disneyland signs are for safety. I am more in favor of the "rubber room" approach used in New York City, although I must admit I have never been to New York City, nor do I know anyone in New York City. The approach was described to me as a sequestration of those against whom complaints are filed until a time when the complaints are resolved. Similarly, questions with complaints should be removed from the question bank until the complaints are resolved. Ultimately, complaints emanate from various sources depending on users' knowledge, experience, and community. Neither the number of complaints nor the skill level of the plaintiff determines the validity of the complaint.
What determines validity? 1) a break with a rule of grammar, assuming knowledge of exceptions to the rule 2) a break with semantics 3) a break with common parlance, possibly using google hits or something to this effect 4) a break with phonetic rules 5) a break with common speech Questions: 1) Does the phrase break a rule of grammar? 1-5 2) How likely is it that a native speaker would identify an irregularity in the phrase? 1-5 3) How likely is it that the meaning of the phrase is altered? 1-5 (order determiner) 4) Does the phrase break a phonetic rule? 1-5 5) How likely is it that a native speaker would identify an irregularity in speech? 1-5 a score of 7? or higher would indicate a true complaint,,,3 for rule break + 4 for native speaker Questions 2) and 5) require input from a self-identified native speaker.
Most of the errors I have encountered can be dealt with either as errors in the English language or errors in the French language. I have yet to encounter a true translation error in practice exercises, where the wrong word is used or the phrase is changed incorrectly, as I have encountered, and hopefully have not committed, in the immersion section.
Someone learning the language would not be able to enter a true complaint until a native speaker agrees with them. You could open up the question to the French learning English group or handle it internally. Reroute the question under the bell icon or something as a possible way to earn a lingot. Ultimately, some complaints still may be "noise," but you have at least prioritized them, a real beast in some workplaces. More importantly, the question will be removed from the question bank until its accuracy is determined.
The validity of your skill level determination is quite low, as I, who self-identify as a native speaker, tested mid-level in English. Currently, I have obtained a higher skill level in French than in English, but I would not have been able to write this paragraph in French without incurring many more errors. Therefore, skill level should not influence one's ability to submit a complaint. (I believe someone else pointed out that problems may arise at any skill level.)
Problems with using the absolute number of complaints as a determiner of complaint validity: 1) You may be the smartest person in the world and be the only person to identify the error. If this is the case, then perhaps you should write a book. 2) Some errors are harder to identify than others, but the validity of the error is not determined by its probability of detection. Unlike many calculation errors, a significant proportion of language errors cannot be detected by absolute rules or algorithms. More human input is required. 3) Some errors may be seen by many but are so minuscule that taking the time to point it out would be a waste. e.g. There is no period at the end of the sentence.
It seems glaringly obvious that a test of skill level should be entirely in one language. Suppose you can convert pesos to dollars and dollars to pesos. Does that prove you can set up a trust fund for Mexican immigrants to the United States? What kind of knowledge would be required to perform such an action? Before I leave this train of thought, I would like to ask duolingo how they can offer English to Chinese, but not Chinese to English? If you have no Chinese speakers on your staff, then how do you know your translations into Chinese are correct? If you have Chinese speakers on staff, then why not offer Chinese to English speakers? There is a vast divide between our French-to-English and our English-to-French course? Pas vraiment.
Back to "ça va." I began learning Dutch through duolingo the other day. I knew zero Dutch words before beginning the course, and I have had no problem using duolingo's hover method to figure out what new Dutch words mean. That is really the only way to do it without looking them up in a dictionary. My final point is that the persons who complain about "ça va" are persons who are overqualified to be taking those lessons. Therefore, restricting access to those more highly qualified individuals would be pointless. The truly clueless do not know where the "ça" to "va."
Let me start by apologizing: I've reported errors only to find an error in my version after my report. So for all of those excesses, sorry. I hope it hasn't put me on the "ignore list".
But there are other solutions than filtering based on user level that might address the issue of an overabundance of error reports:
Situations like the one I mentioned above might be improved by better descriptions of the errors made. If the correct sentence and the incorrect input are shown side by side with the errors highlighted then users will be clear on what exactly they did wrong. This happens in some cases, but not always.
Instead of focusing on individual errors, people looking to make corrections and fixes could prioritize based on the number of reports for a given question. If only one user has a problem with question A but 100 have a problem with question B, then its clear that B is far more problematic. Fixing those problems with B will also do far more to reduce the flow of error reports (it's the bottleneck).
Perhaps once a question reaches a certain threshold of error reports it should be at least temporarily deactivated. Of course more complex sentences will have a higher rate of error reports, but even in such a case that information is useful and might be suggesting that that sentence is TOO complex or complicated for that particular lesson level or duolingo entirely.
As a user I have no idea how any of duoling works behind the scenes and perhaps this is all already happening. If so, sorry for the redundancy.