List of structural problems with Duo’s vocabulary training/spaced repetition
Hi! I am a regular user, who finished the German tree long ago, but who have continued to use the vocabulary training for repetition. In doing that, I've noticed some problems (or properties of the learning system that I perceive as problems), which I've tried to list below. Some of it is immediately obvious, some of it only become a problem at a higher level.
I appreciate Duolingo a lot, so this isn't meant as complaining, but as part of a constructive discussion about how its methods for learning can be made even better. (It is possible that some of the things below have been improved recently, and that I just haven't noticed it yet. In that case I apologise.)
The timed practice is the most entertaining, but (as many others have pointed out) the lack of time for revision, for checking discussion and for reporting errors make them problematical. This forces the serious learner to chose the non-timed practice.
The non-timed practice, however, is in a way even more problematical. If I get four answers wrong, I fail, and I assume the relevant wrongly answered words get weakened. But what about all the words I did get right? They don’t seem to get strenghtened, and I have to practice them again the next time. This is not only very boring and annoying, but goes against the whole idea/science of spaced repetition. I don’t care if I don’t get any ”game points”, but I shouldn’t have to immediately redo exercises I had no problem getting right.
The multiple choice questions are often worse than useless. They are useless, since the answer in nine out of ten cases is obvious whether one knows the words or not (just choose the only alternative that is not obviously ungrammatical or absurd). Many others have already complained about this. But they are also worse than useless, since the only occasions when they are hard to answer, are when two or more alternatives may be correct. And then answering correctly often involves pure guesswork. How can I for example predict if Duolingo in this particular case will accept both ”es” and ”das” as a translation of ”it”, or both ”that” and ”this” as a translation of ”das”? I can’t, since it’s not a clear question of correct and incorrect answers, but a question of context and degree. Also, the easiness of anwering most multiple choice questions without even really reading them invites speedy sloppyness, causing unnecessary errors from just pressing the wrong key, etc.
Words only seem to be strenghtened, when they are the main word to be trained. At least sometimes this is the case. For example I recently answered the (annoying) sentence "Frankreich, Österreich, Italien und die Schweiz" correctly. "Österreich" was marked as the word to be trained, and was of course raised to four bar strenght afterwards. But why are the other three still at one or three bars? Isn't it a great waste of time to test me tomorrow or whenever for a word that I was reminded of and had no problem with today?
The better one gets, a larger and larger percentage of one’s incorrect answers will be due to other causes than lacking vocabulary knowledge. That is, most errors will be caused by typos, misreading, psychological mistakes (such as writing ”nine” instead of ”ten” because that was the answer in last question, or whatever). These kinds of errors are mostly random, and will thus statistically cause you to get common words wrong more often than uncommon words. Therefore common words, which you probably know well, are the ones that will get weakened most and you will have to repeat more.
Error reporting seems to work well when it comes to adding correct translations (you get a nice e-mail, etc). Less well when it comes to removing incorrect translations. There’s often no alternative for that to chose, and even if they’re pointed out in comments and reported as "other" problems, they tend to remain, and incorrect knowledge will be spread and reinforced.
What I find particularly difficult with the multiple choice questions, is that you are forced to read wrong translations, and that often repeatedly!. I remember that our teacher insisted to never write errors on the board, to prevent us from encoding the wrong spelling/grammar etc. Reading wrong translations so often just cannot be good. I don't like the MC questions in general.
I'm french, and I use the method "Projet Voltaire" for correcting my bad grammar (dyslexia or something like that). The program shows me a sentence, and I have to pinpoint the mistake, if there is a mistake (~75 % of the time). I have corrected some of my mistakes, but know I have new ones, like the homonym "et/est".
Therefore, I don't like the multiple choice questions either.
One way to improve this while keeping the multiple choice questions: Make all the choices grammatically correct sentences, each having a slightly different meaning.
A simple example: 1, He is eating an apple. 2, He was eating an apple. 3, He is eating apples.
This way we don't have to read wrong spelling and grammar. Moreover, it even helps us better deteck the nuances in grammar and word usage.
yes, I think I have seen that for some questions already (which might just be coincide). Hopefully they can adapt that for the others.
one way that I use the multiple choice questions is to look at the sentence to be translated without looking at the choices, and then mentally try and come up with a correct translation before I look at the choices. I find this helps me to use this exercise more constructively and I often also read the conversation thread to reinforce the correct translation. I definitely agree that these exercises would work better if they were all grammatically correct sentences and loved the suggestion below that they should the same sentence in different tenses.
It would be interesting to hear more about your thoughts on this. On point two, for example. Regarding that it should be very easy to implement a change, if you agree at all with the basic philosophical point -- that is, that the science of spaced repetition is more important, and thus should have a higher priority, than the idea of ”the game” (not that I think they’re mutually exclusive, of course). As Duolingo isn’t very transparent on its methodological thinking, it is sometimes hard to know if a property of the system is a mistake, or if it's made on purpose.
I agree especially with points 2 and 4, the way duolingo matches word strength to the lessons instead of to the sentences, and that not all words in each sentence are evaluated in order to come to word strength. This means that duolingo is using only a small portion of the data available to assess word strength, which in both cases causes repetition of words that are not in need of repetition.
On a related note, I also think the user should manually be able to indicate word strength in order to alleviate some problems with the current algorithm that determines word strength.
I don't think Duolingo is using only a small portion of data. Some time ago Duolingo team pointed out that word strength system takes in account most of the words from the sentence not just those we're supposed to learn in that lesson (I can't seem to find that thread :( ). Perhaps the issue stated in #4 is a bug.
you are right, I checked it before (some time ago though) and then I found that the word I had just practiced was marked as "last practiced x days ago". Now I did the same, and I found that indeed the word that I had used was marked as "just practiced". It could be that not all words in each sentence are appropriately linked to the word in the vocabulary, since both me and Bartleby had the same impression, and I actually tested it.
That still leaves #4 though... On the main thread you answered that this is a motivational thing in your opinion, but I was strictly speaking about practicing the words after the lessons. For instance, I am now stuck with a lesson for which I know all the words, but that have sentences with a lot of prepositions that I keep forgetting :(. Of course learning this is also usefull, but I would also learn it if different words would appear around these propositions.
Recently, I have found the multiple choice questions in the Spanish section to be as challenging as can be with only one letter differences between the options - I can't say much for the German tree since I only began it.
However I am curious about points two and four that you raised - what happens to our word strength for words that we got correct but failed the overall practice session and for words that aren't the main words of focus (as you explained)?
Also, +1 to point number six - incorrect translations need to be removed because IMHO they're worse than not having a correct alternative for sentence. How much good is it if I'm translating a sentence over and over, believing it's correct, only to find out later down the road that I was learning incorrectly?
Recently, I have found the multiple choice questions in the Spanish section to be as challenging as can be with only one letter differences between the options
Now the problem is straining your eyes to find this one letter that is different. I suggested to highlight the differences between the options, hope they will implement it some day.
Yes, the multiple choice has been improving lately, and it was confirmed by a developer that this is something they've been working on (sorry, can't find a link to the thread). It seems that they're saving users' translation mistakes and using them to supply the wrong choices, which makes it far more challenging. Not all questions have this yet but it seems to be rolling out slowly.
Yes, I agree that the absurdity of the absurd alternatives have been somewhat diminished. But I still think the same basic problems with the multiple choice questions remain.
Sometimes I have wondered what Duo uses to dictate what vocabulary to introduce. It is said that with 1,500 words a person can attain communicative competence for most everyday, oral exchanges. There are many studies on word frequency in oral, fiction, and non-fiction literature. Sometimes I wonder if the developers consider these studies when early on in a language I am learning words such as butterfly and monkey while I still don't know nouns such as meat and vegetables -- words that I might very likely need when traveling.
This is a great post. I agree with most of the points, especially 1+2 (in combination), 3 in what concerns guesswork, and 5 - very good observation. Hope the team won't miss this discussion.
I really like the multiple choice questions. Sure, sometimes an answer is absurd but often they are almost identical and I find I really have to read very closely to pick out the right one. I love both the timed and the non-timed practice. I use them both. Timed ones are fun and the non-timed ones are good when you want to take the time to read the discussions or listen to the audio more than once to remember the phrases better.
If I can add a point- when the "lesson practice" is selected for everything you have learned so far, it seems like there is a lot of repetition in the words they are having you practice. I would like to see more variety in of the much less used words when practicing.
I second those points, and I would also add something related to the inconsistencies in typo mistake detection: sometimes there is indeed a typo, but sometimes I make an actual mistake and it is detedted as a typo. It is especially true for me when I practice German since I sometimes add or forget a "h", which would be considered as an actual mistake if I were to take a written exam.
My opinion is that as for any real exam, there shouldn't be any "acceptable" typo mistake at the beginer levels, where you still need to learn the orthograph. Before some point, say before encoutering a word more than 20 times, any mistake on that word would be considered as a wrong answer (meaning there wouldn't be any typo accepted). After encountering the word more than 20 times, there could be an option to enable the typo mistake possibility.
check number #3 http://blog.duolingo.com/post/62723181502/duolingos-inaugural-hackathon When they implement it I suppose it will solve the lack of time for revision in timed practice.
Hearts and game points are here to motivate users to try harder in order to learn. Strengthening the words we got right from a lesson we've failed kind of makes those hearts and game points unnecessary. Don't get me wrong I agree with you on this issue, but I don't think it is an easy one to solve.
I am not a big fan of multiple choice type of questions but at least they have made some interesting ones in German recently.
Referring to point 2, I don't think that strengthening the words we got right in a lesson that we've failed makes hearts and points unnecessary. I think it would make the practice sessions focus more specifically on what you need to improve to get higher skill points and improve in the language in general (the assumed goal). The problem is if you have to keep practicing a bunch of sentences with material you do know just to get to the few sentences with material you don't know over and over again, it gets boring and frustrating and at some point, counterproductive.
Thanks. Certainly helps my understanding of how the spaced repition works but possibly this is a systems error. In Spanish "haces" is listed as needing to be strengthened,last practiced 2 years ago. Since then practiced numerous times but still no change. It is still listed as needing practise.
Wow, finding out this thread is 4 years old really lowers my expectations of duolingo ever fixing these problems that still exist today.
Please continue to expand the sentence variety. One of the problems I find most difficult is my own ability to memorize the answer. In other words, I see the sentence structure and I select the correct answer, but I don't necessarily understand the structure of the sentence itself. It's just that I've seen it so many times that I select it and therefore I am suppose to think I know it, but in fact I don't. I've just memorized it without understanding the various parts of speech within the answer. I find this especially true when translating from French to my native English; however, I would not be able to necessarily translate it from my native English into French. I hope this makes sense to someone.