https://www.duolingo.com/profile/qbeast

Disturbing multiple-choice question statistic

I have read a number of general posts criticizing Duolingo's multiple-choice questions, and I thought that I would add something concrete. As background, I am a retired teacher whose career included a stint writing multiple-choice questions for a national standardized test.

I spent a block of time yesterday morning doing an experiment in the Spanish-from-English course. I tried doing a series of 45 multiple-choice questions without actually reading the questions. I answered based only on the three answers given. My score was 45 right out of 45 questions. In fairness, I guessed on 4 of the answers, based on context. I would choose, e.g., "The flower is beautiful" over something like "The stapler is beautiful" because the first is more likely, even though both statements are grammatically and logically correct. But even 41 right out of 45 is a striking number. I also could have answered over half the questions correctly by simply always picking the shortest answer.

In testing jargon, the wrong answers on multiple-choice questions are called distractors. Ideally, the distractors should be answers that might be picked by a student who didn't understand the key concept which the question was testing. For most Duolingo multiple-choice questions, the distractors are absurd, and such questions are simply a waste of band-width. It shouldn't be possible to do well on a test without reading the questions.

End of rant. Interested parties might want to glance at https://uwaterloo.ca/centre-for-teaching-excellence/teaching-resources/teaching-tips/developing-assignments/assignment-design/designing-multiple-choice-questions

October 4, 2019

110 Comments


https://www.duolingo.com/profile/Tembo441

I think that, like much of Duolingo, the value of the multiple choice questions is that the learner is repeatedly reading correct (usually!) sentences over and over.

I, too, noticed that the correct answer tends to be the shortest one - BUT this happens much less as you progress to higher levels.

Also, almost invariably, the multiple choice questions are closely associated, sometimes identical, to the other questions coming before or after them. They reinforce the learning progression, and they aren't designed to be particularly difficult in most cases, I think.

October 4, 2019

https://www.duolingo.com/profile/uphilldweller

I think, test out and get to the meat.

Then again everyone learns differently so I wouldn't deny students the option of mustiple choice - even if the answer can be obtained without reading the question.

Choice is good. I wish there was more of it on Duolingo.

October 4, 2019

https://www.duolingo.com/profile/DanielVanD18

Sometimes the incorrect choices in multiple choice are related to the correct one, but with something obviously wrong, so it is easy to pick anyway.

When I have not done a skill for ages, sometimes they reinforce words, but only temporarily. I, and a lot of people, need to write it to remember it. They are probably just to get the person doing it to review a few words they will use in the skill.

October 5, 2019

https://www.duolingo.com/profile/Janet240859

The point is, you read the multiple choice answers in your target language.

These are exercises, not tests!

The purpose of the options is to get you used to seeing the language and knowing what makes sense and what doesn't - so in this case, the English translation at the top is the distractor - and the exercise would be nearly as useful without it, as there is almost always something linguistically incorrect or illogical in two of the three options.

Try them again with a "learn", not "test", mindset and see if you reach a different conclusion.

October 4, 2019

https://www.duolingo.com/profile/_M_M_.

"These are exercises, not tests!"

That should be in capital letters above most pages in duolingo: it seems the majority of posters in the forums don't understand this. Every time something new is put in a lesson someone says "how am I supposed to know this if they haven't taught it yet?" But of course, the exercises are how duo teaches the language. Yes, it sticks some info in the lightbulb sections, but that's window dressing.

Duo's main design is to get you to do the lessons over and over again. If you do that, you'll eventually get a feel for what's right and wrong in the language, even if you can't cite the correct grammar rule.

After all, when you speak a language, you can't stop and think of every grammar rule along the way or you'll never finish your first sentence.

If you are truly new to the language, you're not going to be able to do 10 translation exercises or dictation exercises the first time you start a section without repeating all the questions a dozen times; you'll get incredibly frustrated and probably quit Duo within a few days. Duo needs to keep the initial levels relatively simple; the objective is to first expose the user to a new area of the language, and get an idea of that in their short-term memory. (note: the ultimate goal is putting things in their long-term memory; excessively repeating something in a single day doesn't get information into your long-term memory any quicker) The user should then go back to the lesson over the next few days, then over the next few weeks later, etc. until the correct way of using the language is transferred into their long-term memory and they can produce sentences on their own.

[note: this is the opposite of how most schools teach: go over something in depth for a few days so it's in the learner's short-term memory, then review it maybe twice more: once for a test and once for a final. When the next school year starts the students have little in their long-term memory from the year before.]

Going back to the original poster's statement: there's no educational benefit to avoiding easy questions. If a user is using Duo as his principal source of learning, then ideally every time he does a lesson he should be exposed to a few easy questions he'll get right the first time (to give a sense of some accomplishment so he doesn't quit right away if he's having a bad day), a few hard questions that he might need to repeat several times (to try and teach him something brand new), and a bunch of medium difficulty questions that he has a chance of getting a good percentage right the first time, depending on how rusty he is. Of course, what is "easy" and what is "hard" should change depending on how many times the user has done the section.

Duo certainly hasn't perfected how to consistently increase difficulty as you go up the levels, but I think they're trying.

This means the first time you do a lesson you should probably be shown mostly picture questions and multiple-choice (and I question whether the multiple choice questions are all that easy for true beginners). As the user comes back and repeats the lesson these should be phased out for other types of questions.

October 4, 2019

https://www.duolingo.com/profile/cwicseolfor

Thank you so much for this comment. I think it sheds a light not only on what Duo is trying to do with the exercises specifically, but why it works generally, in the absence of aggressive testing: it trusts the audience. The system trusts the learner to be self-motivated, to explore their own limits and the edges of their grasp.

Much as you say, it's the opposite of formal school, when the institution is a young person's duty and where what they do is classed as work: Duo's a gymnasium and a laboratory, a space for productive play.

October 5, 2019

https://www.duolingo.com/profile/sdelta1

Also, getting the answer wrong, especially repeatedly, is how we learn best. The Russian exercises where I kept making different mistakes and needing to write the same answer over and over are the ones that I remember best.

October 6, 2019

https://www.duolingo.com/profile/scarcerer

and I question whether the multiple choice questions are all that easy for true beginners

They are. Although because you can pick the shortest option and be correct more than 75 % of the time without looking at a single word, I can’t really comment on the easiness of the content.

October 7, 2019

https://www.duolingo.com/profile/PurpleHippo666

Although I understand what you are saying, I believe that this style of learning becomes less effective when it comes to harder topics, such as accusative case. There are several ways to say words like 'eating' and 'has/have' and you aren't really told how to identify the difference. It's just answers that obviously fit and ones that don't, no possible answers that represent the other ways of saying such words.

October 6, 2019

https://www.duolingo.com/profile/merkavar

yeah i find the multiple choice to be a bit odd.

like there will be three options, and more often than not it seems there is only one possible answer, as in it is obviously the correct one.

The answers need to be closer.

like changing only one word, in stead of the entire sentence.

like say you are translating a sentence with teacher in it, there is normally only one answer with teacher in it, so you dont even need to know the rest of the sentence.

October 4, 2019

https://www.duolingo.com/profile/hanspersson

Quite often I get questions where only one word is different. The problem is that it's not a word that makes me think about grammar options, but a simple one taught at a much earlier lesson. Just scan for the only sentence option that mentions "horse" (or whatever the question is about) and you're done, without even trying to parse the options for real.

October 4, 2019

https://www.duolingo.com/profile/JiPxB

Well said. Duo occasionally throws in a word with an incorrect ending as a distractor, but that's pretty rare. If it were more common, I would pay attention to the grammar much more carefully. As it is, I just stop if I come to a word stem that is not part of the target sentence and move on to the next candidate. (Yes, I do read the "correct" version several times to make sure I understand the grammar, but probably not as closely as if there was a subtle mistake in one of the candidate sentences.)

October 4, 2019

https://www.duolingo.com/profile/nl38rtaX

Oh, I thought you meant some spooky, unsettling, or disturbing multiple choice questions

October 4, 2019

https://www.duolingo.com/profile/qbeast

Perhaps nothing could be more unsettling than talking owls, beer-drinking bears, and the other whimsical critters that prowl through DL lessons. :-)

October 4, 2019

https://www.duolingo.com/profile/Hope150193

Not worried about them. It’s the drunk angry parrots in the Latin course that are truly unsettling. Don’t ask, but somebody died....

October 7, 2019

https://www.duolingo.com/profile/phfaucon

yep, what's the story about the angry parrots?

October 7, 2019

https://www.duolingo.com/profile/Songve

Ha, ha, that was my first thought as well.

October 4, 2019

https://www.duolingo.com/profile/Druckles

Having been doing the Spanish Course pretty much from scratch again since its update, I've also noticed the same trend: the shortest answer is almost always correct. As a computer scientist I was actually wondering how easy it would be to train a model to pick the correct answer based on only the three answers. You've partially answered that for me :-)

I also have beef with the multiple-choice questions. I tend to glaze over when I'm doing them, rarely read the whole sentence and tend to jump between key words. In my opinion, they're a waste of time in comparison to writing full answers where you're required to mentally process the sentence and translate its meaning. Moreover, the distractors are often grammatically wrong or use words that may be valid but aren't taught anywhere in the Duolingo course. I would go as far as to hypothesise that this is harmful to the learning process.

October 4, 2019

https://www.duolingo.com/profile/Joshorst

You could of course not glaze over the answers.

October 4, 2019

https://www.duolingo.com/profile/Druckles

It's a gamified system. It's essentially designed to train you to put in minimal effort for the most amount of reward (in this case lessons/lingots/gold thingies). Irrespective of what you think is the best learning method.

October 4, 2019

https://www.duolingo.com/profile/davidcwalls

Normal production and interpretation of language is almost certainly a system of putting in minimal effort for the most amount of reward, i.e. transmitted or received information. So it seems possible that what happens during those exercises is similar to what happens during everyday language use. Naturally, in normal use one doesn't compare a good sentence to bad ones, but I imagine that when reading or listening, one generally does only enough analysis to make the sentence fit within the current context.

October 5, 2019

https://www.duolingo.com/profile/Druckles

Yes, absolutely. Which is why I disagree with @Joshorst's oversimplification. We are essentially training ourselves to "glaze over" the less important parts of a conversation and focus on the details. We are training ourselves to make listening and speaking so automatic we don't have to think about it any more. That it flows naturally.

But the multiple-choice questions are missing this key ingredient: the context. Which means they're pretty much useless ;-)

October 6, 2019

https://www.duolingo.com/profile/Tembo441

Yes, Druckles. But we are, I hope, also training ourselves to read automatically, as well as to listen and to write. Being able to rapidly pick out the correct - or even just the relevant - details from a mass of text (a small mass here) is a very important skill. Perhaps even more important than listening and speaking for some people.

I think this is one of the strengths of the multiple-choice, among others, and one of the major reasons it should continue to be included without any drastic changes.

October 7, 2019

https://www.duolingo.com/profile/Joshorst

You've got to convince yourself learning a language, presumably what you came here for, is more rewarding than gold thingies.

October 4, 2019

https://www.duolingo.com/profile/Druckles

Or we change Duolingo so it encourages the right behaviour.

October 5, 2019

https://www.duolingo.com/profile/Kelvin401862

This is actually the most difficult part of using Duolingo for me, I love the gamified part of it, even though I realize it is not always helping me to learn.

And I often wonder where the break-point is, at what point does it become too much of a game.

October 4, 2019

https://www.duolingo.com/profile/MickLynn

"gamified"? Is my bad leg gamified?

October 5, 2019

https://www.duolingo.com/profile/Tembo441

Very English, Mick. But I don't think the word ever evolved past "gammy". Pity.

October 7, 2019

https://www.duolingo.com/profile/Grant7586

qbeast, Just out of interest, and if you don't mind me asking (as a retired teacher), if you were still teaching (at a school) and were teaching languages, would you consider using Duolingo as part of your language teaching (in class)?

Or would it be more something the students/pupils could use outside lesson time (even as homework) in addition to (to perhaps compliment) classroom learning?

Personally I'd be interested to know your thoughts, as various teachers have commented (with their thoughts about Duo) and it is clearly used in the classroom, even in some "developed" countries (which surprised me a bit), yet others have said that they looked at Duolingo (for their school or class) but totally dismissed it.

October 4, 2019

https://www.duolingo.com/profile/qbeast

A teacher's most precious resource is time, and I wouldn't burn up class time with something that kids could do on their own. If I used DL, the computer time would be out-of-class. I would also let students know that they were responsible for vocabulary and grammar covered on DL. The grammar would be reviewed in class, as needed. The vocabulary would be used on quizzes and tests, and also in in-class role-playing exercises. Without such accountability, a lot of students wouldn't really learn the material.

I think that DL is missing a great opportunity. There should be textbooks to parallel the major DL courses, available both in print and online. There could of course be a charge for such material. The program would pay for itself.

As to "developed" countries using DL, do be aware that there is a significant shortage of certified foreign language teachers in the US, especially in rural areas. Online instruction might sometimes be the best option.

October 4, 2019

https://www.duolingo.com/profile/Grant7586

qbeast, Thank you so much for your insight - it's interesting.

I left school as the transition from "traditional learning" to the use of computers gathered pace.

I was fortunate at my school (in the UK) in that although the choice of foreign languages was quite limited (from memory you could certainly choose French, German or Spanish - possibly Italian too) I chose French, and my Frech teacher was actually French (and a nice man and a great teacher to-boot).

For 2 years running I did a school exchange trip, whereby I and my classmates learning French, went to France for (I think) 2 weeks and stayed with a French child and his family. Then, later, he (the French child) came to England and stayed with me and my family.

It was very interesting sitting in the English class in a French school!

I don't know about now, but in the UK then, for the first 3 years of secondary/grammar/senior school it was compulsory to learn a foreign language.

In many parts of Europe (and elsewhere I would guess) I've read pupils have to learn (it's compulsory) another foreign language, in addition to English. So learn at least 2 foreign languages.

Certainly, I would guess the UK is far behind our European neighbours in this respect. And without trying to be anti-American, I would guess the US schools are generally further behind still.

On my original comment, a reason for it, was that when I was at school, you knew that your exam course books/text books had been carefully chosen by the local/regional education authority, school or teacher - so the quality of your learning material was carefully vetted beforehand and mistakes in these books were incredibly rare. Or non-existent.

Duolingo isn't like that. Certainly on the Russian course, there are mistakes in both the Russian being taught, but also in the English used as the base language. Really simple and basic errors.

So perhaps from what you're saying qbeast, even in some "developed countries" (like the US), for many pupils/students, access to foreign language learning is a problem - so to give them access to learn from Duolingo, is better than access to nothing at all.

I'm not "running-down" Duolingo, it has really helped me improve my understanding of Russian, and my conversational ability (in that language) and I've used the system every day for 14 months now. So on a level, I'm quite a fan of it.

Also I guess education methods have changed greatly over the last 20-30 years due to newer technology becoming available.

October 4, 2019

https://www.duolingo.com/profile/Seanchai35

American here - not anti-American at all to say that our school systems, broadly anyway, are way behind the pack when it comes to language learning. Part of this is simple but widespread underfunding - no surprise to most who have worked in educational fields, I suspect.

The US has an additional complexity to our education system which leads to a separate problem, though. In general, and setting aside private schools and charter schools for the moment - what one learns in public (federally funded) schooling is controlled by the individual states... which means you could start French class in Michigan, move to Florida, and end up in a French class that on paper is the same class, but in practice is three months ahead, or three months behind, or is teaching things in an entirely different order, using a different textbook. There have been attempts to standardize the curriculum (google "Common Core" if you're curious, but be warned, it's quite a rabbit hole), but by and large, the fact remains that not just the quality of your education, but what you are actually taught can vary depending on which state you're in. (Ask someone who went to school in the US Midwest about causes of the US Civil War, and then ask someone who went to school in the US Southeast. You will receive fundamentally different answers.)

The general guideline for foreign languages in US public schools is still two years of any one foreign language for college (university) bound students.... and that's it. Most students start at age 14-15 (far too late imo) and have "finished" the requirement by 16-17.

Add in that the quality of foreign language teaching can be everything from "fantastic" to "non-existent," and together, the state of US foreign language education starts to make sense... late introduction + underfunding + sloppy standardization between states/school systems + scarcity of good foreign language teachers + lack of support for those teachers = I sat in a class that was chanting "oono, dough, trees" for the majority of my mandatory language learning, when, as a kid with Cuban family who had at one time spoken fluent spanish (but by then was quite rusty) I should have been in a class that was going over future and subjunctive tenses... something the classes I was forced to take never even got within touching distance of. I literally forgot more Spanish in "Spanish class" than anywhere else.

It's also why the majority of Americans end up never picking up a language later in life - if one's only experience with foreign languages is two years in a classroom full of people struggling with the bare basics, even if you didn't personally struggle, it's easy to come to the conclusion that language learning is both too difficult and not much use relative to the time investment... as opposed to our friends in Europe who are exposed to a greater variety of languages at least existing near them - if not on the street, then on holiday, etc.

That's the sort of vacuum that Duolingo is stepping into. Whether it's a help or a hindrance depends largely on the individual teacher and how supported that teacher is in terms of funding/admin (I had a Spanish teacher for one semester who was absolutely brilliant and who never would have dreamed of using Duolingo simply because she didn't need it... I can envision her letting us use it for extra credit out of class or something, but it definitely wouldn't have substituted for class time in her eyes. The Spanish teacher I had for the other three semesters was so ill equipped both linguistically and in terms of classroom management that "just sit and do Duolingo for an hour" would have been a boon to all of us - we would have learnt more and she wouldn't have spent every day at the end of her rope!)

October 5, 2019

https://www.duolingo.com/profile/carol888177

Some universities in th US have dropped the requirement of two years of a foreign language in high school. So some high school students opt out of learning a language in favor of other classes. A foreign language is considered "hard" and may drop their grade point average.

October 5, 2019

https://www.duolingo.com/profile/Seanchai35

Yep, that's happening too. The lack of standardization is the main thing I was getting at... there's virtually nothing one can claim as an absolute when it comes to learning foreign languages in American schools - some of them do it really well, some don't do it at all, most are somewhere in between.

October 6, 2019

https://www.duolingo.com/profile/b05aplmun.ca

Um, most school funding is local, not federal, although there is some federal funding for specific programs. This creates inequities in what is available where.

October 6, 2019

https://www.duolingo.com/profile/Seanchai35

Yes, I'm aware of that. I cut about four paragraphs from my post explaining in far greater detail how public school funding works, but I figured that a) most people wouldn't care and b) this is a little bit of a thread derail to begin with and most people would probably appreciate it if I reined it in. But yes, gold star, public school funding is complicated and deeply influenced by property taxes (and lack thereof).

October 6, 2019

https://www.duolingo.com/profile/NovemberVee

On a side note...I sometimes wonder if a few of the mistakes in the Russian course aren't put there (or left there) intentionally, so native speakers can say, "Listen to that idiot. Must have learned online." Regardless, it's still better than the other option: "Look at that idiot. You have to wave and point because that's all she understands."

October 4, 2019

https://www.duolingo.com/profile/lullaby180

I don't think so, NovemberVee because in French, there are so many mistakes, it can't have been left on purpose:D So as a teacher, I would not use it in my Classroom, at least for French Learning.

October 5, 2019

https://www.duolingo.com/profile/MickLynn

Grant, booting your teacher, even if he's French, sounds a bit over the top You must be a brexiteer. (i'm kidding, if people take this seriously)

October 5, 2019

https://www.duolingo.com/profile/susu581980

For me the right answer is always easy to pick. The benefit for me is the opportunity to check out the vocabulary and sentence structure that will be used in the more difficult levels before I have to generate it on my own. Somehow it makes the learning process very gentle rather than jumping in the deep end.

October 4, 2019

https://www.duolingo.com/profile/04Amanita

I appreciate the intention, the inquisitive mind, the good use made of the OP's knowledge in this post. Thanks for writing something informative and interesting! If we were enrolled in an educational institution, if I felt my knowledge were really being tested with an end goal in mind, then I would be upset, but this is a wonderful, free, learning resource - it's meant to supplement other learning resources or help you get on track with learning another language. Can't we just enjoy it?
When I want to get certified in another language and put it on my resume, then I'll get picky about my learning experiences, and for sure I'll remember this interesting post. On Duolingo I just want to enjoy myself and putter around the place. In peace. Frankly, I don't care about the multiple choice answers being too easy most of the time. There is nothing in my life that has come close to having the positive impact on my learning that Duo has - this is one of the good dreams of the internet.

October 4, 2019

https://www.duolingo.com/profile/holdredial

Duolingo's distractors are automatically generated. It would be completely impractical to manually craft them individually.

For a multiple choice question, I cover up the three answers with my hand and treat it like a "type it yourself" question.

Even if I can't mentally compose a complete answer, I usually get enough fragments of it so that when I look, I can spot the correct answer right away. I read it carefully to confirm, and don't even read the wrong answers.

In a sense, your rant is missing the point. In a classroom setting, you have to make sure students don't cheat. But when you study on your own, there's no point in cheating yourself.

October 4, 2019

https://www.duolingo.com/profile/hanspersson

Perhaps it wouldn't be practical to create them entirely manually, but it would improve their worth a lot if the course creators at least got to mark parts of the sentence that it is useless for the system to change.

If the sentence is "who gave my horse to me?" the exercise is probably pronouns or genitive or something like that. It's useless for the system to put in three random words for "horse" and still that happens very frequently, making that question very easy to answer by just scanning the answers until one has the proper word for "horse" in it as the first basic requirement.

October 4, 2019

https://www.duolingo.com/profile/_M_M_.

"Duolingo's distractors are automatically generated."

I thought I read somewhere that they were automatically picked from incorrect answers that previous users had submitted, which means they match what another user thought was a correct answer (or at least what someone offered as a guess).

October 4, 2019

https://www.duolingo.com/profile/Deliciae

They're automatically generated, but the algorithm that generates them relies on what the course contributors have provided as input.

Basically, it takes one of the alternative translations of the sentence in question, and then replaces one word with another word that was taught in the same skill as that word. If there are no alternative translations for the sentence, then it only replaces a word.

An example from the Norwegian course:

The highlighted option is the correct one. The other two each contain one incorrect word, "skremmende" and "frukt", respectively. The second option is ungrammatical, as the past tense verb "kastet" was replaced by a present participle. The third option is grammatical, but does not match the prompt.

All the other words in the incorrect options can be found among the alternative translations:

Looking closely at the distractors can actually be a good way to expose yourself to synonyms and alternative sentence structures.

October 5, 2019

https://www.duolingo.com/profile/qbeast

Thank you for the detailed explanation. Generating good distractors by hand can be time consuming, and I understand why Duolingo tries to do this via algorithm. That being said, it should not be possible for someone to answer 45 out of 45 questions correctly without reading them. "Alternative sentence structures?" I wasn't even reading the correct sentence structures. If DL is using an algorithm that consistently produces bad questions, either the algorithm should be revised or the question type should be discarded.

October 6, 2019

https://www.duolingo.com/profile/hanspersson

They never have misspellings, though, which makes that sound less likely.

October 4, 2019

https://www.duolingo.com/profile/zyxwvutsrqpo

Something traditional "education" has long forgotten (or never known) is that the goal of education is / should be to teach, not to test. Am I glad that Duolingo isn't using traditional "teaching" methods.

Yeah, you can often easily guess the answers but the alternative of having to spend ages to find minute differences between options is annoying and a waste of time that can be spend on learning more. I'd rather have a question that doesn't truly test me but does teach above one whose goal is mainly to test.

October 4, 2019

https://www.duolingo.com/profile/Y9107g36

Yes but... You're comparing apples to oranges. Duolingo's multiple choice questions aren't tests. They're exercises. They're reading exercises. The point of the exercise is to read. The question is simply a stimulus.

October 5, 2019

https://www.duolingo.com/profile/qbeast

I lost my Spanish owl when the course was updated, so I recently started spending some of my DL time leveling up to Level 1 on all skills so I could get my bird back. The 45 questions I mentioned in the post that is the header to this thread were all questions on the test to level up. None of them were exercises.

October 5, 2019

https://www.duolingo.com/profile/MickLynn

You didn't level up up level 1 What does that mean?

October 5, 2019

https://www.duolingo.com/profile/uphilldweller

Very interesting.

I must admit I sometimes use the 'read the answer first' technique; especially for timed practice.

PS a good link too!

October 4, 2019

https://www.duolingo.com/profile/VoodooSaints

I do the same, because only one answer ever makes sense, with the rest being nonsense. however, I wouldn't want them to go the other extreme, where the sentecnes are almost identical , except for a few letters, and you have to stare at them to figure out what the difference is

October 4, 2019

https://www.duolingo.com/profile/alien_garbage

holy ❤❤❤❤ i just moved from new orleans nice pfp and username

October 7, 2019

https://www.duolingo.com/profile/MissSpells

I confess I often don't read the multiple choice sentences at all, or just scan them for the first or last word which is usually a givaway. Sometimes I just guess till I get it right. I don't know why, as I really pay attention to all the other question types, reading them aloud.. trying to understand the grammar. One issue for me is that the multiple choice never have any audio, so I just skim them.

October 4, 2019

https://www.duolingo.com/profile/Y9107g36

You're learning how to scan in your target language. That's a good thing.

October 5, 2019

https://www.duolingo.com/profile/Kelvin401862

I really wish they all had audio, because then I could re-listen to the audio, and practice pronunciation. And at least get something more out of it.

October 4, 2019

https://www.duolingo.com/profile/Seattle_Scott

If the audio was real people and not TTS I'd agree. I know on memrise with their native videos I'll rewatch them several times.

October 5, 2019

https://www.duolingo.com/profile/Kelikaku

@qbeast

... It shouldn't be possible to do well on
a test without reading the questions. ...

First, these are simple drills. They're
not tests, by any means.

Second, it's a game. Show me the
people who don't learn more or
as much when drilled with
"distractors" versus "trick" questions.

I'll bet the students that get
the "trick" questions do worse.

Good topic. Thanks for posting.

בס״ד

October 4, 2019

https://www.duolingo.com/profile/Ssuppy

I have not noticed any of this and I think Doulingo is doing a great job and multiple choice helps me in many ways when beginning a language! I love multiple choice and I think that duolingo is doing a great job!

October 4, 2019

https://www.duolingo.com/profile/Y9107g36

You're comparing apples to oranges. They're not tests. They're reading exercises.

October 5, 2019

https://www.duolingo.com/profile/arnold12448

I think its important to remember that Duolingo is largely formative assessment, not summative assessment. Also, there are some studies that show that context is very important for adults who are learning a new language.

For these reasons, it might be more helpful to think of Duolingo as more of a learning tool rather than an evaluation tool. Just my take on things as a fellow test developer.

October 5, 2019

https://www.duolingo.com/profile/Songve

Vietnamese multiple choice is challenging and entertaining to me for the most part. Sometimes I know the right answer because some of the words I know do not "fit" and I can deduce the right answer if I am weak in that section. For the most part, I try to interpret all the sentences and that makes good practice. Some of them are absolutely hilarious. They seem to get more challenging the further I get along in the course. I suspect the degree of difficulty may be due to that and the language taken. There are not many multiple choice questions so that and other reasons don't make it seem problematic.

October 4, 2019

https://www.duolingo.com/profile/Betelleo

Hindi multiple-choice is also quite challenging for me even though I'm conversational in Hindi and can read university-level texts and passages albeit rather slowly. It is easy to spot a sentence that makes no sense, but sometimes they just change a vowel or add a dot somewhere and I have to 'guess' the right answer.

Languages that use the Roman alphabet are also somewhat easier on the eyes. Vietnamese is probably an exception to this though. I started the Arabic course last month and managed to identify whole words simply by looking for some specific letter at the start or end of a word. I'm kind of glad that they let you take shortcuts, otherwise it would have put me off Arabic for good.

October 4, 2019

https://www.duolingo.com/profile/Seattle_Scott

While this is all true, I applaud Duo for doing something other than translation exercises. Until now it’s been translate to and from target language and type or repeat what you hear.

To me this is a sign that Duo is expanding its pedagogy, and that’s a good thing.

October 4, 2019

https://www.duolingo.com/profile/The9

A very good point, but you can be sure it is something that will never be taken into consideration by the Duolingo team.

October 4, 2019

https://www.duolingo.com/profile/A_Sesquipedalian

I was considering complaining about this, but I didn’t think anyone would care. It’s ridiculous that approximately half of the time the sentences don’t even make grammatical sense. They could at least have tried using words spelled in a similar manner or with a similar meaning, or at a minimum words of a similar topic to the correct words (e.g. Flower-Bush rather than Flower-Stapler).

October 4, 2019

https://www.duolingo.com/profile/MickLynn

"half of the time the sentences don't even make grammatical sense" Please tell me how fluent your Spanish is.

October 5, 2019

https://www.duolingo.com/profile/A_Sesquipedalian

That’s not a logical argument. I wasn’t specifically talking about Spanish, and I don’t see why you thought that. Actually, I said that because of the sentences that I’ve seen in German-to-English translation. For example, today I’ve seen ‘That we want to look you today.’ and ‘This we work to show you today.’, when the correct answer was ‘We want to show you that today.’. Perhaps you misunderstood me due to the fact I was referencing the original post, which was predominantly based around the multiple choice questions, but if that is so it would be implied that you didn’t even read it in the first place.

October 5, 2019

https://www.duolingo.com/profile/MooO0O

My goal isn't to get questions right and collect xp or anything other than learning a language. Duolingo's goal is to keep us thinking that we are learning a language by making it extremely easy to get correct answers so that we come back often. That's how they make money from sponsors. By making things easy, we will all feel good about ourselves, and we will be fooled into thinking that we are learning things when we are not. But we have to put up with this because it's free and duolingo has many other things good about it that really helps us to learn. I do love Duolingo, but there are things that make me scratch my head.

October 4, 2019

https://www.duolingo.com/profile/Andre989530

Thank you very much for writing this. I'm glad that DL is apparently letting us opt out of multiple choice questions soon. In the French tree they're mostly complete gibberish and a complete waste of my time.

October 4, 2019

https://www.duolingo.com/profile/DragonPolyglot

Multiple choice is my least favorite kind of promt in most contexts because it’s the easiest promt to answer correctly. The only near-garenteed way to get it wrong is by guessing completely randomly with no prior knowledge or elimination process.

October 5, 2019

https://www.duolingo.com/profile/SionedCati

Just to give a different perspective, I’m dyslexic so for me becoming familiar with patterns (such as spellings and accents as well as word order) is crucial. If the wrong answers were only slightly different from the correct answer, I would be learning incorrect patterns, or getting confused and learning nothing. For me a clear difference between the various answers is vital. Remember, as others have said, this is a learning tool not a test!

October 5, 2019

https://www.duolingo.com/profile/Thomas.Heiss

Hi,

I am mainly learning Portuguese on Duolingo.

To be fair:
The multiple-choice (M-C) difficulty was increased somewhere in 2018, especially with the change of "timed practice" (I think I remember that we got 2 code changes in sequence, not in parallel).

IMHO the "distractors" got changed by the developers (for good) and many answers are - at least for EN->PT - more similar with a difference of only 1-2 (often quite small) words.

..(...)..

Well, I don't like multiple-choice at all and would like to turn them OFF completely as I have already progressed in Portuguese for 2,5 years.

On the Memrise web portal you can basically DISABLE them with Cooljingle's "all typing" Tampermonkey userscript so you can 100% focus on L1 English -> L2 target language translations AND 100% typing (including longer phrases and full sentences which require RECALLING and constructing the sentence by applying correct grammar - of course with missing "alternative answers/synonyms").

I can completely understand your rant, as those recent changes have basically ruined my reverse tree experience (much less typing, "wrong" translation direction) for PT->DE and PT->EN:

https://forum.duolingo.com/comment/31856908

In 2018 "timed practice" was (after a code change) IMHO way too difficult (I often scored 0XP, 1-4 / max 6XP)...but in 2019 I have more multiple-choice instead of typing even for completed L1-L3 crown skills when I hit the PRACTICE button.

Maybe you want to repeat your specific test with "timed practice" turned ON and what your observations for "multiple-choice questions and answer options" are?

October 4, 2019

https://www.duolingo.com/profile/Thomas.Heiss

However, I have to agree with you, that you can pretty much "guess" multiple-choice options and obvious missing words or because of their length.

If you see very difficult words included in 1-2 options, then you have basically found the distractor and you are safe to completely eliminate this answer option from your choice.

It is still happening in 2019 to the more or less extend and could be improved.

My opinion:

Three M-C answer options are NOT enough.
On Memrise I often have 6-8 choices (in learning mode - with 6 steps - for new words, not classic reviews)!!

..(...)..

Basically we need a "difficult level" which is fully customizable by a user in the advanced settings (true beginner, upper-beginner, intermediate with 1,0-2,5 years of experience in the target language) including ratios for question types and typing.

And if I have the choice, I will turn M-C off for L1-L3 crown skill reviews, for sure.

Translations back into the native/English language do NOT make much sense, too!

October 4, 2019

https://www.duolingo.com/profile/uippq

It is is so interesting the different ways people use Duo. I for instance never "turned on" multiple choice. I read the question, think of the answer in my head, then sound it out and then look through the M-C answers to find my answer. Of course sometimes I can't remember a word so often can use the answer to remind myself, since recognition is easier then recall. Also, this helps catch preposition errors, which I commonly make. Obviously not as difficult a task as having to type in the answer, but on the plus side I don't get dinged for typos.

October 5, 2019

https://www.duolingo.com/profile/cartergraysons

For Indonesian every multipule choice question is this way. One right awnser and two distracters. I can easily pick out the awnser and it is not challanging for me.

October 4, 2019

https://www.duolingo.com/profile/hanspersson

Another point: if some alternatives have numbers written as digits and some have numbers written with letters, the ones with numbers are always wrong and the one (I've never seen more than one) written with letters is the right one.

October 4, 2019

https://www.duolingo.com/profile/Gazorpazor6

i agree, and i often find myself unintentionally rushing through lessons and not learning much.

edit: what i said probably made me look like a buffoon, so i should have mentioned that i have ADHD

October 5, 2019

https://www.duolingo.com/profile/Gabrielle589496

One should not make the questions so hard that it is making students feel consistently like failures. In the early levels learning is taking place even if the answers appear to be too easy to some. I think Duolingo presents challenges to all. If students have outgrown Duolingo there are more advanced applications and programs elsewhere. I for one have learned multitudes using Duolingo; and appreciate the difficult task of making language learning available to all no matter what level one may be at.

October 6, 2019

https://www.duolingo.com/profile/JillLefrancois

being able to understand all the possible responses is an achievement for me, it wouldn't matter if the answers were always (A), I'm learning 4 sentences every time I get one

October 7, 2019

https://www.duolingo.com/profile/t2hughes

Good post. In addition to being shorter, the 'distractors' often contain words you haven't learned yet -- a dead giveaway. I agree, these could be improved.

October 7, 2019

https://www.duolingo.com/profile/iindego

That is true. Also the fact that the first word of the sentence is always capitalized making it very easy to tell at least the beginning, unless its just like 'a pen.' Other then that its like: Je suis grande apple flower telephone grandma And it makes no sense, which is why I always type the answers insetad of using the word bank (when given the option)

October 4, 2019

https://www.duolingo.com/profile/purplelotus88

Speaking from the business/startup side of things, it's probably not that important because what they worry about is that a stricter test may cause more users to fail to advance in a language. When that happens they are more likely to drop a product if there's too much 'friction' and they aren't getting positive feedback from it. This means less users, fewer subscriptions, which hurts funding, and many other business-y things...you get where I'm going here.

October 5, 2019

https://www.duolingo.com/profile/necronudist85

You're an educator, Duolingo is a company... you've got different goals.

October 5, 2019

https://www.duolingo.com/profile/qbeast

The bottom line here is that DL is striving hard to have its courses meet international standards for A2 and even B1 fluency. Also, hundreds and perhaps thousands of schools are relying on DL to provide quality courses. Neither of these goals will be meet without authentic assessment. I love this website, and I hope that the developers fix the problem. I did this stuff for a living for decades, and I recognize spongy instructional design when I see it.

October 5, 2019

https://www.duolingo.com/profile/MTStroke

"DL is striving hard to have its courses meet international standards for A2 and even B1 fluency."

Now that's really bad. Learning for test conformity is one of the worst learning strategies (although it can be a winning strategy if the goal is not to learn, but to pass a test and get a certificate). Hope Duo won't sacrifice too much for the CEFR.

October 5, 2019

https://www.duolingo.com/profile/davidcwalls

CEFR is primarily a framework to describe language ability. It is really important for anyone learning language for a functional reason. if you want to get a job, talk to your in-laws, visit another country, or move to a new country, it is helpful to be able to know how good your skills should be, to assess your own progress, and to demonstrate your progress.

In my view, adapting to CEFR provides a better way for DL to organize their material, fill-out gaps in coverage, and do a better job of helping learners understand what they can accomplish on the site.

October 5, 2019

https://www.duolingo.com/profile/MTStroke

It is really important for anyone learning language for immigration purposes, yes. It was designed for that. A2 for spouses, B1 for menial temporary workers, B2 for professional workers, C1 for specialists, yes.

But the number of the people learning, say, English (I take English as the language most useful for practical reasons) for migrational purposes is tiny compared to those who learn it to read professional texts. These people are extremely diverse in their background, they are repairmen and professors, managers and doctors, IT professionals and photographers.

Take, for example, an electrician from a Slovakia (or Bulgaria, or Russia). He really needs to read manuals, manufacturers' websites, national and international standards, professional magazines and, probably, books. English is a must, but German, French and Italian are highly important as well -- any moment he can come to the point when the needed manual is available on Legrand's site in French, and another is on Doepke's in German only.

Can he learn these four languages to the level he needs? Not with the CERF-oriented courses. What he needs is between B2-C1, and he has a job and a family, and not much time for the language learning. To achieve even a B2 in one language is probably too much for him. Well, if the employer would set up a worktime course, maybe...

But he can learn all these languages up to the level he needs, if he forgoes much of the CERF-required stuff.

He doesn't need most of the required vocabulary, he doesn't need to memorise, say, German nouns together with their gender, he doesn't need to memorise that in English there is no 'b' sound in 'debt' or 'climb', he doesn't even need to learn that B2-level grammar as well as he would for exams: he can forget when to use English past perfect progressive or German Zustandpassiv, he only needs to recognise and comprehend them when they happen in the text he is reading.

Yes, he needs to do some exercises on the active competences, since these exercises do help the passive. But he doesn't need to do it on exam-ready level.

For him the 'mission accomplished' would be not a B1, but an ability to read the manual looking up the dictionary only once or twice, and 'mission really accomplished' would be not C2, but that state of comprehension when the preferred language for the professional reading becomes not 'my native' or 'my native, then English', but 'the original'.

October 5, 2019

https://www.duolingo.com/profile/Janie490270

I agree with other commenters that these are exercises, not tests. In Spanish, Romanian and Italian I have found this to work fine. But in the French version some of the answers are nonsense (not even proper words.) You only need to glance at them to know which one is correct, you don't have to think at all. No thinking = no learning.

October 5, 2019

https://www.duolingo.com/profile/bibanav

In my opinion, the greatest problem with the sentences is that you actually do not even need to read them. You can simply do visual pattern recognition. I'm currently doing the arabic course and often times the target words appear in two out of the three sentences, one target word in one non-target sentence, the other target word in the other non-target sentence. For fun I let my friend who has no knowledge of arabic some of these multiple choice questions to answer. She immediately recognized the pattern and solved them correctly.

in addition, it would be useful if there were an audio example for every question. For questions in which one should not "cheat" by simply listening instead of reading, the audio could be made available after logging the correct answer. At that point, not only the audio, but also the translation, and in the case of languages with a different alphabet (Korean, Chinese), the phonetic transcription (Pinyin or whatever they may be called) should be displayed.

October 5, 2019

https://www.duolingo.com/profile/MeadowlarkJ

You can hear the audio for a sentence by going to the discussion for that sentence. That’s what I sometimes do if I have an English to Spanish translation. Duo will tell me if my answer was correct, but then I might want to go to the comments and listen to the audio at the top of the page.

October 7, 2019

https://www.duolingo.com/profile/Klgregonis

The multiple guess questions used to be a bit more complicated - they sometimes even had two correct answers and you had to select both. Interestingly, in French, at least in some of the lessons higher up the skill tree, the full sentence questions seem to have been largely replaced by word scrambles, which makes you think about the grammar and the meaning of the words. They also have a different sort of multiple choice - they ask you to click on the meaning of a new word in a sentence from context. Much better, in my opinion.

October 5, 2019

https://www.duolingo.com/profile/MossQuartz

You were "reading" the possible answers? Isn't it more important to have the ability to read? If you were able to read the possible answers then that is good, and if you selected the answer matching the question then that is also good. If DuoLingo were teaching economics or accounting then the selection of the answer would be more important than the mechanics of reading the words. Your rant makes sense for most subjects, but the "feel good" result of being able to make any sense at all of foreign words in grammatically foreign sentences is one more way of feeling our way through learning.

October 6, 2019

https://www.duolingo.com/profile/ml15171031

You can find the right answer without reading the question by eliminating the ones that are not well-formed utterances. To do so you need to know what a well-formed utterance looks like. How is that not a useful skill?

October 6, 2019

https://www.duolingo.com/profile/naya.snow

That's simple to avoid by not looking at the answers until you've formulated your own.

The absurd ones don't bother me as much as the ones that are so similar they're almost homonyms, because I click them without parsing every letter (specifically when it does it for your original language and not the target one - for example I'll often get: Translate this to English "" and in one of the answers it will say "thorough" instead of "through" in the middle of a two-line sentence)

October 7, 2019

https://www.duolingo.com/profile/Shiela559872

I'm learning German. I wondered today whether I'd be correct most of the time by choosing the longest answer! (for the record I love German and love the compound words.)

October 7, 2019

https://www.duolingo.com/profile/FlinkPoyd

I've been using Duolingo for a few days now and I'm beginning to notice a pattern with many (but not all) of the multple choice questions. Apart from the correct answer, there is a distractor and usually a totally absurd sentence which make no sense. The distractor often has a very small change from the correct answer - a wrong verb ending, or a grammatically correct sentence but the wrong verb. Certainly, one must concentrate and read the sentence carefully to be sure your answer is correct - it's easy to misread the translation and choose the wrong sentence. One must also have paid attention to the grammar and words being used, verb endings etc, or, again, one might get the wrong answer. Especially for a language that uses a different writing system or alphabet such as Greek.

In the Romanian course, there are two sentences in one multiple choice question that read as following (it might not be exactly this example, but close): "Ei au un băiat și o fată." Meaning: they have a boy and a girl. The distractor reads: "Ei au un băiat și e fată." Meaning: They have a boy and (he) is a girl. One small letter different and completely different meaning.

These distractors mean that a more able learner will pay close attention so as not to make a mistake, and the obviously wrong nonsense senstences help the weaker learners, as they can more easily detect the wrong answers and are less likely to make mistakes, thus motivating them. In any lesson, there needs to be a balance between being either too easy or too hard, so as to benefit all kinds of learners.

October 7, 2019

https://www.duolingo.com/profile/J3zbo6Nq0VYj

I don't care. Hear me out.

Duolingo can be used by the devoted student to good effect by merely forcing oneself to answer in one's head, or aloud, before selecting a response. This gives you 100% of the value of a question (ie, you attempt it, you get either affirmation or correction). The only other thing you have to do is "cap" the lesson to level 5. You get plenty of repetition in that period to "overlearn" the information.

October 8, 2019

https://www.duolingo.com/profile/Eichho

I don't see a problem making the distractors a little more challenging. For instance, any answer that has a number that is not spelled out is not going to be the correct choice.

October 5, 2019

https://www.duolingo.com/profile/SuperFluff4YOU

Duolingo should just keep both multiple-choice and typed questions. They could add an option to disable multiple-choice questions in the settings, though.

October 6, 2019

https://www.duolingo.com/profile/betsys2003

In French many of the MC distractors are so obviously not correct at a glance because there will be things like J'vais (you never use J' before a word that starts with a consonant) or something super obvious to anyone beyond a very clear beginner.

I agree, they're horrible questions.

October 6, 2019

https://www.duolingo.com/profile/Hope150193

But that’s the point... they’re exercises (not tests) meant to help total beginners get a feel for basic rules of French. If you want more test-like exercises use the stories section where all sentences are viable or the test section. Which I’ve not got round to yet but presumably... er... tests our actual French knowledge. Some courses don’t yet have all these options but in French we have lots of choices. Because no one tree could possibly be ideal for every learner variety makes sense. If the French tree was altered to suit your preference (or even mine) then the forum would be full of total beginners complaining about how the multiple choice questions were stealing their hearts and they could only practice the same lesson over and over and never learn enough to progress.

October 10, 2019

https://www.duolingo.com/profile/b05aplmun.ca

I've found Duolingo very useful for reviewing Spanish (which I studied through fifth-year Spanish in high school), but don't feel that I get to complain about exercises slanted towards rank beginners, which is what the program is geared for.

October 10, 2019

https://www.duolingo.com/profile/Marqwert

Ducking questions in general aren’t that good, not just the multiple choice

October 7, 2019

https://www.duolingo.com/profile/K4ttan

In Duolingo this is even at play in the word bank. True one can opt to use the keyboard, but if I'm on mobile I never do that because typing is hard enough. So I notice when the word bank has reasonable distractors and ridiculous ones. The most reasonable are singular and plurals of the noun you're supposed to use. That has tripped me up. However absurd answers make me double check.

October 7, 2019

https://www.duolingo.com/profile/Jane531211

You can always choose to use the keyboard.

October 9, 2019

https://www.duolingo.com/profile/MarkRusso12

ouch

October 6, 2019

https://www.duolingo.com/profile/Nous-Nous

hmm

October 7, 2019
Learn a language in just 5 minutes a day. For free.