Duolingo is the most popular way to learn languages in the world. Best of all, it's 100% free!

https://www.duolingo.com/Cian8656

Duolingo's focus on usage metrics vs teaching effectiveness

It seems to me like Duolingo is too focused (almost obsessed) with A/B testing absolutely everything and then valuing usage metrics from these tests over actual learning effectiveness.

Don't get me wrong, I think Duolingo is a fantastic resource and gained lots of out of it so far, and have enjoyed myself while doing so. But it just seems lately that the decisions the site is making are based only on how it affects usage statistics, with language learning effectiveness being an afterthought.

I'm worried that Duolingo will only get easier from here on out as it focuses on increasing its audience, instead of focusing on how to take people even further in their language learning journey.

This is just my opinion and I mean no disrespect to all the people that I'm sure are working very hard behind the scenes, and they have created something truly great here. I just hope it doesn't lose sight of its original goal.

If you've read this far, thank you for your time, and I'd be interested to hear your own opinion on the topic.

1 year ago

34 Comments


https://www.duolingo.com/SenorDustin
SenorDustin
  • 15
  • 14
  • 9
  • 8
  • 5
  • 27

As a teacher I have already quit using it as an extra resource. Memrise is more effective at helping my students retain a little extra Spanish.

1 year ago

https://www.duolingo.com/KenBookmye

Duo has repeatedly said they do not attempt to teach to profiency. I can't see how long term retention is even a concern with that stated. It seems more likely the objective is to build membership as large as possible to make the company attractive for buy out. Which is ok, they aren't charging me. I am free to quit when I want unfortunately in my case that's less than six months out. I won't be starting another language regardless of what they add and have finished both the tree and reverse.

1 year ago

https://www.duolingo.com/SenorDustin
SenorDustin
  • 15
  • 14
  • 9
  • 8
  • 5
  • 27

"they do not attempt to teach proficiency" effectiveness is why I used to use it in the classroom. Ease of use and lowering of effectiveness is why I use Memrise as a supplementary material in class. They have to maintain something

1 year ago

https://www.duolingo.com/hughcparker
hughcparker
  • 23
  • 21
  • 21
  • 14
  • 9
  • 7
  • 4
  • 383

They teach the basics, yes, and there's no point learning the basics unless they're retained. They've not only said that they aim for retention, but they've even published the details of how they do so:

1 year ago

https://www.duolingo.com/KenBookmye

We are speaking about two different types of retention. The articles you posted are about language retention and how Duo optimizes learning. I am speaking about retention of people on Duo. When they opted to end immersion they ended the open ended ability to progress indefinitely in a specific language. Assuming someone only wants to learn one language there is currently a point where spaced repetition will suggest they don't need practice for months or years. At which point the single language learner should stop using Duo and rely solely on other sources. ken

1 year ago

https://www.duolingo.com/LeeBrownst1
LeeBrownst1
  • 25
  • 25
  • 25
  • 605

You say "there is currently a point where spaced repetition will suggest they don't need practice for months or years." My experience suggests otherwise. I find that two Spanish lessons per day are barely enough to compensate for skill decay.

When I started using Duolingo a couple years ago, it was possible to raise almost every word to strength level 4. A bit more than a year ago, they made that impossible. Enough words are now presented so rarely that there is always a large pool of words at strength level 1. I think they carefully contrive to prevent any end point from being reached.

1 year ago

https://www.duolingo.com/KenBookmye

When you say two are barely adequate do you mean to maintain your gold status in Duo or for you to personally recall the word meaning? I don't know or practice Spanish but there are Spanish words I can always correctly recall. Uno dos tres sombrero easy stuff but I don't need Duo to remember them

1 year ago

https://www.duolingo.com/LeeBrownst1
LeeBrownst1
  • 25
  • 25
  • 25
  • 605

I keep the Spanish tree gold continuously, and I'm at 58% fluency score, which I believe is maximum, so these measures are no longer helpful.

An alternative measure is fluency score with several decimal digits. You can find this at http://www.duoling.com/users/USERNAME, where you substitute your own user name for USERNAME. Search for the string "fluency_score"; it should be in the third line. But that's not what I use.

Instead, I go to the "Words" tab and click the "Strength" column header to sort the words in increasing order of strength. I count the number of "Page Down" presses until I see the first two-bar score, then for the first three-bar score and finally the first four-bar score. Here are the measures I made for the past 7 days, on which I did two "Strengthen Skills" lessons each day:

35 36 36; 35 36 36; 36 37 37; 36 36 37; 36 36 37; 36 36 37; 36 36 37; 36 36 36;

It's just barely holding steady. At about 20 words per page, I have about 36 * 20 = 720 words at level 1; that's 1/4 of the 2834 words in the word list.

1 year ago

https://www.duolingo.com/hughcparker
hughcparker
  • 23
  • 21
  • 21
  • 14
  • 9
  • 7
  • 4
  • 383

How else do you find out what's effective, if not by testing it?

...but you have a point about retention vs effectiveness. That's always going to be a trade-off - on the one hand, people can't learn if they've left, but on the other, yes, sometimes something is lost in making the site more friendly to every possible student.

1 year ago

https://www.duolingo.com/Cian8656

True, I agree about the value of testing. I guess I just mean they seem to be testing the wrong thing

1 year ago

https://www.duolingo.com/Jack.Elliot
Jack.Elliot
  • 25
  • 25
  • 25
  • 25
  • 21
  • 13
  • 10
  • 10
  • 9
  • 9
  • 8
  • 5
  • 3
  • 2
  • 2

https://www.duolingo.com/Cian8656

To quote from what you linked:

CEO: "We're working on hundreds of features, some of which you may never see because they won't increase our usage metrics when we test them on a small fraction of the users. But you'll see the ones that increase usage metrics, and hopefully you'll like them :)"

Commenter: "I get that "usage metrics" are vitally important, but to my mind, the features that help people to better learn and retain their language skills should be equally, if not more important than "usage metrics". This is a language-learning site, after all."

CEO: "Sorry, I should have been more clear: we definitely look at learning outcomes as well."


It seems like usage metrics are indeed king and learning outcomes are an afterthought

1 year ago

https://www.duolingo.com/Luscinda
Luscinda
  • 25
  • 25
  • 25
  • 15
  • 11
  • 10
  • 8
  • 7
  • 6
  • 6
  • 5
  • 4
  • 3
  • 3
  • 3
  • 2

So the people who point out that the thing is full of errors or dodgy pedagogy (because they have enough knowledge to spot when they are being told something that isn't quite right or when the computer is rejecting good [insert language] in favour of slang or nonsense) start to fall away. Meanwhile, the people who just submit the computer's desired answer and who think it's a game stick around, supplying the fixed answer to accrue points. They're not learning and not bothered about that fact. Your metrics say that a higher proportion of people are giving you the answer your computer wants. You conclude, erroneously, that more people have improved in their command of the language.

The only way to gain real information ('metrics', if you will) is to test people on their real acquired command of the language objectively, with a real examiner/native speakers - not to measure whether they can copy-paste a bad answer where that's what the computer demands. And in my experience, that is what some of these 'courses' demand.

1 year ago

https://www.duolingo.com/ove_sundberg
ove_sundberg
  • 25
  • 25
  • 25
  • 23
  • 1326

The crux of the problem with retention is this: the vast majority of users have no clue as to how much a given tool is helping them until years have passed.

So... what happens when someone puts their trust in Duolingo, and sticks at it for 17, or 18 months - only to realize they haven't learned much at all (and ought to have gone with Memrise or Lingvist). If you go by retention metrics, that still looks like a big success. After all, if a user sticks around for a year and a half, that user must be pretty happy! Of course, in reality the user has wasted hundreds of hours of her life on an ineffective tool.

You can build a business around that (i.e.: luring noobs to a system that doesn't work well, and hoping they don't notice), but - yech! - it sounds like a soul-destroying work environment.

I kind of think that obsessing over "user retention" is a refuge for the incompetent, or for CEOs who care more about their VC money than their product. The alternative is to hire smart people, and let them loose to dream up some great stuff. The chances of failing completely may be far greater, but at least you'll also have a shot at creating something truly wonderful.

1 year ago

https://www.duolingo.com/ove_sundberg
ove_sundberg
  • 25
  • 25
  • 25
  • 23
  • 1326

I also believe that most of Duolingo's shortcomings are attempts to ease server load (i.e.: by favouring static cache-friendly content).

So it's a vicious cycle. The faster you attract new users, the harder it will be to keep up with load. And that puts further pressure on your team to avoid dynamic content (such as better tailored Spaced Repetition).

In other words, the more people like the site, the less the site is able to function for its alleged purpose: to teach you a second language.

The inevitable conclusion of this is that a decade from now, Duolingo will look a lot like FarmVille.

1 year ago

https://www.duolingo.com/annika_a
annika_a
  • 22
  • 17
  • 17
  • 16
  • 11
  • 11
  • 5
  • 2
  • 2
  • 2

...assuming it actually still exists. They're hoping to break even financially some time this year, so there's that's, but will they still be providing a relevant and attractive product?

1 year ago

https://www.duolingo.com/ove_sundberg
ove_sundberg
  • 25
  • 25
  • 25
  • 23
  • 1326

@Kocksedan "Relevant and attractive" to people who don't want to be quizzed on the words they already know. I think that's pretty much everyone on this site, regardless of number of languages, or time per month spent studying. The only difference is that the more time you spend here, the surer you are that the algorithm doesn't do an effective job of choosing review vocabulary.

1 year ago

https://www.duolingo.com/Knocksedan

Relevant and attractive to hobbyists who want to "learn" a dozen or more languages, or to the vast majority of people who only want some help to learn one?

1 year ago

https://www.duolingo.com/Queen.of.Candy

I agree.

1 year ago

https://www.duolingo.com/EvelynOlson0

I totally agree!

1 year ago

https://www.duolingo.com/Deyan161
Deyan161
  • 20
  • 17
  • 14
  • 8
  • 7
  • 6
  • 2
  • 2
  • 862

This certainly seems to be a trend now.

1 year ago

https://www.duolingo.com/Katzenperson
Katzenperson
  • 20
  • 13
  • 11
  • 10
  • 9
  • 9
  • 6
  • 5
  • 5
  • 4
  • 3
  • 3
  • 2

Quick question, by "usage metrics", are they referring simply to the number of people who use the site on a regular basis, or does it mean something more?

1 year ago

https://www.duolingo.com/Cian8656

From what I understand, it only means using the site

1 year ago

https://www.duolingo.com/Knocksedan

"Usage metrics" is more than simply the number of people who use the site, it also measures things like what parts of the site people use, how long they use them for, how often they use the site, whether there are differences between identifiable groups of users (do night-owls make different progress than early-birds, for instance), highlighting particular skills that cause an issue for users, and, when an A/B test is in place, how the behavior of the A and B groups differs.

It's a far more complex area than simply counting how many users log in each day.

1 year ago

https://www.duolingo.com/Knocksedan

if 100 people start a course on Duolingo, which is the best measure of "teaching effectiveness"? 1) 99 of them drop out, but the one that keeps going becomes perfectly fluent, 2) 90 of them drop out, but the 10 that remain gain basic fluency 3) 50 of them drop out, but the 50 that remain gain a basic vocabulary and enough confidence in their ability in the language to be able to move on to further and more advanced studies.

Obviously these numbers are just pulled from the air, but before you question Duolingo's methods, you need to clearly understand and state what you mean by "teaching effectiveness" - and the chances are that Duolingo has a far better understanding of what that really means, and the data to back it up, than you do.

1 year ago

https://www.duolingo.com/chirelchirel
chirelchirel
  • 21
  • 21
  • 18
  • 12
  • 11
  • 10
  • 9
  • 4
  • 7

In your example the problem is that the fifty who learn the basics don't have the language skills (or even language learning skills) required to be independent learners (levels B1 and B2 on European http://webcgi.oulu.fi/kielikeskus/o/assessment_grid_B2C1.html) so they'll hit a wall if the material on Duolingo is too easy and ends too soon. It's not easy to find good material of suitable level (challenging, but not too challenging) on your own. So making Duolingo easier and easier might be a disservice to all the regular learners who aren't used to learning languages and only want to learn that one language. On the other hand, the language hobbyists like me, will manage and just stubbornly plough through the internet in order to find what we need.

Another thing: I think the website and the app work well together. I use the site to learn and the app to repeat what I have learned. I take responsibility over my learning by adjusting the difficulty myself (an opportunity to actually adjust the difficulty would be nice, but apparently it's not necessary for me). Here's what I do: at first I botch every sentence that has a new word or a word I think I don't know well enough. I rarely guess. This is to tell to algorithm to give me the same sentence again. If a word seem difficult, I hover over it, again for the same reason. In language learning, repetition is the key. I read the comments for every sentence. I repeat every lesson twice in a row or if it was really difficult I repeat it three or four times. I strengthen a skill the next day and the day after that and then I strengthen it on the app. If a skill changes color I strengthen it until it's golden + one more time. Like I said, repetition is the key and I think Duolingo doesn't make us repeat the skills often enough. With enough repetition, there's also more translations into the target language. When using the app, at first I rely on all the things that might help me, then I stop using them one by one, expecting more from myself. I don't need the site or the app to do that. I can try to translate the sentence in my mind before I look at the words. This way the app is a great tool, too. The problem is, a lot of people have no idea they need to challenge themselves or how to do it, so they will not advance.

1 year ago

https://www.duolingo.com/Cian8656

On the app, the first word of the sentence is capitalised. They tried removing this capitalisation as it's an obvious hint that it's the first word of a sentence, but usage metrics went down so they kept it.

In my opinion, that's a hit to usage metrics that you should be willing to take if you want to say you're in the business of making a language learning app, rather than a game that just happens to have a background in languages

1 year ago

https://www.duolingo.com/hughcparker
hughcparker
  • 23
  • 21
  • 21
  • 14
  • 9
  • 7
  • 4
  • 383

Really? I hadn't heard that. Can you post a link to where you saw that?

1 year ago

https://www.duolingo.com/Cian8656

Link here, quote below:

> I hate that too! Unfortunately whenever we try removing it, more people complain and our metrics go down :/ Sometimes you just can't win.

--Luis (CEO of Duolingo)

1 year ago

https://www.duolingo.com/ove_sundberg
ove_sundberg
  • 25
  • 25
  • 25
  • 23
  • 1326

Ha, that's rather myopic (and I'm using "rather myopic" as a euphemism for "profoundly idiotic").

Why not take the same attitude further? For example:

Question:

I notice that you changed the website this year so that when I switch to my German course, it's a web gallery with nothing except photos of kittens. I hate that I can no longer use the site to learn actual vocabulary.

Answer:

I hate that too! Unfortunately whenever we try removing it, more people complain and our metrics go down. Our metrics show that more people enjoy looking at kittens than studying. Therefore, a German course with 100% cat pictures helps more people learn German.

1 year ago

https://www.duolingo.com/ove_sundberg
ove_sundberg
  • 25
  • 25
  • 25
  • 23
  • 1326

I don't have a strong opinion as to whether capitalization is better or worse for the learner.

I do have a strong opinion as to how not to find out: user retention metrics!

1 year ago

https://www.duolingo.com/AmineHadji1
AmineHadji1
  • 21
  • 19
  • 19
  • 14
  • 13
  • 12
  • 7

You are just pointing out the obvious. I don't understand what is your point. Well, I do understand the message, but I don't understand the purpose.

Duolingo is a compagny and its main purpose is to get the most people. If they see that "getting things easier" will allow more people to be interested, they will do so. You can think it's "bad", but that's how companies work all over the world.

Besides, I have to point out something. I teach languages as a freelancer and I notice something among learners: they DON'T want to learn. When things become difficult, they drop out. That's the exact same problem with Duolingo, if things are not easy enough, people drop out. So, as said before between 99 drops and 1 user with a B2 level or 25 drops and 75 users with an A2 level, Duolingo made its choice. You can be disappointed by that, I am, but at the end of the day, Duolingo is free. Do you know how much it costs to reach an A2 level in a private institute? Well, thanks to Duolingo, I could skip all of that and enter directly at the level A2.2 in a Dutch language institute. Of course, I would have prefered Duolingo to teach more and so... However, I understand why they are doing what they are doing and I don't think I can blame them for that.

1 year ago

https://www.duolingo.com/chirelchirel
chirelchirel
  • 21
  • 21
  • 18
  • 12
  • 11
  • 10
  • 9
  • 4
  • 7

The problem is that it seems like it's not working as they intended. They do get more users, but then the site can't handle all the traffic, so what's the point of trying to lure in more and more users all the time?

And I do think Duolingo is throwing away a valuable asset by making things too easy for the advanced learners or language hobbyists. These people have been helping others a lot in the forums if thy feel like they are not getting anything from the site anymore, who's going to do all the explaining when the new learners get stuck somewhere?

1 year ago

https://www.duolingo.com/AmineHadji1
AmineHadji1
  • 21
  • 19
  • 19
  • 14
  • 13
  • 12
  • 7

I can't answer your first relevant remark. I am not a CEO. However, I worked in marketing fields, and it appears that business people are more attracted by numbers than by commitment. More users mean more users who are potentially attracted to the ads of Duolingo's app or wanting to pay not to lose their streak.

Anyway, your second question, while relevant, is exactly what I answered in my post. Do you really think Duolingo is not aware of what we will potentially lose? It's just that they made their choice. They chose quantity over quality (an aberrant choice for an individual, but a normal choice for a company).

1 year ago