https://www.duolingo.com/profile/Usagiboy7

The Tenets of A/B Testing from Duolingo’s Master Growth Hacker

This article interview of Gina Gotthilf (VP of Growth at language education platform Duolingo) was full of interesting information about Duolingo's A/B tests. They talk about multiple tests. So, I just picked the one to do with Streaks since I recently wrote about streaks.

A/B TEST #2: Streaks

THE QUESTION: Duolingo, like many other apps, wants to encourage users to develop a regular behavior — in this case, completing a language lesson every day. In fact, the product essentially doesn’t work unless a user visits regularly. “Over my life, I’ve studied four languages — and can speak five. One of my takeaways is that there’s no way one can learn a language on just Saturdays or Sundays,” says Gotthilf. “We need to get people to do it every day or every other day for languages to stick.”

But that’s easier said than done. “With all the MOOCs and educational platforms out there, there are a lot of ways to learn awesome things online,” says Gotthilf. “People get really excited about signing up, but those services have abysmal retention rates. Because getting people to come back is really hard.”

That’s why the Duolingo team looked for inspiration in an industry outside the education space: the world of gaming. “In order to consider Duolingo as a game, we basically had to find ways to make things more fun,” says Gotthilf. “We’d look for examples from games that we really liked and say, ‘If we were to transpose this into Duolingo, what would that look like?’”

For Duolingo, streaks — keeping track of how many days in a row a user has used the product — have been a game changer. And a metric that Gotthilf and her team continues to focus on. “It's like in a video game, where you have to do something every day or you lose your rank,” she says.

Streaks reward — and gamify — persistence. THE TEST: The introduction of the Duolingo streak marked a major turning point in how users interacted with the app — and how Gotthilf and her team could respond.

For starters, a streak requires an established goal, so the team added goal-setting to the account setup process. “You determine how much Duolingo you want to do per day,” says Gotthilf. “If you don't hit your goal, you can't keep your streak. It’s not just about opening the app every day.”

Click to read the full article

July 20, 2017

15 Comments

Sorted by top post

https://www.duolingo.com/profile/lala2178

Is it just me or does Usagiboy7 always post the most interesting posts here on Duolingo? I truly enjoy reading them. You are a great moderator and truly help a lot, thank you, sincerely.

July 20, 2017

https://www.duolingo.com/profile/Portofan

Otherwise put, where is this change in your funnel? In the case of Duolingo, for example, a change to the sign-up page — which most users see — touches more people than a change to a page buried deep in a language lesson.

This sentence makes me sad, but it's a good explanation of what's going on here. Pages buried deep in a language lesson - unimportant. Signup page, notifications and "Your're great" messages - important. All this is so far from the needs of an interested, self-determined learner.

July 20, 2017

https://www.duolingo.com/profile/ginafrombrazil

Happy you guys thought it was interesting! Our team is constantly trying to find ways to motivate people to learn more and more.

July 20, 2017

https://www.duolingo.com/profile/Speir_

That was an interesting article to read, thanks for sharing with the community!

July 20, 2017

https://www.duolingo.com/profile/Usagiboy7

Thanks! I found it cool to read about what is happening in the background with all of these A/B tests.

(I'm hoping I didn't do a repeat of posting this. I didn't see that anyone else had posted it when I scrolled through the forums. But, I'm also a bit distracted. (At a gathering of friends.))

July 20, 2017

https://www.duolingo.com/profile/Speir_

I think so too, it's neat to see (as you stated) what's currently happening with the A/B tests Duolingo's running.

As for repeating the discussion, I haven't seen any discussion threads on this topic, or at least not very recently.

July 20, 2017

https://www.duolingo.com/profile/Strandfloh

Thanks for sharing!

I'm just curious: How does the A/B test 4 (in app coach) get along with the gems/health feature? Those approaches seem to contradict each other completely.

July 20, 2017

https://www.duolingo.com/profile/Usagiboy7

I wish I knew.

What is A/B test 4?

I am mostly focused on monitoring forum dynamics. So, I mostly Duo through the Desktop (save a small handful of lessons via the Android app, like, less than 10 in a year). As it is, I am just starting to experiment with the apps again because of Japanese and I want to sample the new changes, like badges and stuff. If you or anyone else finds further information about A/B tests though, please do share! :)

July 20, 2017

https://www.duolingo.com/profile/Strandfloh

Sorry if I wasn't clear enough. The number refers to the article, and I put the topic in braces, so it's about the in-app coach.

I also use only the desktop version, and I certainly won't change that as long as the health feature stays.

July 20, 2017

https://www.duolingo.com/profile/RandomTed

What is a DAU?

July 20, 2017

https://www.duolingo.com/profile/Strandfloh

Daily Active User.

July 20, 2017

https://www.duolingo.com/profile/Aliciaftw

Do you know what's up with the A/B tests for the profile page and activity stream? I've been gone awhile and got an email from Duolingo saying an old pal wrote on my wall asking how things were going for me in life. (VStarTraveler, if you see this, I have no way to reply to you!) I can't see my wall, although it appears other people still can. Are they removing social aspects like that? I didn't see anything about that in the article.

Kinda a pity to come back to.

July 20, 2017

https://www.duolingo.com/profile/SprightBark

The modifications to the profile page and removal of the activity stream are unrelated to A/B tests.

This is a forum post about the removal of the activity tab, I haven't clicked it because I suspect it has > 600 comments and I'm on a mobile device, but it probably has information you're asking about: https://www.duolingo.com/comment/22226909

Duolingo have said:

What will replace the Activity tab?
We have nothing concrete to share... yet. But we do plan to replace this social feature with something better!

This was in an article about the website rewrite, where they say the removal of the Activity tab was "unrelated to the web rewrite". The above link is from there too. https://support.duolingo.com/hc/en-us/articles/115002969903-Duolingo-s-New-Website

Until recently (maybe about three weeks ago?), I think some profile pages were still allowing others to post messages to you, but neither the person sending the message nor the person receiving it could see it on the profile page or activity stream; it was only visible by email or the notification bell.

July 20, 2017

https://www.duolingo.com/profile/Aliciaftw

Thank you so much! This is exactly what I was looking for. My friend must have wrote me before this all was removed then.

I hope they reconsider some of this; I'd say the community of Duolingo is one of it's best points. If I were in their place, I'd be working to make that even better, not limit it.

July 20, 2017

https://www.duolingo.com/profile/Langmut

If you want to support communication coming back you might want to add upvotes in this discussion.

July 20, 2017
Learn a language in just 5 minutes a day. For free.