The Tenets of A/B Testing from Duolingo’s Master Growth Hacker
This article interview of Gina Gotthilf (VP of Growth at language education platform Duolingo) was full of interesting information about Duolingo's A/B tests. They talk about multiple tests. So, I just picked the one to do with Streaks since I recently wrote about streaks.
A/B TEST #2: Streaks
THE QUESTION: Duolingo, like many other apps, wants to encourage users to develop a regular behavior — in this case, completing a language lesson every day. In fact, the product essentially doesn’t work unless a user visits regularly. “Over my life, I’ve studied four languages — and can speak five. One of my takeaways is that there’s no way one can learn a language on just Saturdays or Sundays,” says Gotthilf. “We need to get people to do it every day or every other day for languages to stick.”
But that’s easier said than done. “With all the MOOCs and educational platforms out there, there are a lot of ways to learn awesome things online,” says Gotthilf. “People get really excited about signing up, but those services have abysmal retention rates. Because getting people to come back is really hard.”
That’s why the Duolingo team looked for inspiration in an industry outside the education space: the world of gaming. “In order to consider Duolingo as a game, we basically had to find ways to make things more fun,” says Gotthilf. “We’d look for examples from games that we really liked and say, ‘If we were to transpose this into Duolingo, what would that look like?’”
For Duolingo, streaks — keeping track of how many days in a row a user has used the product — have been a game changer. And a metric that Gotthilf and her team continues to focus on. “It's like in a video game, where you have to do something every day or you lose your rank,” she says.
Streaks reward — and gamify — persistence. THE TEST: The introduction of the Duolingo streak marked a major turning point in how users interacted with the app — and how Gotthilf and her team could respond.
For starters, a streak requires an established goal, so the team added goal-setting to the account setup process. “You determine how much Duolingo you want to do per day,” says Gotthilf. “If you don't hit your goal, you can't keep your streak. It’s not just about opening the app every day.”
Otherwise put, where is this change in your funnel? In the case of Duolingo, for example, a change to the sign-up page — which most users see — touches more people than a change to a page buried deep in a language lesson.
This sentence makes me sad, but it's a good explanation of what's going on here. Pages buried deep in a language lesson - unimportant. Signup page, notifications and "Your're great" messages - important. All this is so far from the needs of an interested, self-determined learner.
Thanks! I found it cool to read about what is happening in the background with all of these A/B tests.
(I'm hoping I didn't do a repeat of posting this. I didn't see that anyone else had posted it when I scrolled through the forums. But, I'm also a bit distracted. (At a gathering of friends.))
I wish I knew.
What is A/B test 4?
I am mostly focused on monitoring forum dynamics. So, I mostly Duo through the Desktop (save a small handful of lessons via the Android app, like, less than 10 in a year). As it is, I am just starting to experiment with the apps again because of Japanese and I want to sample the new changes, like badges and stuff. If you or anyone else finds further information about A/B tests though, please do share! :)
Do you know what's up with the A/B tests for the profile page and activity stream? I've been gone awhile and got an email from Duolingo saying an old pal wrote on my wall asking how things were going for me in life. (VStarTraveler, if you see this, I have no way to reply to you!) I can't see my wall, although it appears other people still can. Are they removing social aspects like that? I didn't see anything about that in the article.
Kinda a pity to come back to.
The modifications to the profile page and removal of the activity stream are unrelated to A/B tests.
This is a forum post about the removal of the activity tab, I haven't clicked it because I suspect it has > 600 comments and I'm on a mobile device, but it probably has information you're asking about: https://www.duolingo.com/comment/22226909
Duolingo have said:
What will replace the Activity tab?
We have nothing concrete to share... yet. But we do plan to replace this social feature with something better!
This was in an article about the website rewrite, where they say the removal of the Activity tab was "unrelated to the web rewrite". The above link is from there too. https://support.duolingo.com/hc/en-us/articles/115002969903-Duolingo-s-New-Website
Until recently (maybe about three weeks ago?), I think some profile pages were still allowing others to post messages to you, but neither the person sending the message nor the person receiving it could see it on the profile page or activity stream; it was only visible by email or the notification bell.
Thank you so much! This is exactly what I was looking for. My friend must have wrote me before this all was removed then.
I hope they reconsider some of this; I'd say the community of Duolingo is one of it's best points. If I were in their place, I'd be working to make that even better, not limit it.