https://www.duolingo.com/delapouite

A/B Testing : how to discover your groups.

Hello.

It appears that discussions about A/B testing are all the rage on the forum nowadays, especially since the recent introduction of the Coach feature.

People often have no clue of which side of the fence they are. So here's a simple and useful tips to get the answer. It relies on the same info used by the site itself and the different userscripts shared by other users which enhance the interface.

In your web browser, press F12 to get access to the dev console. Then simply type the following command to inspect the variable :

duo.user.attributes.ab_options

You should then get a list like this :

android_never_disable_mic_experiment: true

android_tab_bar_experiment: false

coach_track_experiment: "coach"

coach_web_experiment: true

dictionary_phase2_experiment: false

italian_tts_experiment: "Carla"

etc...

Nothing new or really secret, but could be useful to some of you.

Cheers.

October 15, 2014

126 Comments


https://www.duolingo.com/iron_bun

Love the job advert in the code.

October 15, 2014

https://www.duolingo.com/5string

You see that a lot. Because only coding nerds usually bother to check out a page's source code, it becomes a hidden but nicely focussed place to pop in a job advert. Google does that all of the time, and Apple a few times.

October 16, 2014

https://www.duolingo.com/Yoruji

I liked the following part as well "// I am unsure under what conditions this would occur. if(!_.isFunction(mixpanel.register)) { "

October 16, 2014

https://www.duolingo.com/makahiwitk

Thanks. It's nice to see that my list is a series of 'falses' and 'controls'. Apart from two, which really stand out, and are the group for longer sentence lengths and more listening challenges - yeh, lucky me. The least they could have done regarding the listening challenges is to also give me Carla, but no, I've still got old gasping for breath francesca16.

October 15, 2014

https://www.duolingo.com/TheBryce

Francesca16's voice makes learning Italian awkward.

"Una gasp donna gasp mangia gasp la gasp mela."

October 16, 2014

https://www.duolingo.com/tariqnisarahmed

Good thing Italian is not a pressing language for me -- I am also Francesca bound.

October 16, 2014

https://www.duolingo.com/makahiwitk

Unfortunately, she makes me cringe, as I do know what Italian should sound like. It's just that having more listening challenges makes it more 'in your face' or perhaps that should be 'more in your ear'. Still, free is free, and you get what you get. :-)

October 16, 2014

https://www.duolingo.com/Klange05

whose francesca

February 29, 2016

https://www.duolingo.com/Geo430

I have no Italian tree yet I have Carla. It's like I have a lot of a currency and nothing to spend it on. But that might be a bit of a realistic analogy for Duo users.

October 17, 2014

https://www.duolingo.com/makahiwitk

Go figure! No tree, but you have Carla. Then the sorting of users into the various groups is akin to Lotto - it's just the luck of the draw! :-)

October 17, 2014

https://www.duolingo.com/Jonathan--

Huh, I just changed coach_web_experiment from "false" to "true," went to the homepage, and there was the coach. So at least in some cases, if you want to see how the other half lives, you just need to write yourself into the other half.

October 16, 2014

https://www.duolingo.com/tariqnisarahmed

If you change a parameter on one browser on one device, does it affect every other session? Try changing one of your mobile, android, iOS settings from your desktop/laptop. Then see if it changed the experience on your mobile device... Answer: Tried it and nothing changed on the mobile device. So any changes only effect the browser where you made changes.

If the changes you make in your browser only affect that session, meaning that the effect is local, then I wonder how much that would mess up the data collected by Duolingo? I mean we have proven that the coach can be enabled (Jonathan) and disabled (AlexisLinguist, who, admirably perhaps, turned it back on again, too).

What if all the users whom DL designated as Coached simply turned it off. Then DL's Coach data would be skewed unless they could perceive whether a browser F12 hack had been performed...

And to go one step further, there are a lot of users using tweaked browsers already. They change the DL colors, etc. Some bring back the coach. Some don't. Unless DL has a way of monitoring all those hacks, then all (or part) of the test data for those users may be bunk.

Yup, if DL has no way to gauge what tweaks users have made, or even whether users have made changes, then every person who makes a change is like a gremlin run amok in the DL labs.

October 16, 2014

https://www.duolingo.com/AlexisLinguist

(AlexisLinguist, who, admirably perhaps, turned it back on again, too)

Please don't remind me. It's right there, yet I cannot touch it. So close, yet so far. I may need, a Snickers bar.

Yup, if DL has no way to gauge what tweaks users have made, or even whether users have made changes, then every person who makes a change is like a gremlin run amok in the DL labs.

If we're talking about real gremlins here, then I'm Spike.

October 16, 2014

https://www.duolingo.com/tariqnisarahmed

But you're a reformed Spike. You tasted the food after dark but then went back to being a Mogwai.

October 16, 2014

https://www.duolingo.com/AlexisLinguist

Who said that? Maybe I like being bad. :P

October 16, 2014

https://www.duolingo.com/shalfyard

"then every person who makes a change is like a gremlin run amok in the DL labs." Not really... It's all very local to that instance. The second they navigate somewhere or refresh (get a page load in general) the settings will get refreshed back to what the server has set for you.

October 16, 2014

https://www.duolingo.com/tariqnisarahmed

My point being that DL thinks the person is using environment A (the sum of a bajillion tests being run simultaneously) while in fact any usage of the site reflects environment G (what the Gremlin has wrought on his own screen).

"Amok" like the rest of what I wrote being colorful language.

October 16, 2014

https://www.duolingo.com/feanarosurion

Everyone might be a gremlin, but I think a lot of the people who might actually go ahead and decide to make these changes are probably devoted enough that they're within the "firmly retained" camp for their precious metrics. Plus, the values probably aren't changing on Duo's end. So it probably doesn't matter if a few people change a setting or two. All hail the divine infallible metrics!

October 16, 2014

https://www.duolingo.com/shalfyard

The settings are all javascript based and get set by the server when you visit a page. Thus they get reset to the server instance the second you navigate or refresh... Which is after you click strengthen, answer a question, etc. Essentially you are getting a glimpse into what you could have but not actually getting that experience to mess up their metrics.

October 16, 2014

https://www.duolingo.com/EvelynOlson0

HOW!?

February 22, 2016

https://www.duolingo.com/tariqnisarahmed

Whoa. Now this is kinda like another movie... (in the Will Ferrell movie, he really never initiated anything, did he?)

Yup, I think Jonathan just chose the blue pill. Someone keep a lookout for Morpheus (not to mention Agents) and cue the techno music because Duolingo has become the Matrix.

October 16, 2014

https://www.duolingo.com/makahiwitk

I've changed some of my settings but I haven't noticed any changes. :-(

October 16, 2014

https://www.duolingo.com/feanarosurion

This is really telling. The number of tests they're carrying out really annoys me. The whole idea of quietly AB testing the site (or anywhere) annoys me. They should tell us what they're working on, implement it for everyone, and let us give feedback. Plus be more engaged in the community in general. </rant>

October 16, 2014

https://www.duolingo.com/writchie4

I'm not sure I understand why this annoys you. A/B testing is a good thing, as much as it stinks to be in the control ("reject") group. It's a proven tool for determining which features have a positive impact on the site, without relying on wishy-washy (and potentially contradictory) written user feedback. As for the number of concurrent tests, that's a function of the size of Duolingo's user base. Having millions of users allows them to test multiple features at once. How is that a bad thing? If anything, it allows them to improve the site more rapidly.

I'm with you on the staff-community engagement point though. It wouldn't kill them to be a little more active on the forums.

October 16, 2014

https://www.duolingo.com/tariqnisarahmed

Agreed. Also, maybe these are all the experiments that have ever been run. Or ever been run since a given user joined. Does not mean every test is in progress at the same time.

October 16, 2014

https://www.duolingo.com/AlexisLinguist

I think these are all currently in progress. The one that enabled the "Start button" for new lessons is no longer there, so old tests disappear from the code.

October 16, 2014

https://www.duolingo.com/tariqnisarahmed

Wow. Then that is a whole lot of testing going on... Think they pay attention to effects of one experiment on the results of another?

October 16, 2014

https://www.duolingo.com/AlexisLinguist

"Think they pay attention to effects of one experiment on the results of another?"

I'd hope so. There are more than several experience "configurations" at the moment, so I'd hope they would be taking that into account. Otherwise, seems like that taint the results, and the whole thing would be a waste of time.

I mean, I already slowed down my usage (learning) when the new design rolled out, and now the Coach...I just want the old design (ha, it's only been two-three weeks now) back.

October 16, 2014

https://www.duolingo.com/tariqnisarahmed

Hey AlexisLinguist, did you try reversing what Jonathan did?

"Huh, I just changed coach_web_experiment from "false" to "true," went to the homepage, and there was the coach. So at least in some cases, if you want to see how the other half lives, you just need to write yourself into the other half."

Maybe you could dispatch the Coach.

October 16, 2014

https://www.duolingo.com/AlexisLinguist

It works. :) For the sake of the test, though, I'll make it true again. Oh, the beautiful progress bar. How I miss you so.

October 16, 2014

https://www.duolingo.com/Lrtward

If they tell us what they're working on, that could skew the results.

October 16, 2014

https://www.duolingo.com/AlexisLinguist

Well, we already know, so all results are effectively skewed. ;) It's interesting that the tests would be "out in the open" like this. Wouldn't keeping them tucked away so users couldn't fiddle with them be better? It's almost like they want us to find them. Shoo, if it's an open invitation...

October 16, 2014

https://www.duolingo.com/Lrtward

Hey, don't come at me with logic, man! That's cheating! It's not fair! Ima down vote all your comments!!!

Good point. We do already know, don't we? Except, fortunately, I have the memory of an earthworm so I already forgot what groups I'm in. Go me.

Not sure they can be tucked away where a web development tool can't fiddle with them. That's the nature of web pages vs. compiled apps, I fear.

October 16, 2014

https://www.duolingo.com/AlexisLinguist

*Woman. ;)

Good for you. Unfortunately, I'm semi-eidetic. :P

Lord, how I want to go back to the progress bar...I don't want to play with the cards I was dealt. But, I guess if I never cheated at Uno ;) , I'll tough it out.

October 16, 2014

https://www.duolingo.com/superdaisy

Interesting!

Here are some of mine:

coach_track_experiment: "trackless" (...but I have Coach on, so I wonder what this means)

enable_web_speech_experiment_v3: "reverse_speak" (don't use the speech component online, but when I did I was asked to translate English to Spanish and then speak)

ios_no_social_signup_experiment: true

sg_tap_challenge_frequency: "more_tap" (...tap?)

web_remove_real_world_practice: true (I just noticed my Real World Practice button is gone from underneath my Strengthen Skills button)

web_give_up_button_experiment: false (interesting, there's a "give up" button? I just refresh...)

October 16, 2014

https://www.duolingo.com/shalfyard

Tap challenges are likely for the app. They let you pick from a set of words to build your translations... Makes it crazy easy to be honest.

October 16, 2014

https://www.duolingo.com/superdaisy

Oh, that makes sense.

Yes, they are pretty easy, though it's nice to practice word order and structure without worrying too much about vocabulary. If the non-answers were more similar (different conjugations, for instance) then the questions would be more challenging.

October 17, 2014

https://www.duolingo.com/makahiwitk

Thanks, so it doesn't apply to me and my laptop. I'd love a tap (dance) challenge though .... it's great exercise ;-)

October 16, 2014

https://www.duolingo.com/makahiwitk

As I'm into tap dancing I'm all for "more_tap", but in the Duo context it sounds scary. Now that I know what I'm true/false/control or more_tap for, I want to read somewhere what that really means, and what they are trying to discover.

October 16, 2014

https://www.duolingo.com/tariqnisarahmed

Bunches of lingot for you! Now I may be up for hours just staring at what I found... You know, most experimental models require that a subject not know they are the subject of an experiment... unless that is being tested...

...hmmm... So does anyone here have knowing_test_subject_experiment? And if so, what is it set to? :)

October 16, 2014

https://www.duolingo.com/tariqnisarahmed

I thought of something that might be fun: create threads for the different experiments. Let users opine away on what they (think they) experience and how much they like the results.

For example, if there were a thread for hover_effect_experiment, as a TRUE, I would opine that it rocks! :) I really do not know what the effect is, mind you, but I think it is what I noticed during immersion. Being able to see translations of words without clicking. If that was the effect being tested, I like it.

Also, just saw this pest: ios_thumb_centric_layout_experiment -- Since I use an iPad as much as an iPhone, I am really glad this is set to false. Having a thumb-centric iPad experience would not be very good. Actually, this calls to mind that iOS series experiments should probably be made more specific. Surely it should be possible to differentiate the iOS experience by device. And for that matter, are all the iOS experiments confined to the app? What about users who access via a mobile browser on iOS?

Anyone know what the "mobile_discussion_experiment" is?

Some of these are a bit annoying... "mobile_turn_off_microphone_forever_experiment: true"

Also sg_tap_challenge_frequency: "fewer_tap" -- More like zero tap. I never get those tile drills anymore. I cannot remember the last time I did one in fact. They were great in my iPad and iPhone browsers...

This one is a new experiment, I think: web_remove_real_world_practice: true I think it is new because I remember seeing that "real world practice" quite recently. No harm. I hardly ever used it to access immersion.

What do you think? If all these experiments were in the forums as threads, would you go comment on them?

October 16, 2014

https://www.duolingo.com/AlexisLinguist

"For example, if there were a thread for hover_effect_experiment, as a TRUE, I would opine that it rocks! :) I really do not know what the effect is, mind you, but I think it is what I noticed during immersion. Being able to see translations of words without clicking. If that was the effect being tested, I like it."

Is that really what it is? In that case, I've been in that test since April at least. Maybe hovering over the streak flame? It gives me the day number whenever I hover over it, if that's anything.

Shoot, it could be the slight fade-out of buttons and skills when you hover over them. :P

October 16, 2014

https://www.duolingo.com/tariqnisarahmed

Hence "...what they (think they) experience..." :)

You know what all this reminds me of? That great under-appreciated Will Ferrell movie where he's an IRS agent just living his life... while this author he has never met is literally writing his life for him.

October 16, 2014

https://www.duolingo.com/AlexisLinguist

Ha ha, it's anybody's guess at this point.

If we're going in that direction, The Truman Show is similar, but instead of life being written out, it's just being observed, and studied. Ring a bell? :)

October 16, 2014

https://www.duolingo.com/Geo430

It's not about immersion, streaks, or fade-outs. I'm set to false and I still get these. But other than those I'm not too sure of what it could be because, well, I'm set to false.

October 17, 2014

https://www.duolingo.com/superdaisy

"mobile_turn_off_microphone_forever_experiment: true"

Mine was set to false. Do you have the option for microphone/speaking exercises on iOS?

October 16, 2014

https://www.duolingo.com/tariqnisarahmed

Good question! Here's what my settings choices are on my laptop: Microphone, Speaker, Voice autoplay, Sound effects

On my iPad in Safari and Chrome: Microphone, Speaker, Voice autoplay, Sound effects

On my iPad in the app: Sound effects, Speaking exercises, Listening exercises

Mobile must only refer to both Android and iOS apps. In the mobile browsers I do have a microphone option. I'm going to try the Blue pill. See if I can change the setting and then see what happens in the app. :)

update nothing i changed in the browser affected the app.

October 16, 2014

https://www.duolingo.com/superdaisy

Oh wait! I think I know what the "turn off microphone forever" means! Know how when you do a speaking exercise, you can choose "I can't use the microphone right now" and then the mic is turned off for an hour? Maybe you can choose to turn it off forever right from that interface?

October 16, 2014

https://www.duolingo.com/AlexisLinguist

Right, that's the case. I hate the hour only limit. :/

October 16, 2014

https://www.duolingo.com/wataya

Hi, if anyone finds out how to activate Carla as the Italian TTS I will shower him/her with lingots (or in case it's Alexis let her put as much in my pockets as she likes). A simple

duo.user.attributes.ab_options["italian_tts_experiment"]="Carla"

does not do the trick.

October 16, 2014

https://www.duolingo.com/R-r-Leo

duo.user.attributes.ab_options returns as an object, so you need to set the value calling it as a method:

duo.user.attributes.ab_options.italian_tts_experiment = "Carla"

but I'm not sure this value can be set on the client and re-send to server every time. Most probably, it's got from the server when the lesson starts (and any assignment on browser side doesn't really change it).

October 16, 2014

https://www.duolingo.com/wataya

No, this doesn't work either. Do you have Carla? Which URL is being called for the TTS for you? For me it's

https://d7mj4aqfscim2.cloudfront.net/tts/it/sentence/{Sentence-Id}

Maybe something like "it2" for the language would do the trick? ("It2" doesn't)

October 16, 2014

https://www.duolingo.com/R-r-Leo

It's https://d7mj4aqfscim2.cloudfront.net/tts/it/carla/sentence/{Sentence-Id}

and yes, being replaced your url works for me, so mine should work for you.

October 16, 2014

https://www.duolingo.com/wataya

So, if anyone wants to put it in a user script: if(duo.user.attributes.learning_language==="it"){ duo.tts_path="tts/{voice}/carla/{type}/{id}"; } else{ duo.tts_path="tts/{voice}/{type}/{id}"; }

October 16, 2014

https://www.duolingo.com/R-r-Leo

Thanks for the shower (I think I'll share some stones with the topicstarter who has given a good jog with his investigation).

I made the following script (mostly based on your code), which doesn't change anything for people in the AB group with Carla voice. https://monkeyguts.com/code.php?id=541

October 16, 2014

https://www.duolingo.com/Queen_Ziga

thank you so much!!!!!

October 16, 2014

https://www.duolingo.com/wataya

And it does! Bingo!

EDIT: To use it in Italian lessons, I have to set:

duo.tts_path="tts/{voice}/carla/{type}/{id}"

When I'm done with the Italian exercise I can set it back to duo.tts_path="tts/{voice}/{type}{id}"

October 16, 2014

https://www.duolingo.com/wataya

So, here is your lingot shower :)

October 16, 2014

https://www.duolingo.com/makahiwitk

Using your script at monkeyguts has made my day/week/month/almost year :-) Thank you soooooooooooooo much. Finally, a good use for all my 100s of lingots.

Edit: and now the lingots are your problem as they appear to breed like rabbits :-)

October 16, 2014

https://www.duolingo.com/Alec100

So you can change which tests you want to take part in?

November 14, 2014

https://www.duolingo.com/R-r-Leo

I don't think so, this info should be stored on server side and you can only view, which a/b group you're taking part in.

November 14, 2014

https://www.duolingo.com/Alec100

So if I tried to change it, It would change it for like 2 seconds and then when the server receives it it changes back to the original?

November 14, 2014

https://www.duolingo.com/davidcwalls

It depends on the specifics of the feature that is being tested. When you change the test setting, if your change works at all, I would expect it to continue to work until a complete page refresh is required (more or less). As an example, imagine there was an A/B test that was comparing two versions of an icon. When you first load the page containing the icon, your browser requests the page contents from the server (including the A/B test settings). The question is: what event causes the icon to load? If the icon only loads during initial page load, you probably won't be able to change the icon at all because it will have already loaded before you have the chance to change the setting. But, it the icon loads during some kind of user interaction with the page, and you change the setting before the interaction in question, then you should see the different icon, and continue to see it until the next complete refresh of the page. There are some other variables, like how caching is handled for the item in question.

November 14, 2014

https://www.duolingo.com/R-r-Leo

Great investigation, thank you!

October 15, 2014

https://www.duolingo.com/Queen_Ziga

italian_tts_experiment: "francesca16" :'(

October 16, 2014

https://www.duolingo.com/CtrlZED

Thanks!

I'm actually really surprised that I didn't see an attribute called "skills_degrade_ridiculously_quickly : true".

October 16, 2014

https://www.duolingo.com/PERCE_NEIGE

I saw one "site_interfact_ sucks_more_and_more and you_cannot_do_anything_with_it_as_it's_everything_but_userfriendly.

April 5, 2017

https://www.duolingo.com/Logo121

just now i saw a "sqrt skill tree", which may explain my skills shed much more than it used to be. shame that one cannot really opt out of these experiments...

May 17, 2017

https://www.duolingo.com/ajitirj

Oh, is this the reason some people have a landmark on their homepage and some (including me) don't?

October 18, 2014

https://www.duolingo.com/tomsherida

I'm not sure I like the sound of "android_13_real_vs_fake_experiment: true" :S, sounds misleading.

April 16, 2015

https://www.duolingo.com/PERCE_NEIGE

Really, I don't know in which group I am, but I hate Duolingo interface!

It's counter-productive.

  • I can't find my own post. When it's downvoted, because people prefer chatting and silly thread than grammar questions, I've tried everything, the "followed" tab: impossible to find again my thread, even if it's very recent, 2 or 3 days, and even when I make a search with the words I know my discussion has inside.

  • I can't see my messages. I have only the last 5. That sucks a lot ! I receive a lot of messages, so if a person message me when I'm not on Duolingo, I lost forever their messages!

  • I have my activity stream flooded with things I don't care, and I flood my followers with things that they don't care, because I have an heavy activity. I want to choose what I'm an interested into, and have friends but not always see whatever if they are too active.

Why it's so bad?

April 5, 2017

https://www.duolingo.com/BMF7

Thank you. I had no idea so much was going on behind the scenes.

April 17, 2017

https://www.duolingo.com/Lrtward

You might want to point out that this is for Internet Explorer (I think).

October 15, 2014

https://www.duolingo.com/delapouite

Nope. It works in all decent browsers : Firefox, Chrome, Opera, Safari and IE.

October 15, 2014

https://www.duolingo.com/HappyEvilSlosh

You consider IE a "decent" browser?!

(shift+control+i brings up development console in Midori)

October 15, 2014

https://www.duolingo.com/makahiwitk

For FF versions 30 and above you need to read the following:

"NB: From Firefox 30, the Browser Console command line (to execute JavaScript expressions) is disabled by default. To enable it set the devtools.chrome.enabled preference to true in about:config, or set the "Enable chrome debugging" option in the developer tool settings."

Once I'd done this via the settings I was good to go. Thanks again.

October 15, 2014

https://www.duolingo.com/jayeidge

Ok, I feel stupid. I have enabled "Chrome debugging" but where in the console do I enter "duo.user.attributes.ab_options" to receive the list?

October 16, 2014

https://www.duolingo.com/makahiwitk

I had written out the rest of the steps for another user in this post but it appears to have been deleted :-(

For FF: Tools > Web Developer > Web Console At the bottom of the text box are two chevrons >> paste the code here and hit Enter. A short version of the list will appear with a more .... link. Click it and to the right of the box another box will appear with the full list.

October 16, 2014

https://www.duolingo.com/jayeidge

Thanks alot

October 17, 2014

https://www.duolingo.com/Lrtward

Huh. Couldn't figure out what to do in Safari. I figure it's Develop - Error Console or Develop - Show Web Inspector (F12 is increase volume) but then I didn't see where to type in duo.user.attributes.ab_options No matter, I'll have access to Chrome tomorrow. Nice post, thanks!

Of course just about as soon as I submitted that, I found it. Develop -> Show Web Inspector. Then type "duo.user.attributes.ab_options" in at the very bottom center of the console. As writchie4 said, the resulting list is collapsed but that's just a mouse click on the little triangle to open it up.

Very cool!

October 16, 2014

https://www.duolingo.com/AlexisLinguist

Lifesaver. I didn't know where the heck to type it in. Too many boxes in that W.I., ha ha.

October 16, 2014

https://www.duolingo.com/FrankySka
Mod
  • 2167

awesome! thanks!

October 17, 2014

https://www.duolingo.com/writchie4

Works fine for me in Chrome, gotta type in the Console tab along the bottom. And then the list is displayed collapsed, which is sort of annoying. But it's definitely there.

October 15, 2014

https://www.duolingo.com/GrandApple

Thank you.

October 15, 2014

https://www.duolingo.com/GrandApple

Inspect element is also on Google Chrome.

October 15, 2014

https://www.duolingo.com/AlexisLinguist

I am false for a ton, but looks like I may have found the bane of my existence!

disable_auto_correct_experiment: true

NO.

October 16, 2014

https://www.duolingo.com/makahiwitk

Yes, I'm also true for that one. Does that mean some users get incorrect answers auto-corrected? Wow - could I whiz through the lessons if that was the case :-) However, I'm here to learn and not just to play games, although the games are soooo much fun ....

October 16, 2014

https://www.duolingo.com/AlexisLinguist

I'm thinking it has to do with the computer function of auto-correct being disabled. I'd noticed that my Mac had stopped correcting me, and lo and behold, this stupid test. :)

Auto-correct no longer works on iOS either (NO....), so it looks like it applies to both. I was dismayed by all the falses, but also astounded at the sheer number of tests they are running. Are they all really necessary..................?

October 16, 2014

https://www.duolingo.com/makahiwitk

At this stage I'm feeling a bit like a lab rat. I don't mind the testing, but I'm now wondering what I'm missing out on, or in the case of the XP bar counting my lucky stars. Unfortunately, my daughter who has just joined Duo has something that doesn't look at all like my Duo, and I can't ask her things like .... "How far until you reach the next level?" as this draws a blank stare.

October 16, 2014

https://www.duolingo.com/AlexisLinguist

"At this stage I'm feeling a bit like a lab rat."

Same. It's just not fun. The user experiences just vary so much from one thing to the next that I wonder if Duolingo is getting the true data that they covet so much. The metrics have to be all over the place with all these things running at the same time. What happened to simplicity, you know?

October 16, 2014

https://www.duolingo.com/superdaisy

The metrics have to be all over the place with all these things running at the same time.

Yeah, that's what bugs me. Whatever happened to the classic research design I learned in college? But maybe there's complicated statistics mojo controlling for all these interactions.

October 16, 2014

https://www.duolingo.com/makahiwitk

There aren't many websites that I visit that change on an almost weekly basis. Probably because the users would stop coming. I also discovered today that all the effort I've put into reporting potential problems has just been filed. I'd wondered why I've never had an email reply as other users have reported. However, I keep coming back because I am learning and I love the community. If there was no community I'd find someplace else, as the connectivity would be gone. I feel absolutely disconnected from the Duolingo staff.

October 16, 2014

https://www.duolingo.com/AlexisLinguist

@Darelle: I don't frequent any websites that change on a weekly basis. This is the first one, ha ha.

Communication is something we all want from the staff. When, and if, we get that are the questions.

@superdaisy: shrugs Maybe.

October 16, 2014

https://www.duolingo.com/makahiwitk

Actually, what I'd like to know is what metrics are used to sort us into the trues and the falses - obviously some things I do are good (I'm left as false), and others I'm not so good at so I get to test a new feature. I've been looking for connections but I can't see anything obvious.

October 16, 2014

https://www.duolingo.com/AlexisLinguist

Pretty sure that A/B group sorting is random. But, what do I know? :)

October 16, 2014

https://www.duolingo.com/peter.doerrie

To those who think that these experiments are useless: this is what duolingo is supposed to be. Go and see the TED talk of its founder, where he explains why A/B testing is at the very heart of duolingo and won't go away. Short version: A/B testing of small iterations of the learning experience on a large number of people makes language learning better for everybody.

As for the multitude of tests running at the same time: it is all a function of the number of people using duolingo. If you have millions of people, you can run a whole bunch of different tests and still control for the various configurations. This might even be obligatory, because some chances may have only a positive effect in combination with other changes and you don't want to miss a case like this.

Look at the whole thing as a form of the tragedy of the commons. Yes, maybe your user experience sucks in a few subtle ways at the moment while other users already have the good stuff. But in the long run we all get to learn foreign languages much faster thanks to the power of science and statistics.

October 16, 2014

https://www.duolingo.com/PERCE_NEIGE

Abuse of A/B testing is the Evil !

I see a lot of AB stuff, but NO improvements. Only things that are worse and worse in the GUI.

April 5, 2017

https://www.duolingo.com/Rompip
  • 2041

Sorry, being of the older generation and not very computer literate, I have never been into the 'dev console' before. Can someone please tell me where I should type in the command? (I have Google Chrome) I really want to know which groups I'm in. I suspect I am in the one for longer sentences as I have had a lot of problems recently because I just can't type fast enough.

October 17, 2014

https://www.duolingo.com/writchie4

The actual instructions will vary according to which web browser you're using (Chrome, Firefox, Internet Explorer).

In Chrome, after hitting F12, you should see a bar across the top of the pane that opens up. This bar says "Elements, Network, Sources, etc". At the end of the bar is the word "Console", click that to open the console. You'll see a line in the console that says "Duolingo is hiring software engineers" and immediately below that, an empty line with a ">" symbol. Click on that line and enter the command shown in the original post. A new line will appear starting with "► Object". Click on the ► to expand the line and see each of the experiment values on their own line.

There is another way to see this information, which may be easier for you. Just visit: http://www.duolingo.com/users/Rompip

On that page, lots of information about your account is shown, including this "ab_options" structure. Once the page loads, hit Ctrl+F and type "ab_options" to find it easily. It's a little tough to read, but hey, that's code for ya. :)

October 17, 2014

https://www.duolingo.com/Rompip
  • 2041

Thanks a lot for this , nice and simple for me ;)

October 18, 2014

https://www.duolingo.com/CatherineK153278

all I ended up with was "Failed to load resource: the server responded with a status of 403 ()" I tried opening the Rompip screen and searched for ab_options to not have it show up at all.

May 24, 2018

https://www.duolingo.com/SethFriday

Hmm, I found out i have Carla. I might give Italian a second chance.

October 20, 2014

https://www.duolingo.com/EvelynOlson0

Can I edit it? If so, how?

February 22, 2016

https://www.duolingo.com/EvelynOlson0

Pressing F12 only turns on airplane mode.

February 12, 2017

https://www.duolingo.com/DeanG6

I don't think this works anymore. The list I get looks like it's for the school version. I don't see anything for Coach or the Italian voice.

February 16, 2017

https://www.duolingo.com/PapyXM

In your web browser, press F12 to get access to the dev console. Then simply type the following command to inspect the variable : duo.user.attributes.ab_options

I don't catch it. When I press F12, the Firefox Inspector panel appears at the bottom, but in what field of which tab do I enter the command ? Could someone tell me?

Inspector

[EDIT]

OK, now I found it, I didn't see the gray double prompt on the last line when the Net tab of the Console tab is active.

In fact, those info are publicly available. For instance, you can look at my profile on this page PapyXM. The A/B settings are listed after the key word ab_options

["ab_options":{"sg_always_measure_learning_experiment":false,"sg_viability_v2_experiment_5":false,"sg_hand_selected_always_counterfactual_experiment":false,"new_years_resolution_experiment2":"undefined","sg_hand_selected_separated_common_phrases_1_experiment":true,"web_noimmersion_experiment":true,"sg_hand_selected_grammar_incumbent_experiment":false,"sg_hand_selected_separated_incumbent_basics_3_experiment":true,"sg_boost_row_test_challenge_probability_experiment":true,"sg_hand_selected_separated_incumbent_basics2_2_experiment":true,"sg_hand_selected_separated_incumbent_basics_1_experiment":true,"sg_hand_selected_separated_incumbent_basics2_4_experiment":false,"ios_25_ring_of_fire_test":true,"sg_hand_selected_separated_incumbent_basics2_3_experiment":false,"ios_19_fluency_end_screen":true,"ios_24_name_challenge_copy_test":false,"sg_sf_fluency_score_experiment":"sf_based","ios_disable_china_dns_test":true,"web_no_tips_and_notes_experiment":true,"no_op_experiment2":false,"no_op_experiment3":true,"sg_hand_selected_grammar_incumbent_experiment2":true,"sg_hand_selected_separated_common_phrases_2_experiment":false,"no_op_experiment4":false,"ios_25_unified_delayed_signup_test":true,"tree_en_ar_v1_experiment":1,"sg_hand_selected_incumbent_grammar_combined_experiment":true,"ios_18_use_characters_test":true,"no_op_experiment":true,"tree_de_en_v3_experiment":3,"ios_no_social_signup_experiment":true,"sg_hand_selected_incumbent_experiment2":true,"sg_hand_selected_separated_food_3_experiment":false,"sg_is_testing_user":false,"ios_24_autoset_timezone_test":true,"sg_test_challenge_probability_experiment2":"0.1","ios_24_no_discard_progress":true,"ios_23_local_grading_test":"control","sg_boost_skill_test_challenge_probability_experiment":false,"sg_always_add_learning_unit_test_challenge":false,"sg_hand_selected_separated_animals_2_experiment":false,"sg_hand_selected_separated_incumbent_basics2_1_experiment":true,"ios_china_cdn_test":false,"sg_hand_selected_separated_incumbent_basics_2_experiment":true,"ios_19_allow_sharing_test":true,"ios_21_history_test":true,"ios_31_local_grading_test":false,"sqrt_skill_strength_experiment":false,"ios_24_register_birth_year_test":false,"web_remove_quiz_experiment":true,"sg_hand_selected_separated_food_4_experiment":true,"kc_based_dynamic_tree_progress_experiment":false} ]

March 11, 2017

https://www.duolingo.com/fkachinski

How do we switch from A to B or vise-versa?

Many people hate the new “health” and want into the other group!

Specially, using the iPhone app.

September 3, 2018

https://www.duolingo.com/AldenDavidson

You can't affect your groups at all, those are set by Duolingo for the purposes of collecting data on new and potentially new features. Eventually they'll either move you back to the previous system - if they decide to not use the new "health" - or you'll stay permanently on the new system - if they decide it's working as intended.

September 3, 2018

https://www.duolingo.com/fkachinski

Doesn’t it make more sense to move someone rather than loose them as a user?

September 3, 2018

https://www.duolingo.com/AldenDavidson

That, unfortunately, would probably open up a pandora's box of problems. Not only would they have to hire on significantly more staff or spend significantly more time handling transfer requests, it would also potentially impact the size and makeup of their testing groups which would impact their results. Duolingo seems to be committed to making decisions almost completely data-driven, so while I sympathize with you I don't see them allowing that anytime soon.

September 3, 2018

https://www.duolingo.com/jfernatt

Very cool thanks!

October 15, 2014

https://www.duolingo.com/GrandApple

Wait, where do I type it in inside the source code? Oh, never mind.

October 15, 2014

https://www.duolingo.com/mehlon

Ahh, I've been looking at the wrong object the whole time! I thought it was simply duo.ab_options. Thanks.

October 16, 2014

https://www.duolingo.com/DamedesLangues

Hi! I'm curious to see what test groups I'm in, but my tech skills are non-existent. I've clicked on browser settings, then F12 developer tools. This is where I came to a sudden halt. I have no idea where exactly to type the command.

Whilst some helpful users have pointed out how to go about this on Chrome, Safari and Firefox, I can't find one explaining the procedure for IE. Could someone help me, please?

October 19, 2014

https://www.duolingo.com/superdaisy
October 20, 2014

https://www.duolingo.com/DamedesLangues

Thank you so much! I'll let you know how it goes. :-)

October 22, 2014

https://www.duolingo.com/louis.vang

Where should i type the command

duo.user.attributes.ab_options ?

April 7, 2015

https://www.duolingo.com/tomsherida

Far right hand side tab 'Console'

April 16, 2015

https://www.duolingo.com/rooseveltnut1

I hit F12 and got nothing but code. Oh well. No surprise. Duolingo doesn't like me much lately............;-)

June 15, 2015

https://www.duolingo.com/cathgrl

I'm confused. Do you type the command at the filter output box? Because when I do that, nothing comes up.

June 18, 2015

https://www.duolingo.com/cathgrl

I found it. Thanks!

June 18, 2015

https://www.duolingo.com/deoxy144

Either I'm doing something wrong, or this doesn't work anymore. I tried it it Chrome, FireFox, and Edge. In all of them, I get errors that imply the "user" object does not exist.

March 25, 2018

https://www.duolingo.com/writchie4

Still works for me. Make sure you're doing it from a tab with Duolingo open and logged in, then paste "duo.user.attributes.ab_options" (without quotes) in the console.

March 25, 2018

https://www.duolingo.com/deoxy144

Strange. I tried doing it when on various parts of the Duolingo website (logged in at the home page, in the middle of a lesson, etc.), and I just get an error. Here's the Chrome version of it: Uncaught TypeError: Cannot read property 'attributes' of undefined at <anonymous>:1:10

March 26, 2018

https://www.duolingo.com/writchie4

I get the same error if I try to follow the directions in Chrome's incognito mode. Perhaps that's your issue, or maybe you've got some sort of browser extension interfering? NoScript, something like that?

March 26, 2018

https://www.duolingo.com/AlysiaGoss

Thank you!

June 14, 2019
Learn a language in just 5 minutes a day. For free.