A/B Testing : how to discover your groups.

5string

You see that a lot. Because only coding nerds usually bother to check out a page's source code, it becomes a hidden but nicely focussed place to pop in a job advert. Google does that all of the time, and Apple a few times.

Yoruji

I liked the following part as well "// I am unsure under what conditions this would occur. if(!_.isFunction(mixpanel.register)) { "

Thanks. It's nice to see that my list is a series of 'falses' and 'controls'. Apart from two, which really stand out, and are the group for longer sentence lengths and more listening challenges - yeh, lucky me. The least they could have done regarding the listening challenges is to also give me Carla, but no, I've still got old gasping for breath francesca16.

[deactivated user]

Francesca16's voice makes learning Italian awkward.

"Una gasp donna gasp mangia gasp la gasp mela."

Good thing Italian is not a pressing language for me -- I am also Francesca bound.

Unfortunately, she makes me cringe, as I do know what Italian should sound like. It's just that having more listening challenges makes it more 'in your face' or perhaps that should be 'more in your ear'. Still, free is free, and you get what you get. :-)

whose francesca

I have no Italian tree yet I have Carla. It's like I have a lot of a currency and nothing to spend it on. But that might be a bit of a realistic analogy for Duo users.

Go figure! No tree, but you have Carla. Then the sorting of users into the various groups is akin to Lotto - it's just the luck of the draw! :-)

Jonathan--

Huh, I just changed coach_web_experiment from "false" to "true," went to the homepage, and there was the coach. So at least in some cases, if you want to see how the other half lives, you just need to write yourself into the other half.

If you change a parameter on one browser on one device, does it affect every other session? Try changing one of your mobile, android, iOS settings from your desktop/laptop. Then see if it changed the experience on your mobile device... Answer: Tried it and nothing changed on the mobile device. So any changes only effect the browser where you made changes.

If the changes you make in your browser only affect that session, meaning that the effect is local, then I wonder how much that would mess up the data collected by Duolingo? I mean we have proven that the coach can be enabled (Jonathan) and disabled (AlexisLinguist, who, admirably perhaps, turned it back on again, too).

What if all the users whom DL designated as Coached simply turned it off. Then DL's Coach data would be skewed unless they could perceive whether a browser F12 hack had been performed...

And to go one step further, there are a lot of users using tweaked browsers already. They change the DL colors, etc. Some bring back the coach. Some don't. Unless DL has a way of monitoring all those hacks, then all (or part) of the test data for those users may be bunk.

Yup, if DL has no way to gauge what tweaks users have made, or even whether users have made changes, then every person who makes a change is like a gremlin run amok in the DL labs.

(AlexisLinguist, who, admirably perhaps, turned it back on again, too)

Please don't remind me. It's right there, yet I cannot touch it. So close, yet so far. I may need, a Snickers bar.

Yup, if DL has no way to gauge what tweaks users have made, or even whether users have made changes, then every person who makes a change is like a gremlin run amok in the DL labs.

If we're talking about real gremlins here, then I'm Spike.

But you're a reformed Spike. You tasted the food after dark but then went back to being a Mogwai.

Who said that? Maybe I like being bad. :P

shalfyard

"then every person who makes a change is like a gremlin run amok in the DL labs." Not really... It's all very local to that instance. The second they navigate somewhere or refresh (get a page load in general) the settings will get refreshed back to what the server has set for you.

My point being that DL thinks the person is using environment A (the sum of a bajillion tests being run simultaneously) while in fact any usage of the site reflects environment G (what the Gremlin has wrought on his own screen).

"Amok" like the rest of what I wrote being colorful language.

feanarosurion

Everyone might be a gremlin, but I think a lot of the people who might actually go ahead and decide to make these changes are probably devoted enough that they're within the "firmly retained" camp for their precious metrics. Plus, the values probably aren't changing on Duo's end. So it probably doesn't matter if a few people change a setting or two. All hail the divine infallible metrics!

shalfyard

The settings are all javascript based and get set by the server when you visit a page. Thus they get reset to the server instance the second you navigate or refresh... Which is after you click strengthen, answer a question, etc. Essentially you are getting a glimpse into what you could have but not actually getting that experience to mess up their metrics.

HOW!?

Whoa. Now this is kinda like another movie... (in the Will Ferrell movie, he really never initiated anything, did he?)

Yup, I think Jonathan just chose the blue pill. Someone keep a lookout for Morpheus (not to mention Agents) and cue the techno music because Duolingo has become the Matrix.

I've changed some of my settings but I haven't noticed any changes. :-(

feanarosurion

This is really telling. The number of tests they're carrying out really annoys me. The whole idea of quietly AB testing the site (or anywhere) annoys me. They should tell us what they're working on, implement it for everyone, and let us give feedback. Plus be more engaged in the community in general. </rant>

2682

I'm not sure I understand why this annoys you. A/B testing is a good thing, as much as it stinks to be in the control ("reject") group. It's a proven tool for determining which features have a positive impact on the site, without relying on wishy-washy (and potentially contradictory) written user feedback. As for the number of concurrent tests, that's a function of the size of Duolingo's user base. Having millions of users allows them to test multiple features at once. How is that a bad thing? If anything, it allows them to improve the site more rapidly.

I'm with you on the staff-community engagement point though. It wouldn't kill them to be a little more active on the forums.

Agreed. Also, maybe these are all the experiments that have ever been run. Or ever been run since a given user joined. Does not mean every test is in progress at the same time.

I think these are all currently in progress. The one that enabled the "Start button" for new lessons is no longer there, so old tests disappear from the code.

Wow. Then that is a whole lot of testing going on... Think they pay attention to effects of one experiment on the results of another?

"Think they pay attention to effects of one experiment on the results of another?"

I'd hope so. There are more than several experience "configurations" at the moment, so I'd hope they would be taking that into account. Otherwise, seems like that taint the results, and the whole thing would be a waste of time.

I mean, I already slowed down my usage (learning) when the new design rolled out, and now the Coach...I just want the old design (ha, it's only been two-three weeks now) back.

Hey AlexisLinguist, did you try reversing what Jonathan did?

"Huh, I just changed coach_web_experiment from "false" to "true," went to the homepage, and there was the coach. So at least in some cases, if you want to see how the other half lives, you just need to write yourself into the other half."

Maybe you could dispatch the Coach.

It works. :) For the sake of the test, though, I'll make it true again. Oh, the beautiful progress bar. How I miss you so.

Lrtward

Mod

If they tell us what they're working on, that could skew the results.

Well, we already know, so all results are effectively skewed. ;) It's interesting that the tests would be "out in the open" like this. Wouldn't keeping them tucked away so users couldn't fiddle with them be better? It's almost like they want us to find them. Shoo, if it's an open invitation...

Lrtward

Mod

Hey, don't come at me with logic, man! That's cheating! It's not fair! Ima down vote all your comments!!!

Good point. We do already know, don't we? Except, fortunately, I have the memory of an earthworm so I already forgot what groups I'm in. Go me.

Not sure they can be tucked away where a web development tool can't fiddle with them. That's the nature of web pages vs. compiled apps, I fear.

*Woman. ;)

Good for you. Unfortunately, I'm semi-eidetic. :P

Lord, how I want to go back to the progress bar...I don't want to play with the cards I was dealt. But, I guess if I never cheated at Uno ;) , I'll tough it out.

Interesting!

Here are some of mine:

coach_track_experiment: "trackless" (...but I have Coach on, so I wonder what this means)

enable_web_speech_experiment_v3: "reverse_speak" (don't use the speech component online, but when I did I was asked to translate English to Spanish and then speak)

ios_no_social_signup_experiment: true

sg_tap_challenge_frequency: "more_tap" (...tap?)

web_remove_real_world_practice: true (I just noticed my Real World Practice button is gone from underneath my Strengthen Skills button)

web_give_up_button_experiment: false (interesting, there's a "give up" button? I just refresh...)

shalfyard

Tap challenges are likely for the app. They let you pick from a set of words to build your translations... Makes it crazy easy to be honest.

Oh, that makes sense.

Yes, they are pretty easy, though it's nice to practice word order and structure without worrying too much about vocabulary. If the non-answers were more similar (different conjugations, for instance) then the questions would be more challenging.

Thanks, so it doesn't apply to me and my laptop. I'd love a tap (dance) challenge though .... it's great exercise ;-)

As I'm into tap dancing I'm all for "more_tap", but in the Duo context it sounds scary. Now that I know what I'm true/false/control or more_tap for, I want to read somewhere what that really means, and what they are trying to discover.

Bunches of lingot for you! Now I may be up for hours just staring at what I found... You know, most experimental models require that a subject not know they are the subject of an experiment... unless that is being tested...

...hmmm... So does anyone here have knowing_test_subject_experiment? And if so, what is it set to? :)

I thought of something that might be fun: create threads for the different experiments. Let users opine away on what they (think they) experience and how much they like the results.

For example, if there were a thread for hover_effect_experiment, as a TRUE, I would opine that it rocks! :) I really do not know what the effect is, mind you, but I think it is what I noticed during immersion. Being able to see translations of words without clicking. If that was the effect being tested, I like it.

Also, just saw this pest: ios_thumb_centric_layout_experiment -- Since I use an iPad as much as an iPhone, I am really glad this is set to false. Having a thumb-centric iPad experience would not be very good. Actually, this calls to mind that iOS series experiments should probably be made more specific. Surely it should be possible to differentiate the iOS experience by device. And for that matter, are all the iOS experiments confined to the app? What about users who access via a mobile browser on iOS?

Anyone know what the "mobile_discussion_experiment" is?

Some of these are a bit annoying... "mobile_turn_off_microphone_forever_experiment: true"

Also sg_tap_challenge_frequency: "fewer_tap" -- More like zero tap. I never get those tile drills anymore. I cannot remember the last time I did one in fact. They were great in my iPad and iPhone browsers...

This one is a new experiment, I think: web_remove_real_world_practice: true I think it is new because I remember seeing that "real world practice" quite recently. No harm. I hardly ever used it to access immersion.

What do you think? If all these experiments were in the forums as threads, would you go comment on them?

"For example, if there were a thread for hover_effect_experiment, as a TRUE, I would opine that it rocks! :) I really do not know what the effect is, mind you, but I think it is what I noticed during immersion. Being able to see translations of words without clicking. If that was the effect being tested, I like it."

Is that really what it is? In that case, I've been in that test since April at least. Maybe hovering over the streak flame? It gives me the day number whenever I hover over it, if that's anything.

Shoot, it could be the slight fade-out of buttons and skills when you hover over them. :P

Hence "...what they (think they) experience..." :)

You know what all this reminds me of? That great under-appreciated Will Ferrell movie where he's an IRS agent just living his life... while this author he has never met is literally writing his life for him.

Ha ha, it's anybody's guess at this point.

If we're going in that direction, The Truman Show is similar, but instead of life being written out, it's just being observed, and studied. Ring a bell? :)

Geo430

It's not about immersion, streaks, or fade-outs. I'm set to false and I still get these. But other than those I'm not too sure of what it could be because, well, I'm set to false.

"mobile_turn_off_microphone_forever_experiment: true"

Mine was set to false. Do you have the option for microphone/speaking exercises on iOS?

Good question! Here's what my settings choices are on my laptop: Microphone, Speaker, Voice autoplay, Sound effects

On my iPad in Safari and Chrome: Microphone, Speaker, Voice autoplay, Sound effects

On my iPad in the app: Sound effects, Speaking exercises, Listening exercises

Mobile must only refer to both Android and iOS apps. In the mobile browsers I do have a microphone option. I'm going to try the Blue pill. See if I can change the setting and then see what happens in the app. :)

update nothing i changed in the browser affected the app.

Oh wait! I think I know what the "turn off microphone forever" means! Know how when you do a speaking exercise, you can choose "I can't use the microphone right now" and then the mic is turned off for an hour? Maybe you can choose to turn it off forever right from that interface?

Right, that's the case. I hate the hour only limit. :/

Hi, if anyone finds out how to activate Carla as the Italian TTS I will shower him/her with lingots (or in case it's Alexis let her put as much in my pockets as she likes). A simple

duo.user.attributes.ab_options["italian_tts_experiment"]="Carla"

does not do the trick.

2529

duo.user.attributes.ab_options returns as an object, so you need to set the value calling it as a method:

duo.user.attributes.ab_options.italian_tts_experiment = "Carla"

but I'm not sure this value can be set on the client and re-send to server every time. Most probably, it's got from the server when the lesson starts (and any assignment on browser side doesn't really change it).

No, this doesn't work either. Do you have Carla? Which URL is being called for the TTS for you? For me it's

https://d7mj4aqfscim2.cloudfront.net/tts/it/sentence/{Sentence-Id}

Maybe something like "it2" for the language would do the trick? ("It2" doesn't)

2529

It's https://d7mj4aqfscim2.cloudfront.net/tts/it/carla/sentence/{Sentence-Id}

and yes, being replaced your url works for me, so mine should work for you.

So, if anyone wants to put it in a user script: if(duo.user.attributes.learning_language==="it"){ duo.tts_path="tts/{voice}/carla/{type}/{id}"; } else{ duo.tts_path="tts/{voice}/{type}/{id}"; }

2529

Thanks for the shower (I think I'll share some stones with the topicstarter who has given a good jog with his investigation).

I made the following script (mostly based on your code), which doesn't change anything for people in the AB group with Carla voice. https://monkeyguts.com/code.php?id=541

Queen_Ziga

thank you so much!!!!!

And it does! Bingo!

EDIT: To use it in Italian lessons, I have to set:

duo.tts_path="tts/{voice}/carla/{type}/{id}"

When I'm done with the Italian exercise I can set it back to duo.tts_path="tts/{voice}/{type}{id}"

So, here is your lingot shower :)

Using your script at monkeyguts has made my day/week/month/almost year :-) Thank you soooooooooooooo much. Finally, a good use for all my 100s of lingots.

Edit: and now the lingots are your problem as they appear to breed like rabbits :-)

Alec100

So you can change which tests you want to take part in?

2529

I don't think so, this info should be stored on server side and you can only view, which a/b group you're taking part in.

Alec100

So if I tried to change it, It would change it for like 2 seconds and then when the server receives it it changes back to the original?

davidcwalls

Mod

Plus

2772

It depends on the specifics of the feature that is being tested. When you change the test setting, if your change works at all, I would expect it to continue to work until a complete page refresh is required (more or less). As an example, imagine there was an A/B test that was comparing two versions of an icon. When you first load the page containing the icon, your browser requests the page contents from the server (including the A/B test settings). The question is: what event causes the icon to load? If the icon only loads during initial page load, you probably won't be able to change the icon at all because it will have already loaded before you have the chance to change the setting. But, it the icon loads during some kind of user interaction with the page, and you change the setting before the interaction in question, then you should see the different icon, and continue to see it until the next complete refresh of the page. There are some other variables, like how caching is handled for the item in question.

2529

Great investigation, thank you!

Queen_Ziga

italian_tts_experiment: "francesca16" :'(

CtrlZED

Thanks!

I'm actually really surprised that I didn't see an attribute called "skills_degrade_ridiculously_quickly : true".

PERCE_NEIGE

I saw one "site_interfact_ sucks_more_and_more and you_cannot_do_anything_with_it_as_it's_everything_but_userfriendly.

April 5, 2017

Logo121

1562

just now i saw a "sqrt skill tree", which may explain my skills shed much more than it used to be. shame that one cannot really opt out of these experiments...

May 17, 2017

ajitirj

Oh, is this the reason some people have a landmark on their homepage and some (including me) don't?

October 18, 2014

tomsherida

I'm not sure I like the sound of "android_13_real_vs_fake_experiment: true" :S, sounds misleading.

April 16, 2015

PERCE_NEIGE

Really, I don't know in which group I am, but I hate Duolingo interface!

It's counter-productive.

I can't find my own post. When it's downvoted, because people prefer chatting and silly thread than grammar questions, I've tried everything, the "followed" tab: impossible to find again my thread, even if it's very recent, 2 or 3 days, and even when I make a search with the words I know my discussion has inside.
I can't see my messages. I have only the last 5. That sucks a lot ! I receive a lot of messages, so if a person message me when I'm not on Duolingo, I lost forever their messages!
I have my activity stream flooded with things I don't care, and I flood my followers with things that they don't care, because I have an heavy activity. I want to choose what I'm an interested into, and have friends but not always see whatever if they are too active.

Why it's so bad?

April 5, 2017

BMF7

Thank you. I had no idea so much was going on behind the scenes.

April 17, 2017

Lrtward

Mod

You might want to point out that this is for Internet Explorer (I think).

delapouite

Nope. It works in all decent browsers : Firefox, Chrome, Opera, Safari and IE.

MeltingPsychicPlus

You consider IE a "decent" browser?!

(shift+control+i brings up development console in Midori)

For FF versions 30 and above you need to read the following:

"NB: From Firefox 30, the Browser Console command line (to execute JavaScript expressions) is disabled by default. To enable it set the devtools.chrome.enabled preference to true in about:config, or set the "Enable chrome debugging" option in the developer tool settings."

Once I'd done this via the settings I was good to go. Thanks again.

jayeidge

Ok, I feel stupid. I have enabled "Chrome debugging" but where in the console do I enter "duo.user.attributes.ab_options" to receive the list?

I had written out the rest of the steps for another user in this post but it appears to have been deleted :-(

For FF: Tools > Web Developer > Web Console At the bottom of the text box are two chevrons >> paste the code here and hit Enter. A short version of the list will appear with a more .... link. Click it and to the right of the box another box will appear with the full list.

Thanks alot

Mod

Huh. Couldn't figure out what to do in Safari. I figure it's Develop - Error Console or Develop - Show Web Inspector (F12 is increase volume) but then I didn't see where to type in duo.user.attributes.ab_options No matter, I'll have access to Chrome tomorrow. Nice post, thanks!

Of course just about as soon as I submitted that, I found it. Develop -> Show Web Inspector. Then type "duo.user.attributes.ab_options" in at the very bottom center of the console. As writchie4 said, the resulting list is collapsed but that's just a mouse click on the little triangle to open it up.

Very cool!

Lifesaver. I didn't know where the heck to type it in. Too many boxes in that W.I., ha ha.

FrankySka

Mod

Plus

2719

awesome! thanks!

2682

Works fine for me in Chrome, gotta type in the Console tab along the bottom. And then the list is displayed collapsed, which is sort of annoying. But it's definitely there.

Thank you.

Inspect element is also on Google Chrome.

I am false for a ton, but looks like I may have found the bane of my existence!

disable_auto_correct_experiment: true

NO.

Yes, I'm also true for that one. Does that mean some users get incorrect answers auto-corrected? Wow - could I whiz through the lessons if that was the case :-) However, I'm here to learn and not just to play games, although the games are soooo much fun ....

I'm thinking it has to do with the computer function of auto-correct being disabled. I'd noticed that my Mac had stopped correcting me, and lo and behold, this stupid test. :)

Auto-correct no longer works on iOS either (NO....), so it looks like it applies to both. I was dismayed by all the falses, but also astounded at the sheer number of tests they are running. Are they all really necessary..................?

At this stage I'm feeling a bit like a lab rat. I don't mind the testing, but I'm now wondering what I'm missing out on, or in the case of the XP bar counting my lucky stars. Unfortunately, my daughter who has just joined Duo has something that doesn't look at all like my Duo, and I can't ask her things like .... "How far until you reach the next level?" as this draws a blank stare.

"At this stage I'm feeling a bit like a lab rat."

Same. It's just not fun. The user experiences just vary so much from one thing to the next that I wonder if Duolingo is getting the true data that they covet so much. The metrics have to be all over the place with all these things running at the same time. What happened to simplicity, you know?

The metrics have to be all over the place with all these things running at the same time.

Yeah, that's what bugs me. Whatever happened to the classic research design I learned in college? But maybe there's complicated statistics mojo controlling for all these interactions.

There aren't many websites that I visit that change on an almost weekly basis. Probably because the users would stop coming. I also discovered today that all the effort I've put into reporting potential problems has just been filed. I'd wondered why I've never had an email reply as other users have reported. However, I keep coming back because I am learning and I love the community. If there was no community I'd find someplace else, as the connectivity would be gone. I feel absolutely disconnected from the Duolingo staff.

@Darelle: I don't frequent any websites that change on a weekly basis. This is the first one, ha ha.

Communication is something we all want from the staff. When, and if, we get that are the questions.

@superdaisy: shrugs Maybe.

Actually, what I'd like to know is what metrics are used to sort us into the trues and the falses - obviously some things I do are good (I'm left as false), and others I'm not so good at so I get to test a new feature. I've been looking for connections but I can't see anything obvious.

Pretty sure that A/B group sorting is random. But, what do I know? :)

peter.doerrie

To those who think that these experiments are useless: this is what duolingo is supposed to be. Go and see the TED talk of its founder, where he explains why A/B testing is at the very heart of duolingo and won't go away. Short version: A/B testing of small iterations of the learning experience on a large number of people makes language learning better for everybody.

As for the multitude of tests running at the same time: it is all a function of the number of people using duolingo. If you have millions of people, you can run a whole bunch of different tests and still control for the various configurations. This might even be obligatory, because some chances may have only a positive effect in combination with other changes and you don't want to miss a case like this.

Look at the whole thing as a form of the tragedy of the commons. Yes, maybe your user experience sucks in a few subtle ways at the moment while other users already have the good stuff. But in the long run we all get to learn foreign languages much faster thanks to the power of science and statistics.

PERCE_NEIGE

Abuse of A/B testing is the Evil !

I see a lot of AB stuff, but NO improvements. Only things that are worse and worse in the GUI.

April 5, 2017

Rompip

Sorry, being of the older generation and not very computer literate, I have never been into the 'dev console' before. Can someone please tell me where I should type in the command? (I have Google Chrome) I really want to know which groups I'm in. I suspect I am in the one for longer sentences as I have had a lot of problems recently because I just can't type fast enough.

2682

The actual instructions will vary according to which web browser you're using (Chrome, Firefox, Internet Explorer).

In Chrome, after hitting F12, you should see a bar across the top of the pane that opens up. This bar says "Elements, Network, Sources, etc". At the end of the bar is the word "Console", click that to open the console. You'll see a line in the console that says "Duolingo is hiring software engineers" and immediately below that, an empty line with a ">" symbol. Click on that line and enter the command shown in the original post. A new line will appear starting with "► Object". Click on the ► to expand the line and see each of the experiment values on their own line.

There is another way to see this information, which may be easier for you. Just visit: http://www.duolingo.com/users/Rompip

On that page, lots of information about your account is shown, including this "ab_options" structure. Once the page loads, hit Ctrl+F and type "ab_options" to find it easily. It's a little tough to read, but hey, that's code for ya. :)

Rompip

Thanks a lot for this , nice and simple for me ;)

October 18, 2014

CatherineK153278

all I ended up with was "Failed to load resource: the server responded with a status of 403 ()" I tried opening the Rompip screen and searched for ab_options to not have it show up at all.

May 24, 2018

SethFriday

Hmm, I found out i have Carla. I might give Italian a second chance.

October 20, 2014

EvelynOlson0Plus

Can I edit it? If so, how?

February 22, 2016

EvelynOlson0Plus

Pressing F12 only turns on airplane mode.

February 12, 2017

DeanG6

I don't think this works anymore. The list I get looks like it's for the school version. I don't see anything for Coach or the Italian voice.

February 16, 2017

PapyXM

In your web browser, press F12 to get access to the dev console. Then simply type the following command to inspect the variable : duo.user.attributes.ab_options

I don't catch it. When I press F12, the Firefox Inspector panel appears at the bottom, but in what field of which tab do I enter the command ? Could someone tell me?

Inspector

[EDIT]

OK, now I found it, I didn't see the gray double prompt on the last line when the Net tab of the Console tab is active.

In fact, those info are publicly available. For instance, you can look at my profile on this page PapyXM. The A/B settings are listed after the key word ab_options

["ab_options":{"sg_always_measure_learning_experiment":false,"sg_viability_v2_experiment_5":false,"sg_hand_selected_always_counterfactual_experiment":false,"new_years_resolution_experiment2":"undefined","sg_hand_selected_separated_common_phrases_1_experiment":true,"web_noimmersion_experiment":true,"sg_hand_selected_grammar_incumbent_experiment":false,"sg_hand_selected_separated_incumbent_basics_3_experiment":true,"sg_boost_row_test_challenge_probability_experiment":true,"sg_hand_selected_separated_incumbent_basics2_2_experiment":true,"sg_hand_selected_separated_incumbent_basics_1_experiment":true,"sg_hand_selected_separated_incumbent_basics2_4_experiment":false,"ios_25_ring_of_fire_test":true,"sg_hand_selected_separated_incumbent_basics2_3_experiment":false,"ios_19_fluency_end_screen":true,"ios_24_name_challenge_copy_test":false,"sg_sf_fluency_score_experiment":"sf_based","ios_disable_china_dns_test":true,"web_no_tips_and_notes_experiment":true,"no_op_experiment2":false,"no_op_experiment3":true,"sg_hand_selected_grammar_incumbent_experiment2":true,"sg_hand_selected_separated_common_phrases_2_experiment":false,"no_op_experiment4":false,"ios_25_unified_delayed_signup_test":true,"tree_en_ar_v1_experiment":1,"sg_hand_selected_incumbent_grammar_combined_experiment":true,"ios_18_use_characters_test":true,"no_op_experiment":true,"tree_de_en_v3_experiment":3,"ios_no_social_signup_experiment":true,"sg_hand_selected_incumbent_experiment2":true,"sg_hand_selected_separated_food_3_experiment":false,"sg_is_testing_user":false,"ios_24_autoset_timezone_test":true,"sg_test_challenge_probability_experiment2":"0.1","ios_24_no_discard_progress":true,"ios_23_local_grading_test":"control","sg_boost_skill_test_challenge_probability_experiment":false,"sg_always_add_learning_unit_test_challenge":false,"sg_hand_selected_separated_animals_2_experiment":false,"sg_hand_selected_separated_incumbent_basics2_1_experiment":true,"ios_china_cdn_test":false,"sg_hand_selected_separated_incumbent_basics_2_experiment":true,"ios_19_allow_sharing_test":true,"ios_21_history_test":true,"ios_31_local_grading_test":false,"sqrt_skill_strength_experiment":false,"ios_24_register_birth_year_test":false,"web_remove_quiz_experiment":true,"sg_hand_selected_separated_food_4_experiment":true,"kc_based_dynamic_tree_progress_experiment":false} ]

March 11, 2017

fkachinski

How do we switch from A to B or vise-versa?

Many people hate the new “health” and want into the other group!

Specially, using the iPhone app.

AldenDavidson

You can't affect your groups at all, those are set by Duolingo for the purposes of collecting data on new and potentially new features. Eventually they'll either move you back to the previous system - if they decide to not use the new "health" - or you'll stay permanently on the new system - if they decide it's working as intended.

fkachinski

Doesn’t it make more sense to move someone rather than loose them as a user?

AldenDavidson

That, unfortunately, would probably open up a pandora's box of problems. Not only would they have to hire on significantly more staff or spend significantly more time handling transfer requests, it would also potentially impact the size and makeup of their testing groups which would impact their results. Duolingo seems to be committed to making decisions almost completely data-driven, so while I sympathize with you I don't see them allowing that anytime soon.

CMatrix01

What is A/B Testing? I've heard of it but there isn't any explanation for it.

September 4, 2019

davidcwalls

Mod

Plus

2772

When software developers want to make a change or include a new feature, it is common to perform a randomized trial to see how well it works before deciding if it will be permanently included in the product. Users are randomly assigned to two groups: A and B. One of the groups gets the change, the other doesn't. The company then collects stats for some time to see how the two groups compare in whatever measures are deemed appropriate. In the past, Duolingo has indicated that they measure things like how long users stick with their learning, how regular they are with doing lessons, etc.

September 4, 2019

CMatrix01

Thanks.(That was really fast. It took you 8 minutes to answer)

September 4, 2019

CMatrix01

I get this message: TypeError: duo.user is undefined

Very cool thanks!

Wait, where do I type it in inside the source code? Oh, never mind.

mehlon

Ahh, I've been looking at the wrong object the whole time! I thought it was simply duo.ab_options. Thanks.

DamedesLangues

Hi! I'm curious to see what test groups I'm in, but my tech skills are non-existent. I've clicked on browser settings, then F12 developer tools. This is where I came to a sudden halt. I have no idea where exactly to type the command.

Whilst some helpful users have pointed out how to go about this on Chrome, Safari and Firefox, I can't find one explaining the procedure for IE. Could someone help me, please?

October 19, 2014

http://msdn.microsoft.com/en-us/library/ie/gg589530(v=vs.85).aspx

I don't have IE, so I can't demo for you, but maybe one of these links will help?

http://msdn.microsoft.com/en-us/library/ie/dn255006(v=vs.85).aspx

October 20, 2014

DamedesLangues

Thank you so much! I'll let you know how it goes. :-)

October 22, 2014

louis.vang

Where should i type the command

duo.user.attributes.ab_options ?

April 7, 2015

tomsherida

Far right hand side tab 'Console'

April 16, 2015

rooseveltnut1Plus

I hit F12 and got nothing but code. Oh well. No surprise. Duolingo doesn't like me much lately............;-)

June 15, 2015

cathgrl

I'm confused. Do you type the command at the filter output box? Because when I do that, nothing comes up.

I found it. Thanks!

Either I'm doing something wrong, or this doesn't work anymore. I tried it it Chrome, FireFox, and Edge. In all of them, I get errors that imply the "user" object does not exist.

March 25, 2018

2682

Still works for me. Make sure you're doing it from a tab with Duolingo open and logged in, then paste "duo.user.attributes.ab_options" (without quotes) in the console.

March 25, 2018

Deoxy144

Strange. I tried doing it when on various parts of the Duolingo website (logged in at the home page, in the middle of a lesson, etc.), and I just get an error. Here's the Chrome version of it: Uncaught TypeError: Cannot read property 'attributes' of undefined at <anonymous>:1:10

March 26, 2018