https://www.duolingo.com/profile/Rhabarberbarbara

Typing in Japanese - A/B test?

Hey there!

I'm just wondering how many of you still don't get the option to type in Japanese on web. I've seen pictures of people being able to switch between word bank and keyboard input. As I am still stuck with the word tiles, I wanted to find out, if it is an A/B test or just a bug. So how does it work for you?

Happy learning!

October 25, 2017

47 Comments


https://www.duolingo.com/profile/Rhabarberbarbara

My current theory is that it might be an A/B test, because when checking my extended user profile, I found web_tap_toggle (under informant_reference). The name sounds promising.

For me it shows: "treated":true,"destiny":"control","condition":"control","eligible":true

I know this has something to do with the japanese course, because on a second account I got treated:false because I was not doing either Japanese or Korean on that account. After adding Japanese it changed to the same as the above.

Web_tap_toggle is in a part of the user profile not visible to others so I cannot check on the profile of somebody I know can type in Japanese. Anyone want to check it for me?

October 25, 2017

https://www.duolingo.com/profile/testmoogle

Somehow I didn't notice your comment said all this until now. Strange. o.O

Anyway, I did a little more testing. I now understand how "treated" works. And I'm now also 99.9% certain that this web_tap_toggle is the specific A/B test determining whether or not a user is given the Japanese typing toggle button. ^^

Along with that first new trial account I made, I've now gone back and tested six more trial accounts. Here are the results:↓

(Account# = button? -- treated, destiny, condition, eligible

  • #1 = YES -- false, experiment, experiment, true
  • #2 = no -- true, control, control, true
  • #3 = no -- true, control, control, true
  • #4 = no -- true, control, control, true
  • #5 = YES -- true, experiment, experiment, true
  • #6 = YES -- true, experiment, experiment, true
  • #7 = YES -- false, experiment, experiment, true

I confirmed with #7 that "treated" is false until you've encountered a sentence you need to translate into Japanese. For #1 and #7, I looked at my user data first before I even clicked the start button in the placement test.

With #7, a few questions into the placement test I encountered a Japanese typing question, then exited without answering it, and checked my userdata again. It had changed to this: ↓

  • #7 = YES -- true, experiment, experiment, true

Nice to now understand what they mean by "treated". More importantly, this makes 12 accounts we now have results for in this discusson. All 12 results agree with our hypothesis:

For the web_tap_toggle A/B test:

  • experiment group = typing option toggle button
  • control group = stuck with tiles
October 25, 2017

https://www.duolingo.com/profile/Rhabarberbarbara

Thank you for this awesome analysis! I can add another account that agrees with the above data. Unfortunately (for me) it still means that my main account is stuck with tiles. I really hope this A/B test will end soon and bring the toggle to everyone.

October 26, 2017

https://www.duolingo.com/profile/DonutEggRoll

Hi testmoogle. In case you wanted another data point, my web_tap_toggle shows [true, control, control, true], and I do not have the option to type in Japanese nor Korean :(

October 26, 2017

https://www.duolingo.com/profile/testmoogle

Thanks. :)

Yeah, the web_tap_toggle A/B test affects all three of the languages I'm doing. My testmoogle account has the toggle button in all three; my droidmoogle account doesn't have target language typing in any of them. ^^;

October 26, 2017

https://www.duolingo.com/profile/Rhabarberbarbara

In case you are still following this discussion: my toggle button appeared and I now have

"web_tap_toggle":{"contexts":[],"treated":true,"destiny":"control","condition":"experiment","eligible":false}

Now I am once again confused about destiny vs. condition, but whatever! I can finally type on web!

November 9, 2017

https://www.duolingo.com/profile/testmoogle

It was 7~8 days ago that people on web began to receive the button to toggle Japanese typing.

The night people started posting about having it, I checked on my PC and it didn't have it yet. At work around midday I decided to try checking my mobile browser on my phone, and it now had the button. When I came home in the evening I checked on my PC again on the same account, yet it still didn't seem to have the button. Later that evening I tried again and now had the button.

So it didn't seem to me like something that was meant to be given to everyone all at once. However, I would've thought enough days have now passed for a rollout to have completed. Maybe they found a problem they wanted to fix part way through and temporarily paused the rollout? This screenshot I've taken below shows how it can be *slightly* problematic at the moment on mobile phone browsers. ^^; ↓

If it as an A/B test, it would be handy if someone could find the specific experiment string so we could check if we are one of the people who are affected by it. "web_tap_toggle" is the one that stands out most to me. I haven't checked if it's definitely the right one, but it's one that I'm affected by and I'm one of the people who have the toggle button on web... ^^

October 25, 2017

https://www.duolingo.com/profile/Rhabarberbarbara

Interesting, thanks. So what does the web_tap_toggle say for you? That you are in the experiment group?

October 25, 2017

https://www.duolingo.com/profile/testmoogle

I've just finished checking three of my accounts (testmoogle, droidmoogle, and droidmoogle2) and also created a new trial account.

testmoogle and the new account both have the toggle button; droidmoogle and droidmoogle2 don't have it.

web_tap_toggle (treated, destiny, condition, eligible)

  • testmoogle = true, experiment, experiment, true
  • (new account) = false, experiment, experiment, true
  • droidmoogle = true, control, control, true
  • droidmoogle2 = false, control, control, true

So, yep, both accounts in the experiment group have the toggle button and the two that are in the control group don't. I thought it was necessary to be "treated" as well as in the experiment group, but the new account has treated=false?

Another thing I noticed was in the "ab_options" section there is another string that tallies perfectly with the results ↓

"web_remove_quiz_experiment":false

  • testmoogle = false
  • (new account) = false
  • droidmoogle = true
  • droidmoogle2 = true

I assume this "web_remove_quiz_experiment" is the one to look out for, since it is in the "ab_options" section unlike that other string. Would be interesting if this matches other people's findings.

The main thing though is that this should be enough info to say that the toggle button being there or not is simply due to an A/B test rather than a bug. ^^

(* EDIT * Just noticed I'd typed true and false the opposite way round in that last list. Corrected now.)

October 25, 2017

https://www.duolingo.com/profile/Rhabarberbarbara

Web_remove_quiz_experiment shows false for me so the correlation doesn't hold. And while I don't quite understand the difference between ab_options and informant_reference, there definitely are several A/B tests that I am part of in the informant_reference part.

I think the remove_quiz experiment was true for those who lost the progress test before it was removed for everyone.

October 25, 2017

https://www.duolingo.com/profile/testmoogle

Ah, ok. ^^

Its name didn't really correspond well with this feature anyway, so I'm not all that surprised. Glad I can rule that one out. In fact, thinking about it, I'm sure I remember seeing web_remove_quiz_experiment in the list months ago.

Would be nice to know what the difference between the ab_options and informant_reference sections are. I don't know either. I guess I'll regard them as one in the same for now. Would be handy to understand what exactly "treated" means too, as to me it sounds pretty vital.

web_tap_toggle on the other hand has a much more promising sounding name, and I don't remember seeing it in the list before. You're in the control group for this, right?

October 25, 2017

https://www.duolingo.com/profile/Rhabarberbarbara

I'm pretty sure I saw the quiz option several months ago as well.

I guess I don't need to answer the treated part, since you have solved it (also to the best of my knowledge) in your other post.

And yes this account is in the control group for the tap toggle... pity

October 26, 2017

https://www.duolingo.com/profile/garpike

I have Japanese typing on the web and tiles on the app, but no toggle button in either.

October 25, 2017

https://www.duolingo.com/profile/testmoogle

Oh? That's pretty interesting!

I don't expect the app to ever get Japanese typing. I'm not surprised in the slightest that you don't have a toggle button in it.

However, for what you said about web, I am very surprised that it sounds like there's another A/B test going on where some people have only Japanese typing with no toggle button for escaping to the safety of the silly tiles!

So this is kind of like an A/B/C test instead. Of our three groups, I really hope your team win! :P

I do wonder what specific A/B test string yours comes under now though...

October 26, 2017

https://www.duolingo.com/profile/garpike

I don't expect the app to ever get Japanese typing. I'm not surprised in the slightest that you don't have a toggle button in it.

I mistook your mobile browser screenshot for an app screenshot. I'd have thought the app would be an ideal place for a toggle button, particularly as DL can set it as off by default so as not to frighten any horses that might have downloaded it.

I do wonder what specific A/B test string yours comes under now though

"web_tap_toggle":{"contexts":[],"treated":true,"destiny":"control","condition":"control","eligible":true}

"sg_only_tap_challenges":{"contexts":[],"treated":false,"destiny":"experiment","condition":"control","eligible":false}

This would seem to support your theory on 'web_tap_toggle' relating to the button. I'd hazard that 'sg_only_tap_challenges' might need to be condition:experiment for the tiles to show up at all; what does yours say?

October 26, 2017

https://www.duolingo.com/profile/Rhabarberbarbara

Interesting. However so far I thought only_tap_challenges did not refer to Japanese (only), but to all courses. Meaning you wouldn't get to type at all... Any idea what sg stands for? The beginning usually refers to the platform but I have no idea about sg...

Mine says: "sg_only_tap_challenges":{"contexts":[],"treated":true,"destiny":"control","condition":"control","eligible":false}

October 26, 2017

https://www.duolingo.com/profile/garpike

I don't know what 'sg' signifies; there are lots of sg-prefixed tests and they don't seem obviously related only to Japanese or Korean. Clearly my hypothesis was wrong, however, as you have 'condition:control' and still have tiles on the web.

There are various other tap-related tests with mobile prefixes that I presume affect all courses on mobile; the rest appear only to affect Japanese & Korean on the web—unless, of course, anyone is getting tap-the-pairs exercises for other languages here?

Can someone remind me what 'destiny' is in this context? I have a feeling I understood it once, but I've no idea at the moment. I suppose this is because I'm not especially fascinated in the minutiae of how Duolingo works, but merely periodically interested in solving specific problems that crop up with regard to this.

October 26, 2017

https://www.duolingo.com/profile/testmoogle

Interesting that this says eligible:false but treated:true.

I think this most likely means sg_only_tap_challenges was a previous test that's no longer running. For freshly created accounts, I get 0 results regex searching "tested":true.*"eligible":false, but my normal accounts each return around 20~40 tests like that. My oldest account has the most of these, since it's had the most time to be treated to a larger number of A/B tests.

(Oh yeah, and I have no clue what "sg" means either.) ^^;

October 26, 2017

https://www.duolingo.com/profile/testmoogle

Sorry, I've been really ill spending most the day in bed. Here's what mine says:

"sg_only_tap_challenges":{"contexts":[],"treated":false,"destiny":"experiment","condition":"control","eligible":false}

I see it's already been figured out that this test doesn't determine whether a user has typing or tiles. A bigger giveaway though is that it says treated:false and eligible:false. ^^

October 26, 2017

https://www.duolingo.com/profile/garpike

Some people certainly have the new Dutch tree, but my 'tree_nl-NL_en_v2_experiment' is both treated:false and eligible:false.
I'm wondering if it might one of the various tests that begin 'sg_rewrite_'.

October 27, 2017

https://www.duolingo.com/profile/Rhabarberbarbara

Can you check your extended user profile for what it says after web_remove_translation_strings? The name sounds promising.

Mine says: "web_remove_translation_strings":{"contexts":[],"treated":true,"destiny":"control","condition":"control","eligible":true}

October 26, 2017

https://www.duolingo.com/profile/garpike

"web_remove_translation_strings":{"contexts":[],"treated":true,"destiny":"control","condition":"control","eligible":true}

Exactly the same as yours.

October 26, 2017

https://www.duolingo.com/profile/Rhabarberbarbara

Hmm what a pity. I thought it sounded nice.

Also I can't answer to your question above about destiny. So let me do it here: My theory is that this has something to do with rollout or change in conditions, so condition would be the current state and destiny the one that's soon to come. I never bothered to find out if it's true.

October 26, 2017

https://www.duolingo.com/profile/testmoogle

I'm still perplexed that you have Japanese typing on web without a toggle button. You're the only person I've noticed say this in any of the discussions I've read so far. If you hadn't already posted your web_tap_toggle experiment values, I would try seeing if you could double check that you've not overlooked having the toggle button (since it is fairly easy to miss on large screens). But your web_tap_toggle values indeed show that you really don't have the button.

For my three accounts and the eight trial accounts I've gone through so far, six of them have Japanese typing with the toggle button, five are stuck with tiles, and zero have been like your situation. Without having any accounts of my own in the same situation as yours, I can't really test what might be the cause of it. Would be fun to have a go at figuring it out if only I had such an account. ^^;

To me, with your web_tap_toggle values and the fact I've not heard anyone else mention they have no toggle button to bring tiles back, it seems like you should instead be seeing tiles only without a toggle button (the same as Rhabarberbarbara) rather than having Japanese typing only. I'd almost think this might be a bug rather than an A/B test for how rare it seems to be!

October 27, 2017

https://www.duolingo.com/profile/testmoogle

I've had a thought. I had been doing the Japanese course on web for over five months before it was officially released on web. During that time it didn't have tiles and only had Japanese typing.

I only got tiles when it switched from the old lesson design to the new one, which happened when Japanese officially released on web 17 days ago, where it no longer allowed me to type Japanese answers. Then, around a week later, the "use keyboard" toggle button appeared, allowing us to toggle between keyboard and tiles for Japanese answers.

If you are somehow still on the old lesson design for Japanese on web, then you wouldn't have tiles and nor would you have the toggle button! If this were your situation, you would have Japanese typing with no toggle button! ^^

Does it look like this for you, where there's Japanese typing without a toggle button? ↓


(click images to see full size)

Here's what it instead looks like in the new lesson design, which everyone should be on (this particular case having the toggle button): ↓


Or does yours look like the second of those three images minus that "use word bank" button?

October 27, 2017

https://www.duolingo.com/profile/garpike

It looks like this:

I did have the tiles on the web for a week or so, which is why I complained about them so bitterly on the 'Japanese surprise' thread. I certainly have the new design; I think everyone does.

October 28, 2017

https://www.duolingo.com/profile/testmoogle

Thanks. ^^

I've seen your posts in those threads. I should really have been able to answer my own question with what you said in the "First translate to Japanese, with typing" thread. But it's good to now have a concrete screenshot of it anyway.

I agree it's almost certain that everyone is now on the new design, but the old design hasn't been deleted yet. Along with my second and third ones, I also took the first of my three screenshots yesterday too! (as can be seen from it showing my "185" day streak) ^^

However, since that moment just over two weeks ago when the course officially released on web, the course now no longer works correctly on the old design. The character challenge exercises are now fully embedded into the course, and the old design can't load these new types of exercises. So it gets stuck whenever a character challenge question comes round, and then the only way to proceed is to quit the lesson/practice session.

So, what you said in that other thread about still having tiles in the pair matching exercises would mean you couldn't possibly be on the old lesson design. I had been thinking that maybe you had an A/B test that was overriding another one, somehow putting you back on the old design.

I'm thinking now that maybe I didn't complain bitterly enough in that 'Japanese and Korean Surprise' thread... Since you have a streak of over 800 days, are learning 30 languages, and have received over 100 upvotes on that complaint comment, maybe therefore Duolingo decided to pull a one-off and give you Japanese typing without the toggle button! :P

Maybe they could have done this hoping to keep you happy. Or possibly trying to prove a point, hoping you will delete your complaint and beg to be given the tiles back! (Just kidding about this... Well, mostly anyway...)

I'm still a bit envious of your A/B test. I do half-believe they really gave your account a unique A/B test, since I've still not seen anyone else say theirs is like this. But maybe I'll have a few more goes at making trial accounts now that I've seen a screenshot. ^^

* Update * Okay, I went through at least ten more trial accounts without any luck. Even if this A/B test might not really be unique to your account, there's still the other possibility that the eligibility period for this A/B test has passed and new accounts would therefore not even be able to be treated with it anymore. ^^;

October 28, 2017

https://www.duolingo.com/profile/garpike

I think it is very unlikely to be unique, as I've never heard of Duolingo ever doing anything like that for individual users (an individual user is one out of 150 million, so it would seem to be an odd business decision in time-management), and many other people have lots of flags and have made posts complaining about all sorts of things that get lots of upvotes. I try to be not unhelpful on the forum, but many other users do far more than I do, and none of them, to my knowledge, have ever been given any special perks or dispensations of the kind you are suggesting.

This account is quite old, however (2013), as there is certainly precedent for A/B combinations on older accounts being impossible to replicate on newly-created ones (I see no ads or gems, too, and I am always in the very last group when a feature like immersion or streams is phased-out—truly I live a charmed existence!)
Having said that, I just tried the Japanese course from another account I created in 2015 and got only tiles and no toggle button†—the web_tap_toggle settings there were identical to mine on this account. I'm not sure I ever made any older accounts I can try, or indeed remember what they are if they exist.

How are you getting the old design to display, out of interest?

†Interestingly, in both that account and this, it always says 'Translate this text' in these exercises, rather than 'Write this in Japanese', as in your screenshots.

October 29, 2017

https://www.duolingo.com/profile/testmoogle

@garpike - That's very intriguing what you've pointed out about it saying "translate this text" in your screenshot! The text is the same in my screenshot where I was using the old lesson design, except you are seeing it in the new lesson design!

I wonder if you've somehow only got the stylistic changes of the new lesson design while some other aspects have kept how they were on the old one (such as typing in Japanese)? ^^

I can see how you wouldn't have known about the toggle button from the beginning, since most people likely just used the button once and stayed on typing thereafter without mentioning it. I believe almost everyone in that First translate to Japanese, with typing thread who received Japanese typing received the button simultaneously—the same as I did. So the button started appearing at the same time that you were given only Japanese typing, nearly two weeks ago.

As for how I'm getting the old lesson design to display... During that week or so after the course officially released on web where tiles completely replaced all typing in Japanese, I was pretty frustrated and came up with loads of crazy ideas trying to fix it myself. Some of these ideas actually worked and so I ended up learning a few neat tricks.

There are two tricks necessary to be able to use the old lesson design for Japanese. The second trick is the really cool part, but annoyingly is too easy to be misused by unscrupulous people for it to ever be a good idea to even describe it (and it's a bit complicated anyway)... However, the first trick is harmless to describe and you will likely have seen it mentioned in the forums already for other uses:

duo.old_web_url_whitelist=["^.+"]
↑ Simply use that in your browser console whilst on the Discussion page, then click Home and do Strengthen Skills. ^^

However, you won't be able to use this properly as it will freeze whenever a character challenge question comes round (like I described in my other post). This is enough to simply be able to "view" the old lesson design though.

October 30, 2017

https://www.duolingo.com/profile/garpike

Incidentally:

I should really have been able to answer my own question with what you said in the "First translate to Japanese, with typing" thread.

No-one on that thread ever mentioned anything about a toggle button; I assumed that they all saw what I saw, and it had simply switched back to typing boxes. This thread was the first I heard of toggle buttons. When exactly did they appear?

October 29, 2017

https://www.duolingo.com/profile/Rhabarberbarbara

So you are saying complaining bitterly might help? ( :

By the way there seems to be at least one other account in garpikes A/B test. Further down in this thread JeffryEvan commented that they have typing but no toggle button on web.

October 29, 2017

https://www.duolingo.com/profile/testmoogle

@Rhabarberbarbara

By the way there seems to be at least one other account in garpikes A/B test. Further down in this thread JeffryEvan commented that they have typing but no toggle button on web.

Oh! Don't know how I missed that post until now!

That account is older than my three accounts, but only "created":"1 year ago". Not a particularly old account unlike garpike's. Well, now I'm even more confused about it! XD

October 30, 2017

https://www.duolingo.com/profile/garpike

That account is older than my three accounts, but only "created":"1 year ago". Not a particularly old account unlike garpike's.

The 'created' value is unreliable (much like comments on the forum '1 year ago' that were actually posted two or three years ago). Mine says '3 years ago', but 'creation_date_new' gives the correct date in 2013 (which, obviously, cannot possibly be three years ago). So that account could potentially be two or three years old.

October 30, 2017

https://www.duolingo.com/profile/syntaxosaurus

I was lucky enough to get to the option to type. Hope that isn't taken away--which I could imagine happening, if the "tiles only" user group sticks with the course a greater percentage of the time.

You know, Duolingo Plus would start to look tempting if it offered the ability to opt in/out of the A/B tests. Of course, then your data would be unusable for purposes of the test, but Duo would still be able to experiment on all the non-paying customers. (...that sounds kinda creepy, when I put it like that.)

October 25, 2017

https://www.duolingo.com/profile/FilipFilip17

I don't have the option to type in Japanese, but I don't know my A/B test group. I remember that I used to be able to access the list of properties by entering “duo.user.attributes.ab_options” in the console, but that doesn't seem to work any more.

October 25, 2017

https://www.duolingo.com/profile/testmoogle

I just tried doing exactly what you said and it worked perfectly fine for me. You click the result and it expands with a list of all the ab_options.

There are a number of different ways to find the data. The way I was using was to simply go to my user data page. ↓

https://www.duolingo.com/users/FilipFilip17

(Other people would need to replace where it says FilipFilip17 with their own username though. And certain sections like ab_options and informant_reference are only visible for the account you are logged in as. I can't view which A/B tests you have.)

Using your method, web_tap_toggle is in a different section to ab_options. You'd need to type "duo.user.attributes.informant_reference" instead and expand that result (or simply type "duo.user.attributes.informant_reference.web_tap_toggle" to get that specific bit of data.) ^^

October 25, 2017

https://www.duolingo.com/profile/FilipFilip17

Thanks for the website link. I didn't know about it. ^^

As for my problems with the console, it seems to work only on Discussion pages, if you tried to use it on Duolingo home page, you get an error.

October 25, 2017

https://www.duolingo.com/profile/testmoogle

Oh yeah, I forgot about it not working when on the home page.

What result did it give you for duo.user.attributes.informant_reference.web_tap_toggle ?

If you still don't have the option to type in Japanese, it seems it should say you're in the "control" group for this, but it would help to have a few more people test this to confirm it. ^^

October 25, 2017

https://www.duolingo.com/profile/FilipFilip17

Yeah, that’s right, I’m in the control group. I hope the test won’t last for too long. In the meantime, I’ve found this userscript that saves me the effort of using a mouse. Perhaps it’s better this way for the time being. It gives me the opportunity to use a simpler input method before moving on to a more advanced one. ^^

October 25, 2017

https://www.duolingo.com/profile/JeffryEvan

i got that about a week ago in web (usually i do practice in mobile) and confused since i don't have japanese keyboard. try to type the romaji but it marked as wrong answer, there's no toggle button either so i skip it

October 26, 2017

https://www.duolingo.com/profile/Rhabarberbarbara

Can you check your extended user profile for what it says after web_remove_translation_strings? The name sounds promising.

Mine says: "web_remove_translation_strings":{"contexts":[],"treated":true,"destiny":"control","condition":"control","eligible":true}

October 26, 2017

https://www.duolingo.com/profile/LuisRodrigoRuiz

I am on two different computers; on my Windows 7 desktop I get to type in Japanese. On my Windows 10 touchscreen, it only lets me use the tiles. Neither offer a toggle.

October 26, 2017

https://www.duolingo.com/profile/Yannis_Ch

Hi, Rhabarbebarbara (the german tong-tie, right)?

If I understood correctly, you need a way to type in japanese, don't you?

Try to download this: GoogleJapaneseInputSetup

It works everywhere, when typing.

October 27, 2017

https://www.duolingo.com/profile/Rhabarberbarbara

Hi Yannis,

I must be famous, everyone has heard about me ( :

Thanks for your recommendation, however you misunderstood the aim of the discussion. On the duolingo Japanese course some accounts have to translate into Japanese with typing while other accounts get to order whole Japanese words. Some accounts are allowed to switch between these input modes, others are not.

My account does not allow Japanese typing. I can of course type in Japanese on my computer if I want to, but not in duolingo.

So I started this discussion to find out if this was a bug or an A/B test. Turns out it is a combination of two A/B tests.

October 27, 2017

https://www.duolingo.com/profile/Elgatobandido

I don't have the option to type in Japanese. Hopefully that will change soon.

October 27, 2017

https://www.duolingo.com/profile/stuffradio

It defaults to typing for me, except when I'm on the match words exercise. When I'm supposed to type Japanese, on my Mac I can easily toggle between all my keyboards I have added for my languages I am working on.

October 27, 2017

https://www.duolingo.com/profile/testmoogle

So your situation is the same as mine then? It lets you type Japanese answers but also gives you a "use word bank" button at the bottom of the screen to switch to tiles?

Does yours look like this and have a "use word bank" toggle button? ↓

It remembers whether you chose to type Japanese or use the tiles the next time you have to translate another sentence into Japanese. If you instead click the "use word bank" button and answer using the tiles, from then on it will default to showing tiles until you decide to toggle back and type an answer.

October 27, 2017
Learn Japanese in just 5 minutes a day. For free.