https://www.duolingo.com/Susanna35

New A/B test?

  • 25
  • 20
  • 17

I know other people have had a similar complaint in the past, but it just happened to me. I finished my tree at level 16, and have been going through all the lessons from the beginning, as well as keeping my tree refreshed. I have now just finished level 18. But just a few lessons ago, I started getting fewer questions. Previously, if I didn't miss any, I was getting 17 questions on each lesson. Now it is variable, but it can be as few as 7, and I haven't seen higher than 13, even when I missed a question. And the questions in general seem to be the easiest ones. How am I going to absorb the language if I only get a few easy questions? Previously, my experience in refreshing gave me harder questions than I had had in the first execution of the lessons, particularly in the higher levels. Please, give me back my 17 questions.

June 10, 2015

43 Comments


https://www.duolingo.com/EdgarChavezIII

Give Susanna35 back her dang 17 Italian questions!

A new A/B test to see if Duolingo can achieve higher retention rates by simply teaching people less. Most people do not want to learn stuff, so if Duolingo teaches less it will be more popular. Great!

June 10, 2015

https://www.duolingo.com/Susanna35
  • 25
  • 20
  • 17

Thanks! I'm here to learn, not to play around. It seems to me that Duolingo should try to make it more attractive for people who want to learn. It's the only reason I can see to take the course. Sure, even with the simple questions, I make a lot of mistakes - typos often, but I think just plan stupid mistakes more often. But at least I am learning. And to jlottey, I don't mind the repetition so much, but I, too, would prefer more repetition on the questions I miss than on the ones I get correct. In any case, repetition helps improve the memory on those repeated, whether right or wrong.

June 10, 2015

https://www.duolingo.com/Myron1313

A dummy down curriculum is not acceptable. If Duolingo is having an issue with curriculum delivery they should address the issue, not lower the bar. This is ridiculous. I don't accept this type of behavior from my town's school district, I should not be forced to endure it here. I just completed a lesson of seven examples/problems and earned 10 XPs. I did not do 10 XP's of work. I am not here to simply advance through levels faster, but to learn language skills. If there is a retention issue, perhaps a duel advancement track needs to be developed. Otherwise, I really have to question the validity and purpose of the site. It certainly appears, in my opinion, that the mission of the site as identified in the "about" page ("We believe everyone should have access to education of the highest quality - for free.") is not being achieved by this behavior. I have also noticed that the variety of examples/problems within the lesson is reduced, creating a depth issue.

June 11, 2015

https://www.duolingo.com/Susanna35
  • 25
  • 20
  • 17

Another problem with this new setup is that all the translations seem to be one-way - from language-2 to l-1.

June 11, 2015

https://www.duolingo.com/pir_anha

This is actually the even bigger problem in my opinion. It makes me feel like my time here is wasted because it's pedagogically unsound to give us so few opportunities to produce our L2 language. I already know English, I don't need to practice typing it. I'd never heard of these A/B tests before yesterday, but I can already see they're going to ruin the site for me. I take my language studies seriously, and all the razzle-dazzle with lingots and the faux-fluency thingamajig doesn't impress me at all. Give me more vocab and more L2 production practice instead.

Fortunately I wasn't planning to rely on duolingo alone anyway.

June 13, 2015

https://www.duolingo.com/northernguy
  • 25
  • 25
  • 19
  • 659

Nor should you rely on it for anything other than what it specializes in.

I'm not getting a steady diet of either what I call long form lessons nor the new short form. I'm not sure what criterion they are using for making the choice of what they deliver to me. Since I am on the third time through the tree in French, but just beginning in others, it could well be different for me than other students.

June 13, 2015

https://www.duolingo.com/Susanna35
  • 25
  • 20
  • 17

You have covered it very well. How can you learn if you just get the easy questions, and only a few of them? I still make a lot of "stupid" mistakes, along with the occasional typo, but that doesn't mean I'm not learning. I wouldn't be on this site if I didn't want to learn, and if it gets much worse, I'll just go back to using the translators, supplemented by my dictionary, grammar book and idom book.

June 11, 2015

https://www.duolingo.com/ClarkStephen
Plus
  • 25
  • 24
  • 24
  • 11
  • 7
  • 7
  • 6
  • 5
  • 766

Well, I'm going to take the perhaps unpopular position of suggesting that we not be able to opt out of the A/B tests, even though I have been on the receiving end of two that I hate: not getting lingots for streaks, and fewer questions per lesson. I think they are experimentally trying to determine what motivates people to learn the best and if that's true, they need random samples. If they let us opt out, the samples wouldn't be random.

June 10, 2015

https://www.duolingo.com/Susanna35
  • 25
  • 20
  • 17

Some of the tests are no big deal, but this one is. Why should you take lessons if not to learn? It's a serious irritant. Anyway, if you opt out, they should consider that a "vote" for the status quo.

June 10, 2015

https://www.duolingo.com/ClarkStephen
Plus
  • 25
  • 24
  • 24
  • 11
  • 7
  • 7
  • 6
  • 5
  • 766

No argument from me: I agree with you completely. Perhaps they are thinking that some users find 17 questions too long, and they are trying for more "bite-sized chunks."

However, I've noticed that even 17 questions doesn't cover all the material contained in a given unit, and so I have come to rely on the strengthening practice to get all the material.

Best of luck in you language studies...

June 10, 2015

https://www.duolingo.com/Arnauti
  • 25
  • 25
  • 25
  • 25
  • 25
  • 21
  • 18
  • 16
  • 13
  • 11
  • 11
  • 9
  • 9
  • 6
  • 6
  • 2
  • 808

I sort of agree with both of you, but I'd like to point out, although it may be unethical somehow to say this, that in practice there is sort of a way of opting out: Create an extra account, and whenever the current bucket you're in drives you mad, turn to the other account. If your second account ends up in the same bucket as your first, create a third oneā€¦ I realize Duo won't like this, but I also feel that the amount of A/B testing here will sooner or later drive people to adopt this strategy.

June 12, 2015

https://www.duolingo.com/ClarkStephen
Plus
  • 25
  • 24
  • 24
  • 11
  • 7
  • 7
  • 6
  • 5
  • 766

I have also noticed that I am receiving fewer questions perhaps it is - as you suggest - a new A/B test.

June 10, 2015

https://www.duolingo.com/Snommelp
  • 24
  • 22
  • 22
  • 21
  • 21
  • 20
  • 18
  • 17
  • 14
  • 4

I got that a while back, then it went away, and now it's back again. I agree, it's very frustrating. I mean, I understand the theory behind it: if you're able to get all of the questions right without looking at the hints, it probably means you know the material. But in reality, it could just mean that you were able to figure out those seven questions from context (especially if they're softballs), which might not even include all of the material for that lesson.

June 10, 2015

https://www.duolingo.com/Snommelp
  • 24
  • 22
  • 22
  • 21
  • 21
  • 20
  • 18
  • 17
  • 14
  • 4

I'm actually wondering if it might even be a glitch, because I just went through one where I did 17 without a single mistake or look at the hints, and my progress bar was only half-filled.

June 10, 2015

https://www.duolingo.com/Susanna35
  • 25
  • 20
  • 17

I wish they'd give you a heads-up for these tests, although people who are a little more familiar with the way the system works have pointed out a way to find out. I didn't write it down, and don't know where to look. Must be something in the user profile. But I wish they'd let you opt out of a "test." I have yet to see anyone defend the lower number of questions.

June 10, 2015

https://www.duolingo.com/rspreng

If people can choose whether or not to be in a test, the test loses validity. The point of testing is to see what results in people learning more, faster. Nothing to do with what we may, or may not, prefer.

June 10, 2015

https://www.duolingo.com/Elorac72
  • 22
  • 15
  • 14
  • 13
  • 10
  • 10
  • 3
  • 3

https://www.duolingo.com/comment/5023812

I never bothered before, but there have been so many changes recently that I thought I'd take a look. Figuring out what the tests actually are is another matter. I'm guessing that those starting sg refer to your experience on any device, as all the others seem device specific.

sg_limit_translation_challenges_experiement is set as "moderate" for me, so I assume there are at least three levels of this.

sg_adaptive_lessons_experiment is set to "true" That at least sounds like a good thing!

June 12, 2015

https://www.duolingo.com/Susanna35
  • 25
  • 20
  • 17

I tried to look, but got nothing. Not sure I put the command in the right place. Can you tell me where to put it?

June 13, 2015

https://www.duolingo.com/Elorac72
  • 22
  • 15
  • 14
  • 13
  • 10
  • 10
  • 3
  • 3

Hi Susanna.

It took me a bit of fiddling about as well. I'm not sure whether the browser affects the way it looks, but for me (using Firefox), I pressed F12 and then had to click on the Console tab. Then there is a box at the bottom of the screen with a >> symbol, and I typed in there. Copying and pasting didn't work. For some reason it stuck when I was typing 'user' but clicking away from the box and going back in solved that.

Once I'd typed in duo.user.attributes.ab_options there was a list ending in '61 more...' Clicking on '61 more...' brought up a scrollable list on the right.

Incidentally, I read somewhere else that sg_ means 'session generator', i.e. attributes that control how each lesson is generated, regardless of platform.

Hope that's helpful!

June 13, 2015

https://www.duolingo.com/Susanna35
  • 25
  • 20
  • 17

Thanks! Got it to work. It's hard to know what they all mean, but two that I saw might either one have to do with this situation: sg_adaptive_lessons_experiment and sg_simpler algorithm_experiment. However, the short lessons doesn't always seem to apply to a "normal" refresh. And in some of the more difficult grammar sections, I make so many mistakes (even if they are stupid ones that I know better than) it's not making a lot of difference, and there don't seem to be many really "simple" questions in them. Like some of the non-grammar-related ones where you have only to translate a noun with its article.

June 14, 2015

https://www.duolingo.com/northernguy
  • 25
  • 25
  • 19
  • 659

Excellent link. Very informative. It and the associated comments open up an aspect of Duo I was entirely unaware of. I am surprised that it can be accessed and modified.

Since I use multiple learning sources, I have no intention of messing with Duo's model what ever changes they make. They have made many in the last couple of years, some of which I liked and others that I didn't like but are now lost in the swirling mists of my memory. The more it meets my needs the more I will focus on it over other approaches. The less it meets my need the more time will be spent elsewhere.

Duo can measure the success rate of different approaches to learning vs. how many students vote with their feet. They do have a business model. I have no idea how much A/B testing to determine best learning plans fits into their model of how to stay alive.

It's them staying in business and me getting a great free service. Like much of the internet, I don't really understand how it all can work. The technology is simple enough but how the small pittance I pay for access can provide all that is available to me for free is beyond my comprehension. Duo being one of the those free services that are just there for the taking.

I let them do what they do and I do what I do. A lot of the time it is the same thing.

June 13, 2015

https://www.duolingo.com/Elorac72
  • 22
  • 15
  • 14
  • 13
  • 10
  • 10
  • 3
  • 3

I don't think it really can be modified that much; the attributes are reset by the server when the pages reload. So for things that affect the look of the site you can see what it looks like as a one-off, and some people wrote browser add-ins that I'm guessing kept resetting the visual attributes, but the way sessions are generated will always come from the server. I've not played around with it because I want to use my time here to learn languages other than html.

I'm not too keen on the way the site keeps changing, because I just work out how to best make it work for me and then it seems they make it work differently so I have to find how best to make that change work for me. I was more annoyed the first time it happened to me because I wasn't aware they did this and because the particular change they made was taking away lingots for streaks, when I'd just decided to focus more on lingot score as an extrinsic motivator.

Now that I know they do this and that the experience and utility of Duolingo will vary, I'm a bit more prepared for it to change. Even if it changes in a way I don't like, if it had been in that state when I first came to it I'd still think it was an excellent tool, especially for what we (don't) pay for it! And as well as learning languages, I also learn better to accept change..

June 13, 2015

https://www.duolingo.com/northernguy
  • 25
  • 25
  • 19
  • 659

I wondered how it could be such a simple procedure to change how the site functions for individual users.

June 13, 2015

https://www.duolingo.com/Kevins121

Happened to me as well. Don't like it either - maybe doing each lesson twice now will mimic the effects of the older longer lessons.

Hopefully after the test, if they implement the new lesson style, we will be able to choose standard test length (10,20,30 questions etc.) I think that would be a good idea for everyone.

June 10, 2015

https://www.duolingo.com/StrapsOption
  • 23
  • 18
  • 14
  • 14
  • 11
  • 11
  • 9
  • 8
  • 8
  • 7
  • 7
  • 6
  • 5
  • 4
  • 3
  • 2
  • 2
  • 2

I said the exact same thing in another thread:

I know some people clearly don't like it, and I agree, for languages like German and Italian it's seems like a bit of a cop-out, and for Norwegian it's just useless. But I was doing some Ukrainian today which is obviously a much harder language, and it really helps if the lesson is shorter. You don't have to take in as much information, and I feel I'm learning much better.

Perhaps it should be limited to harder languages like Ukrainian.

June 11, 2015

https://www.duolingo.com/Susanna35
  • 25
  • 20
  • 17

Perhaps it should be varied depending on the language. But if repeats would give us more of the tougher questions, instead of just a lot of simple questions, we'd be getting more for our efforts. These "short" lessons don't come close to covering all the words that are supposed to be in them. If the "short" lessons are better - and I don't have a lot of trouble with repetition, that's the way most of us learn - why not split the lessons up? And perhaps give fewer point per lesson, but that doesn't really matter. What matters is learning. But, as I said, it does rather irritate me that I am getting tons of points, which gives a false impression of how much I'm learning, because most of the questions are just easy stuff. And although it's not terribly difficult, Italian isn't the easiest language in the world to learn.

June 11, 2015

https://www.duolingo.com/jlottey
  • 25
  • 12
  • 4

Interesting, Susan, I am still on level 17 and although I seem to have the same question come up even after they have been correctly answered a number of times. I lament the lack of opportunity for dialogue because I am certain my phonetics are poor.

June 10, 2015

https://www.duolingo.com/Susanna35
  • 25
  • 20
  • 17

I just zapped right through the complete medical topic (with slow dial-up, and stopping to view discussions on several questions) in less than half an hour. Sometimes it takes almost that long to do a single lesson, if there are tough questions. I am racking up points like crazy, but not learning much of anything.

June 10, 2015

https://www.duolingo.com/minookamuse
  • 22
  • 17
  • 17
  • 561

Are you nearing the end of your tree? I noticed the ones at the end, especially the last two are much shorter than all the others.

June 10, 2015

https://www.duolingo.com/Susanna35
  • 25
  • 20
  • 17

I finished my tree long ago, and am going through all the lessons again. I never noticed the last ones being shorter. This just happened suddenly. And, oddly enough, when I did a normal "refresh" on a topic that said it needed refreshed, I got the full complement of questions. I haven't done anything else since then, so I don't know whether or not anything has changed.

June 10, 2015

https://www.duolingo.com/minookamuse
  • 22
  • 17
  • 17
  • 561

When I got to the last 2 or 3 lessons at the beginning of May, there were only 9 questions in each lesson. It made finishing the tree pretty quick! But all my refresh lessons are normal length now. If it is an A/B test, I agree with you, I would not like shortened lessons. :-(

June 12, 2015

https://www.duolingo.com/Najeka

BRING BACK THE LENGTH!!! I have noticed this sometimes, but I havent had it nearly as badly, and sometimes it gives me extra. I will say that using timed practise can bee a good desision if you are using it as a refresher (you will have a longer lesson if you know it well, but shorter if you make mistakes), but that will also give you lots of points, which you said you didnt want.

Edit: by saying BRING BACK THE LENGTH i meant that mainly for Susanna, because she has it much worse than myself, but constant long lessons would be nice for everyone.

June 11, 2015

https://www.duolingo.com/sigmacharding
  • 18
  • 17
  • 15
  • 15
  • 14
  • 12
  • 11
  • 11
  • 11
  • 11
  • 11
  • 10
  • 10
  • 9
  • 6
  • 5
  • 4
  • 3
  • 3
  • 3

I love the new shorter lessons- sure I just repeat them when I am not feeling comfortable or need more practice- this new system gives people more freedom to move faster through the tree if they want to and they can repeat if they need to.

June 11, 2015

https://www.duolingo.com/Susanna35
  • 25
  • 20
  • 17

But if they repeat, they keep getting the easy stuff over and over again. I had already completed my tree, and I'm getting the easy stuff during the repeats. I used to get new and harder questions on the "refresh" cycle, so I was still picking up new stuff. It looks like the learning is being extremely limited with this system.

June 11, 2015

https://www.duolingo.com/sigmacharding
  • 18
  • 17
  • 15
  • 15
  • 14
  • 12
  • 11
  • 11
  • 11
  • 11
  • 11
  • 10
  • 10
  • 9
  • 6
  • 5
  • 4
  • 3
  • 3
  • 3

Definitely I agree that this is a bad idea. The lessons should be more challenging when going back over skills.

I personally prefer the shorter lessons and find them more effective but it is a pain if the material is not challenging enough also.

Hopefully this issue will be addressed soon.

But in my opinion duolingo will only bring somebody so far before other resources should be used. For example duo teaches reading and writing pretty well but it is often lacking on good listening material and it is very hard to learn to speak well.

I hope that all of these shortcomings are addressed as the website improves but for now here is a podcast which I find to be easy listening for Italian. http://www.podclub.ch/index.php/component/podcastmanager/?format=raw&feedname=12

June 11, 2015

https://www.duolingo.com/Susanna35
  • 25
  • 20
  • 17

It's true that Duolingo will only bring you so far before other resources should be used. Actually, I was using other resources before I ever heard of Duolingo, but they were inadequate, and there was no order to the learning. To be sure, Duolingo is inadequate if you want to learn to speak (or even read) another language fluently, but at least it is orderly, and gives you a goal to reach. But if it just gives easy stuff, it's little better than what I was doing on my own. How can you learn the different grammar structures, for instance, if you get only the simplest examples?

June 11, 2015

https://www.duolingo.com/nattic91
  • 16
  • 14
  • 9
  • 8
  • 4

According to learning theory, as counterintuitive as it might sound to you, things that are shown a lot are not always learned the best. There is more to learning than repeating something over and over again. These tests are conducted to be able to show that either "Susanna, you are right, little questions are bad for learning" or "Susanna, you are mistaken, we have shown that people learn significantly better by having few questions". We might think things are in a certain way, but it can only be proven by conducting statistical tests.

June 11, 2015

https://www.duolingo.com/Susanna35
  • 25
  • 20
  • 17

I never said that the "little questions" were bad for learning; it's simply that, if that's all you get, your learning is extremely limited. When I first finished my tree, and was going back to refresh the other lessons, I often got new, and more challenging questions. Now, most of what I get, I already know. I still make mistakes, to be sure, but those are mostly either typos, or not reading the topic carefully. And I did also mention that they don't seem to have any option to translate into the target language. You can't learn a language adequately just by translating it into your own language.

June 11, 2015

https://www.duolingo.com/Susanna35
  • 25
  • 20
  • 17

Maybe duolingo should set up two options - an "easy" one for beginners, and a more difficult one for those who prefer it. Had I the choice, I might have elected the "easy" one to start with, but I would certainly have gone to the more difficult one once I had completed the easy course. I just finished going through the gerunds course again; it seemed to have, in general, more challenging questions than I have been facing lately, but still had only the short lessons.

June 12, 2015

https://www.duolingo.com/Susanna35
  • 25
  • 20
  • 17

Wow, I just got clobbered! Refreshing Future Perfect - getting plenty of questions, lots of translating into the target language, making tons of mistakes - but learning a lot. Many of the mistakes are unrelated to the topic, but most are. After all my past complaining about the lack of need for future perfect - I have caught myself thinking occasionally using it. It almost seems as if someone at duolingo thought, OK, let's show her. Well, I'm not complaining, except for the fact that some of the basics needed aren't even in the tips. Well, I'll figure it out eventually.

June 12, 2015

https://www.duolingo.com/Nylre536
  • 13
  • 11
  • 10
  • 10
  • 7
  • 2
  • 21

I have the same thing. In my test the bar measuring your progress fills up more quickly if I already know the words in a new lesson. I am now learning Swedish and a lot of words are similar to Dutch. If I get the new word right it seems to skip a lot of the questions with that word in it and just gives me the points. If I hover over the word to see what it means it doesn't do that.

June 11, 2015

https://www.duolingo.com/Oh_Stylo

Is there any confirmation as to whether or not this is an A/B test, and how long the test is running for? I've virtually hit a standstill since this change was implemented. I'm not comfortable enough with my last completed skill to move on to the next, but practicing it with seven questions (the exact same seven each time, in the same order to boot) has done little to increase my confidence with them. Strengthen is too random to be of any particular use when there's only a few you're struggling with. What's the point in being able to complete skills faster if your retention of them is next to nothing? I love duolingo, but I'm 100%, completely, utterly, thoroughly unimpressed by this change. At least give us the option to toggle between seven and 17 questions?

June 16, 2015

https://www.duolingo.com/Susanna35
  • 25
  • 20
  • 17

I've seen no confirmation one way or the other. I do find that doing a "refresh" still gives 17 questions, and more translations to the 2nd language. And as others have suggested elsewhere, after you have finished your tree, you might do the reverse tree (if one exists for your course).

June 18, 2015
Learn a language in just 5 minutes a day. For free.