New version of the German tree April 2016 (A/B test)
Wieder ein neuer Baum!
Last year, we launched a major new version of the course, which came with greatly improved metrics, but of course, we weren't able to fix all the problems of the previous tree, and because of technical limitations, a few new problems emerged. The latest tree version, which we have just launched as an A/B test, is just a minor revision. We've fixed a few things here and there and added a couple of words as well as a skill on relative pronouns.
The usual caveats related to A/B tests apply:
If you get the new tree and previously had a golden owl, you'll have to redo those lessons that have been affected by the new tree version.
The A/B test is part of the process of allowing everyone to have the highest quality course. Don't fret if you don't have it yet. If the metrics show that the new version actually is better, you'll get it too in due time.
Is this why the Philosophy skill had to be done again? Did you accidentally make the change available to all learners?
If only the Philosophy skill changed for you, you haven't got the new tree and there isn't any new content. That's an unrelated glitch.
I think I have the tree. But I'd like to suggest more EN --> DE and less DE --> EN. I think a lot of other users feel the same way. But danke for this awesome course!
Am I alone in thinking that A/B testing, as used by Duo, may not be the not the best way to judge changes to the course?
It seems foolproof. Duo proposes a change, divides the population into two groups A and B, giving the new system to A and keeping the old format with B. After a period of time it compares the results. If group A are doing better than B it brings out the change for everyone, if not it keeps things as they were. What could possibly be wrong with this method?
In my opinion a problem with A/B testing could arise when the population being tested is not homogeneous, and when it is the only method of assessment being used. If we were all identical in ability, experience and motivation then relying on A/B testing alone would appropriate. Since we are not then I am unsure about too much reliance on it in education.
Let me give an example. A school divides the pupils into two random groups A and B in order to test the effectiveness of a new teaching method. At the end of the year the average results of A are higher than the average results of B. Does this mean that the school should now teach all pupils using method A? Not necessarily. What if on analysis the school found that although on average the results were better certain groups within the population did worse under method A? What if for example girls did worse under A, even though the average improvement in boys more than made up for this? What if the method was good for the bulk of pupils of around average ability, but was harmful for pupils who were either very able or of low ability? If the school wants to do the best for all pupils then it will not just introduce A for everyone, but will look for a method C that deals with the issues discovered.
My point is that because Duo does not know anything about us as learners, it does not know when using an A/B test whether the new method, while good for the bulk of the users, is harmful for significant minorities. I suggest that when doing an A/B test Duo should offer a short questionnaire to a sample of the people in groups A and B. This questionnaire should ask about: gender; age group; level of formal education reached; previous experience in language learning; and reason for learning a new language. When comparing the results of the A/B trial Duo will then able to see if new format is beneficial not just for the group as a whole, but for the various sub-groups within it.
That is not how A/B test works. With A/B testing, there are two groups.
- The control group: Group that gets the same tree as before
- The test group: Group that gets the new tree
For the test to be run, the following conditions should hold true
- The test group should be selected randomly. Selecting a group using any other method, such as a questionnaire introduces bias. The size of the test group would determine how long the test has to be run.
- The test needs to be run for a sufficiently long time to remove bias that might occur due to "seasonal" variations.
- For the test to be successful, result of the test group should be "statistically significant"
Source: engineer who implemented A/B testing at work
I am sorry that you have misunderstood my comment. I did not suggest that Duo select the A/B groups by using a questionnaire, but that they would be able to analyse the results better if they did one.
Let me explain why I made my comment. Duolingo is a fantastic program for me. It is just hard enough, contains enough explanation of language rules, and enough variety of question to have kept me using it every day for nearly two years. I have recommended it to lots of my friends and family and found that all but one of them very quickly gave up. Some found it too hard, and some too easy. Some thought it was too repetitive, others that there was not enough repetition.
Why do so many people give up Duo very quickly? How could it be changed to make it more useful for complete beginners? No amount of crude A/B testing will ever answer those and many other questions.
Sorry for misunderstanding your comment! I think I have a clearer idea now on what you meant.
I feel that the problem is "one tree fits all" doesn't really work. What Duo probably does is that it tries to find a tree that most people would be comfortable with. This is also the same problem with regular German classes. Most of them are very rigid and unless the class size is very small, the students of the class will feel the same things you mentioned.
Perhaps a tree changes itself based on user's skill, speed and pacing could work better, but sadly it might be much harder to implement, test and maintain.
Does duolingo inform a user he/she has been selected for testing? Does duolingo inform the user when testing has ended? I'm asking primarily because I know a lot of users are learning more than one language, but probably not simultaneously.
Say, for example, a user has two languages -- German and Spanish. The user gets selected for an A/B test in German, but that user (according to a self-designed plan of study) is now studying the other language -- Spanish. That user might go back to German if he/she knew about the A/B test selection, but I think most would want to know the duration of the testing in order to plan out future language study. Furthermore, I can see a user willing to go back to a language for a few weeks for an A/B test, but less willing to go back for a test that just goes on indefinitely. On top of that, some might think it is a waste of time to participate in A/B testing if they think that it is already underway. I'm kind of a big fan of the "unseen hand" when it comes to testing but, at the same time, the more test subjects you have, the more reliable the data in the end.
I see that you are studying more than one language yourself, Adam9812, so perhaps you understand what I'm saying.
No, no notifications for being included in a test, or when the test is done. At least not in my case. I only find out about them by reading others' posts here in the forum.
And you're right, I checked my German tree to see if I was in the test (I am not), but if I had been, I would of course have gone through the new lessons.
there is no new tree for me and all are gold and gold owl remains but the Words list is totally screwed up and does not even update when i complete lessons again,
is this going to be resolved or is Words list just no longer any good as far as strengh levels?
I don't remember ever having a gold owl, but my tree always remains golden. But the words list is not working and shows tons of words that need to be practiced even though I have practiced those words and have even tried not to advance in my tree just to make all the words golden. It's not working. I think this is a recent development, as I was always able to keep all words golden before.
OK, I've got the same glitch. Philosophy to redo (why?), but not other changes beside that.
The only skill that has changed was philosophy. I saw that you said this is not the new tree. When it will be available?
I'll ask just out of plane curiosity - as you work so hard on new and new versions of the German tree for English speakers, do you plan to update the English tree for Germans as well? Because, let's be honest, it could use some adjustments.
Oh, that's a good one indeed :-)
And I meant things like totally missing Tips and Notes and the awkward absence of present continuous tense which is only partly introduced at the very end of the tree. While it should have been one of the very first skills - to teach the difference between "I walk" and "I am walking". And so on...
I wasn't expecting that. Wow. That was pretty funny, so thank you for sharing it with us. Ya know something, though? It kind of makes me like duolingo just that much more. Some elements of it, like the one you pointed out, are a work in progress. If you think about it, many things in life are works in progress.
Never stop learning and never stop improving.
Hello! I'm new in this website. I was told there was an option called "immersion" where I can translate documents on it. Why can't I do it?
Immersion is not available for some courses and is not available for many newer users due to an A/B test.
When I finish a lesson I now see a different screen. It has a little circle for each day of the week, and a big circle that turns orange half way around when I complete one lesson and fully orange when I complete two lessons. After that I am left to guess how many lessons I have completed that day. I really hate this, as I know how many lessons a day it takes to keep my tree gold, and it is not two. Is this supposed to be an improvement?
It isn't meant to predict how much you need to keep your tree gold but how much you need to reach your daily goal. At least that's how it works for me. Do you happen to have a goal of 20 XP?
But I'm pretty sure this isn't the update this post has announced. The language teams can improve the words, sentences and skills, but the rest is out of their hands.
I know that my daily goal is 20XP. However, the previous screen that appeared the end of each lesson (on my iPad) told me how many XP I had thus far completed that day, and I found that information useful. Now I have to login to my account in a web browser to see that number. Drag. This new screen first appeared last week.
i've had that on my mobile device the whole time. only on browser does it show the days and the weekly with numbers. try logging in on a browser on your ipod and using the regular site and it might show?
I can't tell if I've received the new tree or not. My tree looks the same, but I've been getting lots of new practice sentences despite not having learned any new skills for five days.