Suggestion for Lingots and veteran users
I was thinking about how Duolingo does A/B testing for implementing new features. As I understand it, the goal is to increase user retention but these results are only relevant for new users. For veteran users (like myself) there should be a pretty inelastic relationship between new features and user retention. In fact, sometimes I am a little disappointed when I hear about new features that I cannot see - e.g. I was not in the test group for the words / flash cards, and got a little jealous!
So, here was my idea:
Veteran users (maybe... 3+ months?) get NO A/B testing. They only receive features that are deployed to everyone. Instead, veterans will see a super secret list of any features being tested and for a small price (maybe... 20 Lingots?) can "buy in" to the test.
As I see it, this would help spice up DL for those of us that have stuck around, it would give new users something to stay around for, and it would also give me something to do with the Lingots I've been accumulating. Also you could collect some other fancy statistics about what people want (i.e., buy) vs. what actually keeps people around.
Just a thought :)
In addition to user retention, I think they look at how well a person retains new information presented to them, how long they spend learning per day, how many days they are present vs. absent, and other things. I mean, I created a Duo account ages ago but only REALLY came here and stayed in the last 6 or 7 months.
I do like the idea of letting people with a certain number of XP buy A/B skills for lingots... but then that might skew the data. But I love the idea!
Not sure why 'veterans' should not be in A/B testing. Retention applies to everyone, not just people that have been here less than three months (which I consider much less than 'veteran' status). A new feature may 'turn off' an old timer -- I think of the first dueling feature that was imposed on us without A/B testing on us and led to many, including old timers like myself, to say "Change this or I am leaving."
Wait, so, you didn't like dueling? That was actually a pretty good way of practicing....
I think so-called veterans should be on for at least a year before they get anything special. I mean, three months isn't that much. Maybe the user would have to be level 8 and with a 10 day streak (not at the three month mark, just ever) and THEN they would test out stuff.
I have a paltry 382. :) Though I had 444 at one point, and that was pretty cool.
You want to have an unbiased A/B testing group so determining for yourself if you want in or out does not work for statistics. Furthermore you want to evaluate Junior-Medior-Senior members with each other and the effect on both groups to improve the site.
In my post I siad: "there should be a pretty inelastic relationship between new features and user retention [for seasoned users]"'
What I mean here, is that a senior user leaves because of lost interest in language learning, finished a tree, etc.. They leave because they are finished with Duolingo and not because the site didn't interest them.
If you assume what I just said, it is very straight forward to (mathematically) prove that their contribution to the A/B statistics is in fact 0 and removing them by letting them choose the tests will bias nothing! In other words, they leave from the A group, and the B group at the same speed independent of the change itself -- plus it's very straight forward to un-bias a known bias in your sample set.
I only see the really major changes as possibly affecting seasoned users. Things like the incubator, or when the apps came out, but these are announced beforehand and go through no A/B testing anyway.
I've been on this site for over 2 years, and I have seen a lot of changes! It would take a pretty dramatic change (e.g., having to pay or something...) to drive me away, and I'm not the only one who thinks that.
I've read your post again and see indeed that you focus on user retention, I missed that point. I was talking about improve effectiveness of learning and that is related to anyone (Junior or Senior users). So we are talking about two different topics, in my case you don't want to exclude users based on their seniority. In your case, testing stuff because of retention could perhaps be excluded for senior users.