1. Forum
  2. >
  3. Topic: Duolingo
  4. >
  5. Artificial Intelligence vs. V…

https://www.duolingo.com/profile/Larisa_L

Artificial Intelligence vs. Volunteers

I remember back in time when we only started as volunteers in the incubator (that was 5 years ago), Duolingo team used to acknowledge that we volunteers were those who allowed free learning for everyone. Duolingo was positioned as a massive online collaboration project, similar to Wikipedia. Unfortunately, in all Luis's recent interviews there is no mention of all the work volunteers do. Here is one of the latest interviews: The Future of Education: Online, Free, and With AI Teachers?

Quote:
"Duolingo is able to adjust how it teaches -- for each individual student -- based on the feedback it receives. If students who learn about plural nouns before adjectives are progressing faster, it can adjust the lesson plans. The goal is to maximize the overall proficiency of all students.

The AI doesn't require a salary or health insurance. But it is able to personalize lesson plans, account for the fact that students learn at different rates, and teach at a consistent level at a global scale.

This could be a quantum leap for education. The United States is fortunate enough to have a rather well-developed education system, which is structured and can attract talented educators. But in many developing countries, that's often not the case. AI could level the playing field for many who might not otherwise have access to formal schooling or to good teachers."

So according to Luis, it's all about algorithms. I would argue that that's not true at all. The way the trees are built doesn't allow changing the order of skills. All the AI can do is to select the assignments within 1! skill. It cannot teach you "about plural nouns before adjectives" if adjectives are given earlier on the tree than plural nouns. It's impossible. So actually, the success of teaching depends on the carefully developed tree structure, on the choice of words and sentences for each skill and on the adequate set of translations and exercises for each sentence. ALL of that is done by volunteers. On top of that we, volunteers, write grammar notes, help users in the discussions, read reports and fix all the issues with the sentences. None of that is done by AI. If we never did any of that, Duolingo would be a very limited system with only few underdeveloped courses available. If all of us stop doing anything today, Duloingo might even survive thanks to all those work hours we've already invested. Maybe that's the reason they don't care to acknowledge our major role in the project any longer. They think they can survive without us.

Sorry Luis, whatever you believe about amazing jog you do with AI it is worth nothing without real people creating and supporting the courses, volunteers and those on salary alike.

October 18, 2018

102 Comments


https://www.duolingo.com/profile/pQp7zNt2

I looked at your bio and you have been with this organization for a long time. I think the volunteers are often overlooked because we think free is free.

I can only imagine how many hours you have dedicated to this platform. And I say thank you.

October 18, 2018

https://www.duolingo.com/profile/Larisa_L

Thank you :) Now when I'm rereading my message I realise that I didn't clearly state why I even bothered to write it. My point was that if we want Duolingo to thrive and develop then we need to promote those things which actually make it great, and that's not AI, that's the community of volunteers. My personal contribution would also mean very little if there weren't hundreds of others in the incubator. This unique community of people from all around the world is the true creative power behind Duolingo.

October 18, 2018

https://www.duolingo.com/profile/Rick392366

Thank you for posting this message.
I volunteered as a contributor less than two months ago to help in a new course.
I think you're right.
When I read Luis' interviews it feels like a stab in the back.
But then again, maybe that's just the whole point of these interviews, to point out that at the end the volunteers are not needed anymore and that they will be replaced by AI.
That makes Duolingo so much more interesting for investors.
It's all about money.

October 19, 2018

https://www.duolingo.com/profile/daKanga
Mod
  • 298

I am not convinced that

It's all about money.

I have seen Duolingo do even this year some things that I thought quite interesting as a public stance for an American company to take. Especially in regard to Something Like Home — a short documentary about refugees using Duolingo

As well the initiative to also http://making.duolingo.com/how-duolingo-achieved-a-5050-gender-ratio-for-new-software-engineer-hires

Which for also this 50/50 initiative, I have seen some startling blow back from. Along the lines of how unfair this initiative is of course.

From my interactions with staff over the years, I have also observed genuine passion for the stated object that Duolingo has remained consist to, to date. That of :

We created Duolingo so that everyone could have a chance. Free language education – no hidden fees, no premium content, just free.

that you can see if you check out https://www.duolingo.com/info , and scroll down to the orange page.

In that, the issues that are being aired here, I consider are VERY IMPORTANT.

And as always remain impressed how Duolingo continues to support difficult discussions such as this, to remain open, and able to be discussed.

You can also read about its policy on this here:
https://support.duolingo.com/hc/en-us/articles/205079384-What-are-the-Moderator-Guidelines-

Including : Moderators will NOT: ... Delete threads just because they are critical of Duolingo

And this is a continuing example of this. Though of course - that it is also Duolingo who is also supporting this discussion to remain up and able to be discussed.

October 20, 2018

https://www.duolingo.com/profile/daKanga
Mod
  • 298

I agree with you Larisa_L, in that the success of Duolingo is also significantly based on the :

unique community of people from all around the world is the true creative power behind Duolingo.

Though , you will also note I have inserted the word also.

For it is also the platform of Duolingo that has enabled this unique community from all over the world to collaborate on creating courses. And this is not just one aspect of the platform. It is a combination of features as well as how the community is supported that has allowed this to occur, in a way that no other platform on the internet has allowed.

sub note: Be it that many of these resources are so clunky, it is because of the good will and intelligence of the community, that the things behind the scene are able to function. And - yes - that is another story as well.

I am trying to hold myself back from listing what all the numerous features that enable this collaboration and work to occur. However it is a suite of them.
Of which also this forum here is a critical component of.

And also part of this is also this AI component that is also being presented as a counter point to the esteemed value of the huge body of work done by volunteers, and also by staff creating course content.

I am immensely thankful that you have raised this discussion and important issue Larisa_L.

( and yes - I am also likely to write more on this very important topic, which also contains huge issues of concern and relevance for myself :) )

October 20, 2018

https://www.duolingo.com/profile/Luis

It was not my intention to downplay the work of all the amazing contributors, but I can see why it came across that way. I will try to do better next time.

October 18, 2018

https://www.duolingo.com/profile/coto.i
  • 1718

Sorry to say this, but I feel that you are more and more disconnected from the human element here, which is the community of contributors and users.

October 18, 2018

https://www.duolingo.com/profile/Andresb_UY

Sorry, this is an automatic response. Please try again in a few years.

October 18, 2018

https://www.duolingo.com/profile/xMira_

Yes-- that, I agree. With crowns, with his whole statement about not listening to forum users, now this... it just seems like they/he are/is only listening to the numbers rather than the people, you know?

October 18, 2018

https://www.duolingo.com/profile/Luis

:( I really do read the forum every day, as do others here at Duolingo.

October 18, 2018

https://www.duolingo.com/profile/E.T.Gregor

I really don't want to sound like I'm questioning your truthfulness, but if this is the case, then why do so many questions that get repeatedly asked on the forums stay unanswered? You could make the life of many people easier if you were to occasionally answer the most common questions. If you read the forums every day, I'm sure you know what they are...

October 19, 2018

https://www.duolingo.com/profile/JimOser

@Luis It is great that you are at least reading the Duolingo in English forum every day. Are you "eating your own dog food?", are you reading the forum the same way that your users are reading it by going to forum.duolingo.com and using a browser, or are you reading with special developer tools that put everything in chronological order? Or maybe you have a special tool, to only see the posts, that have been made since the last time that you looked. Or possibly a poor undergrad is organizing the posts for you.

We can't even find all posts from you. There isn't anyway to find all posts from Luis. Also your posts sometimes show up in the popular section, sometimes in the new section, and sometimes in the followed section.

How could your recent post (currently 2 months old) , "What changes would you like to see?" with 319 likes, 320 lingots, and 992 comments, not be listed in the popular section? https://forum.duolingo.com/comment/28202855

So many of the new users posts are the same? Can't you have a visible FAQ, that really addresses the frequently asked questions?

Why can't posts have exact dates? What is with "2 months old"? Why not have the exact date? 4 months from now the post will by "6 months old" but the exact date hasn't changed.

Anyway, I am really glad that you are reading posts. I certainly hope, it is really you.

P.S. Are you reading any other forums, besides the Duolingo in English forum?

October 19, 2018

https://www.duolingo.com/profile/Luis

I use the forum the same way everybody else does: on a regular browser. If we had time to develop special tools for it, we would release them to you.

I know the forum could be improved, and we are working on it, but we have to balance this with the many other priorities we have — improving how we teach, contributor tools, Stories, etc.

October 19, 2018

https://www.duolingo.com/profile/xMira_

Oh, wow-- I really didn't expect that-- I meant more to the thing you said to this interview-- hmm.. let me find it. Ah, this one: https://www.entrepreneur.com/article/290664 which I think is painted harsher than it actually is, but it sounds really bad as a forum user.

I really didn't expect you to be reading that, honestly, I didn't mean to offend you at all.

October 18, 2018

https://www.duolingo.com/profile/Larisa_L

xMira, I think Luis was absolutely right in his statement in that interview. Don't take it out of context, he didn't just say not to listen to the users. He said that some tools like forums are sometimes not a good tool for listening. You get responses mostly from angry people, and you don't get feedback from those who are satisfied. The problem with Duolingo, especially for volunteers (less for Luis as his team has some other tools) is to how understand the whole picture and not to waste too much time on those users who are never happy. It is a challenge as sometimes all this negative feedback is very demotivating (especially for volunteers). So I agree with Luis here, I've learned not to be over-concerned with users' complains on the forum and take them with a grain of salt.

October 19, 2018

https://www.duolingo.com/profile/xMira_

Sure-- I agree with that. Again, I think the title of the interview is harsher than it actually is, but that doesn't change the fact that it's kind of hurtful rhetoric? IDK, that interview has always been sited when someone brings up feedback (even good feedback) like "Oh look at this Luis is never gonna listen"

October 20, 2018

https://www.duolingo.com/profile/sapere__aude

Thanks for sharing.

I too teach languages and don't have enough time to do everything I want—to make everyone happy. Spending time to reflect amid all this is difficult and it's something I respect.

Please remember that no amount of learning science, A/B testing, funky new contributor tools, or tree-system tinkering, will matter if the basic materials are incomplete and under-loved.

Keep the courses alive.

Hire somebody (even a non-Vietnamese native!) to correct English sentences for the Vietnamese course.

Thanks!

October 27, 2018

https://www.duolingo.com/profile/Luis

I should also emphasize that I wasn't the one who said the line about AI not needing a salary or health insurance. That was the reporter, who's specifically interested in AI.

October 18, 2018

https://www.duolingo.com/profile/Larisa_L

Luis, thank you for your response. I did figure out that the reporter probably added a lot from his /her own imagination. But I've been reading and listening to many of your team's recent interviews. There are lots of overstatements which are not true. Like with AI being able to change the order of grammar to be taught (I've seen this statement multiple times). Again, I get it why you make such statements, you want to impress people, get them interested - fine with me. What I don't like is that the message is always: we're such a high tech company, we have such advanced algorithms which allow revolutionary new approach to teaching and all for free. While in reality the true revolutionary thing about Duolingo is that so many people are willing to spend hours and hours of their time for free to develop new courses, to listen to the users and then use their feedback to develop even better courses. If those people had better course development technologies, if they had access to some of the data your algorithms and tests produce, then they would be able to create something amazing. But you don't share any of that information with us, do you? (our course statistics are very basic). You don't invest much of developers' time in creating good tools in Incubator (we rely on using Java scripts to make our lives easier). Again, no matter what you do, volunteers are moved by the true love to languages and enormous desire to help others, so we will keep doing our work even under worst circumstances.

You're an amazing entrepreneur Luis, you've created so many awesome things. I encourage everyone here to actually give you a break, because you do really care about Duolingo and volunteers as well. I think you just don't realise where the true power is hidden. Just think about Matlab and Python. In many universities nowadays they started replacing Matlab with Python. Why? Matlab has awesome team of professional developers. And Python is free with a community of volunteers constantly developing new things. The latter wins. And there are many other similar examples with various software. Embrace the community you created, give them tools you have and Duolingo will get hundred times better.

October 19, 2018

https://www.duolingo.com/profile/ShellMarg

Luis, are you testing NOT using Moderators/Contributors? The new staff developed EN>FR Tree4 does not allow access for any of the EN>FR Moderators/Contributors. This appears to be a huge change in the Duolingo concept of teaching, eliminating the Moderator/Contributor role.

October 19, 2018

https://www.duolingo.com/profile/hughcparker

Duolingo does have something approaching AI - its spaced repetition algorithms, for optimising the timing of revision.

The crazy thing is, though, that's the part of the website they've turned off. They're still recording the data, but you have to install an extension or use duome.eu to find out which skill it's best to revise next.

It's really frustrating. Sometimes Duolingo seems to be run on the basis of efficiently and carefully implementing well-thought-through ideas, and sometimes it seems to be run on the basis of drunken bets.

October 18, 2018

https://www.duolingo.com/profile/hanspersson

Are there extensions to get the kind of spaced repetition that the old system used to have? Where do I find those?

October 18, 2018

https://www.duolingo.com/profile/Thomas.Heiss

@hanspersson

Quote: Are there extensions to get the kind of spaced repetition that the old system used to have?

I started a new thread as it gets a bit off-topic here.

"[HOWTO]: Different ways for spaced repetition with skill strength viewer (user script and extension)": https://forum.duolingo.com/comment/29304553

October 18, 2018

https://www.duolingo.com/profile/ronkisimo

or the ai chatbots that they took away.

October 18, 2018

https://www.duolingo.com/profile/Gunnar388832

Duolingo (or any other platform) can only optimise that based on how we use their platform, not based on our entire language learning experience (be it other platforms, reading, talking to people, listening/watching media or something else). I find that rather discouraging as it gets you stack in your platform, rather than encourage you to actively use your language. The current crown system is much better as you can test out skills you know well from elsewhere and focus on your weak points.

October 18, 2018

https://www.duolingo.com/profile/daKanga
Mod
  • 298

Have you checked out things being tried out in :

https://www.duolingo.com/labs

Especially https://events.duolingo.com/

And https://stories.duolingo.com/?referrer=web_tab is also worth checking out.

I have not yet checked out https://podcast.duolingo.com/

Anyway - the above two I would suggest could be worth while for a more immersed language learning experience.

October 20, 2018

https://www.duolingo.com/profile/oskalingo

From the interview:

For example, we teach Irish. It turns out there are 94,000 native Irish speakers and we have about a million people learning Irish on Duolingo. So we can actually multiply it by ten, the number of people who are speaking that language when we put it in the platform.

No. Maybe a million people signed up to the Irish course at some stage in the last year and did some amount of lessons (most sign-ups still do very few). Whether the very small fraction who complete that course (which is notorious for being one of the poorer courses on here) have really learnt much is another matter. Whether you learn much of any language on duolingo is also quite debatable (despite Luis' unsubstantiated claims). And as for saying "we multiply by 10 the number of people speaking Irish", that's a highly questionable claim when the base he's comparing to is native speakers. Very few people end up speaking Irish at anything close to native level after only doing the duolingo course. In fact, I would guess basically zero.

Yes, with the right VC investors you can get a tame interviewer from The Motley Fool to ask you tame questions in a pretend interview. Where are the interviews where Luis gets asked real probing questions from an independent and professionally sceptical journalist?

* Edited this comment to tone down the language and to bring in the correction that scarcerer made in their comment below (I was using duolingo's old definition of learner).

October 18, 2018

https://www.duolingo.com/profile/scarcerer

Well, Irish dropped from, what, 3-4 million “learners” to 998k “active learners”. So it’s not one million since the course was released four years ago but one million within the last year. Still, that doesn’t tell us anything about the amount of people who finish course, let alone become proficient once they move on from Duolingo.

October 18, 2018

https://www.duolingo.com/profile/oskalingo

So it’s not one million since the course was released four years ago but one million within the last year.

Ok thanks, I appreciate that correction.

October 18, 2018

https://www.duolingo.com/profile/svrsheque

also, accounts != people.

October 18, 2018

https://www.duolingo.com/profile/MissSpells

Comparing numbers of native Irish speakers to people learning (to whatever degree) Irish on Duolingo, and failing to mention that Irish is a mandatory school subject in Ireland is really skewed. I guess the people learning/having studied Irish in school in Ireland (which, if I understand correctly, is most of the population) don't count statistically.

October 18, 2018

https://www.duolingo.com/profile/Songve

"an independent and professionally sceptical journalist?" In the US we have a political party and followers that refers to that simply as "fake news".

October 18, 2018

https://www.duolingo.com/profile/Multi0Lingual4

In the US, the news isn't what it used to be - on both sides.

October 18, 2018

https://www.duolingo.com/profile/Robert-4-Christ

Sadly that is very true :(

October 18, 2018

https://www.duolingo.com/profile/Seattle_Scott

Clearly this interview is targeting investors and future funding, but aside from the lack of mention of volunteers, the line “AI doesn’t require a salary or health insurance” is insulting. That’s what investors want to hear. They can make money without paying humans.

What a strange dichotomy- it’s good for humans because it provides free education, and it’s good for investors because it doesn’t require humans. What have we become?

October 18, 2018

https://www.duolingo.com/profile/lizsue

"What a strange dichotomy- it’s good for humans because it provides free education, and it’s good for investors because it doesn’t require humans."

Maybe the idea is that once the free marketers have profited by not requiring humans, laying off their humans, and therefore all laying off each other's customers then the workers who got laid off and can't afford to be customers anymore can still get educated for free?

October 20, 2018

[deactivated user]

    I genuinely appreciate you writing up this post. That interview is...well, it's extremely problematic. There's a lot I'd like to say here about the tech world and their cynical grift aimed at siphoning off our public education funding and their deep contempt towards human educators (especially those of us who are dedicated to educating less affluent students and who believe even the poorest kid deserves a proper classroom with a salaried, professional human teacher), about how Luis has always lived in this ridiculous bubble of his own snake-oil sales pitches, about how out of touch Duolingo Inc. has always been with the Duolingo user community, what we have wanted from this site and our support of volunteer contributors. But there's only so many hour in the day...

    However, I will say this: there is a reason why in many of my own posts where I have been critical of Duolingo that I point out that the people creating and maintaining these course are unpaid volunteers and that Duolingo has literally built their brand on these volunteers' work. I have been a Duolingo member since 2012 too and I really don't think I would be still using this site if it wasn't for the content created by volunteers. I suspect many other users can say the same. Volunteers didn't just create the content that Duolingo Inc. claims proprietorship over--they made Duolingo something worth investing our time in. Regardless of what Duolingo Inc. says or doesn't say, and regardless of what direction Duolingo Inc. takes in the future, I think I can safely speak for many users past and present that the work of volunteer contributors has been greatly appreciated. Thank you to all of you.

    October 18, 2018

    https://www.duolingo.com/profile/Larisa_L

    A really good point you brought up about human teachers. I'm a big proponent of online education (attended Harvard and Oxford mostly online). I think the future is in just a few star teachers and professors creating top notch courses for people to use all around the world. But! that doesn't mean there should be less teachers in the classrooms. They should have a different purpose though, instead of creating the course materials, they should be spending all their time with students giving them true human care, paying attention to their individual needs, motivating and encouraging, inspiring and supporting. No AI will ever do that as good as humans can do. And such jobs can be truly satisfying for those who have them.

    October 19, 2018

    https://www.duolingo.com/profile/lizsue

    Yes, exactly! :)

    October 19, 2018

    https://www.duolingo.com/profile/Davey944676

    The term "A.I." seems to be over-used these days to imply genuine Artificial Intelligence. The DuoLingo system might use some clever and useful programming, but none of it sounds like it needs actual Artificial Intelligence (as I personally would think of it).

    It's still the humans doing the clever, nuanced, insightful, helpful stuff. Thank you, all you human contributors.

    October 18, 2018

    https://www.duolingo.com/profile/Kuraikari

    As a correction I will write about Neural Networks in particular:

    NNs are able to learn on itself, so there would be a lot of resources used for the vast information here.

    Human language are for machines still a bit over the top. That won't change for another few years.

    Even if they create a working NN, the efficiency would be at n^n and use too much resources for the current standard at Duolingo.

    Even after they've done it, they still would need human resources to maintain the product as convenient they are they make errors.

    October 19, 2018

    https://www.duolingo.com/profile/Shady_arc

    Actually, clever algorithm and even a fairly dumb algorithm IS an AI. What is usually understood by AI is a system that lets computers solve problems involving different variables and conditions, regardless of how complicated the underlying algorithm is.

    October 19, 2018

    https://www.duolingo.com/profile/Kuraikari

    Okay sorry, I will correct it! Forgot that they didn't mean Neural Networks :) However there are different categories of AI and the only applicable one for an ever changing thing like the natural language would be a selflearning one.

    October 19, 2018

    https://www.duolingo.com/profile/garpike

    This is all very true for the moment, but takes no account of the speed at which AI is improving; the article is called 'The Future of Education', after all. Undoubtedly the number of things AI is able to do well will continue to expand, and DL's reliance on volunteers will decrease, as it's much easier to deal with an AI compared with a lot of random people from different time zones who each have a relatively small amount of time to spare.
    Of course, this is a double-edged sword for DL, for once AI is as good at teaching languages as humans, it will also be as good at translating them, and fewer people will have the incentive to slog through language-learning courses for reasons of practical communication.

    Unfortunately, in all Luis's recent interviews there is no mention of all the work volunteers do.

    If the rumours of a DL IPO next year are true, he probably also doesn't want to dwell on the fact that his company is currently reliant on unpaid volunteers for the majority of its content. Talking about the potential of AI is much more likely to drive up the future share price. Bear in mind that he gave this interview to the Motley Fool, a website targeted at investors.

    October 18, 2018

    https://www.duolingo.com/profile/Larisa_L

    You're absolutely right :) And I would probably do the same in his place.

    However, I think if Luis wants to reduce his reliance on volunteers he should actually start investing into them? 0_o how so? If there were really good tools in the incubator, many things could be done by fewer people. I personally found that I spend more time on teaching new volunteers than they then later spend on developing the course. Very few volunteers actually become highly productive. Keep only those productive ones (one-two per course) and give them good tools, with lots of ways to automate the process: word selection from frequency databases, sentence search on a given set of words or grammar, translation generator and so on. Or even simple tools to prioritise and clean out reports. Simplify the system, there is too much freedom of translation on Duolingo, which creates enormous work for volunteers and lots of arguing on the forums. This freedom should be limited a bit and replaced with more types of automated assignments.

    Managing volunteers is not an easy task (I've had about 30 people on my team in 5 years), so not an easy one for Duo team to handle with all those people coming and leaving the incubator constantly. And they do try to come up with new ways of communicating with us and helping us. But so far, I think they haven't found a really efficient way. If they figure this out, then only sky will be the limit for them.

    October 19, 2018

    https://www.duolingo.com/profile/pQp7zNt2

    “Managing volunteers is not an easy task (I've had about 30 people on my team in 5 years)”

    This sounds like this volunteer scenario has transitioned into a high ranking job (for free).

    I wonder when it became comfortable for you to do this. Your time in this life is so precious even when your helping others.

    There has to be some type of limits before you become resentful.

    October 20, 2018

    https://www.duolingo.com/profile/Larisa_L

    Not sure I understand your question, so forgive me if what I'm saying below is irrelevant :)

    I think I'm always going through waves of frustration and then enthusiasm. From time to time I would become extremely frustrated and irritated by all the challenges we have in the incubator, and not being able to achieve some goals even after investing so much time (you would not really believe how much, given that I have a full time job and a little baby). But then I have a wave of enthusiasm and energy again, especially when I see some success stories from users or have a great chat with one of hundreds of our volunteers, who are all very interesting people. And I've learned a lot from this experience, in my daily job I also have to work with remote teams, and my experience on Duliongo is very helpful with that.

    October 21, 2018

    https://www.duolingo.com/profile/nueby

    How do you imagine it will all play out if the informational liberties Luis has been taking (like using the present tense to describe the desired future situation that may or may not ever be reached by this particular company) continue as we near that IPO?

    I understand why he is doing it, but imagine what is likely to happen if people like Larisa are kept out of the "public's" eye until one day the Incubator gives them all an error for a whole new reason, true Duolingo style.

    For a controversy-averse outfit, Duolingo could be charting a challenging course for itself.

    October 18, 2018

    https://www.duolingo.com/profile/Usagiboy7

    It would be very helpful if there were two positions for devs: one for Contributors and one for moderators. The current system is stressful on staff and volunteers both. I regularly get a sense that there is a lack of appreciation because we are swamped by so many bugs and we have to make many of our own tools because our bids for tech time were not approved over another team's. (That other team or project is clearly also important. I don't want to take away from them. I would like to have more resources to be shared around in the background.)

    We are aware that when we bid for dev time, it will be put up against the question: "can the volunteer arm of Team Duolingo manage alongside x bug or without x tool? Yes, yes they can." After so many assessments of "yes they can", we are actually working around many different bugs and resource shortages. It is time and energy consuming to do simple things. Some of the best volunteers in Duolingo's history finally left because it was stressful. And if we were short even just jrikhal we would be in dire straights as he bends himself hours upon hours weekly to find ways to pad us against as many of the negative consequences of not having a dedicated dev for our team. The volunteers stay under this system of such steep resource competition and repeated deprioritization because the Duolingo dream is changing the world. It wouldn't be what it is without the volunteers who have been here up to this point. There is no paycheck or benefits to help pad those frustrations. There is a dream and each other who we've bonded with. How many times will the company draw from that well though? We could use more support.

    Not that long ago a community member made a post called "the history of Duolingo". They used interview articles and wikipedia and from start to finish they didn't mention volunteers. (They went back and edited it after realizing volunteers were a key factor. But, it had been easy for them to miss the relevance of volunteers among the many interviews.)

    Budget is an issue. Not enough Plus memberships probably for the two dedicated dev positions I've asked for. But, I hope you will consider making it a priority once the budget allows.

    Thank you for showing up to this discussion, where you knew you'd be getting a lot of criticism. I appreciate that, considering all of the pressure and stress you and the other staff members regularly encounter, including many negative comments in the forums that you read while you jot down feedback to bring into meetings. This discussion was clearly going to have plenty of that. But, you stepped in to be here. Thank you.

    Thank you also Larisa for bringing this concern forward. You've been a long time, very dedicated volunteer. :)

    October 19, 2018

    https://www.duolingo.com/profile/Larisa_L

    Not that long ago a community member made a post called "the history of Duolingo". They used interview articles and wikipedia and from start to finish they didn't mention volunteers. (They went back and edited it after realizing volunteers were a key factor. But, it had been easy for them to miss the relevance of volunteers among the many interviews.)

    I think that's exactly what I was trying to point out. If you only read about Duolingo outside Duolingo forums you might not even realise that there are any volunteers.

    Btw, I had a case when one of the community members tried to prove that I was paid (and quite well) and was just hiding it for some reason :)))))

    October 21, 2018

    https://www.duolingo.com/profile/benmaja

    I've learnt a lot in this discussion that I didn't know before about the volunteers, and it seems to me that many of you volunteers have been so valuable, so hard-working, so truly 'professional' that you ought either to be paid or given the tools you have identified as necessary to do the best job you can.

    October 22, 2018

    https://www.duolingo.com/profile/Sitesurf

    Thank you for your nice words. However, it seems that 'real' professional linguists and translators have been hired or sub-contracted from outside the volunteer cohort to write the latest in-house courses. I don't know if they will be more valued than we have been. The difference is that the only substantial compensation we have ever had was (and still is) learners' gratitude expressed on the forums, mostly sentence discussion forum threads (those Duolingo's staff does not read).

    October 23, 2018

    https://www.duolingo.com/profile/ShellMarg

    Sitesurf, thank you from the bottom of my heart. I have read and benefited from your clear, concise and prolific grammar advice on the EN>FR tree for years. It has made my journey an exceptional experience.

    October 23, 2018

    https://www.duolingo.com/profile/Larisa_L

    Same here. Sitesurf was the person who inspired me to help people on the forums. Though I'm not at all as good at explaining things and not as patient :)

    October 23, 2018

    https://www.duolingo.com/profile/Sophiafilm

    I want to thank you and all those amazing volunteers who work on Duo's courses. Really it's amazing. Personally I'm in awe of the Korean course and especially of the Tips and Notes.
    Previously I tried to learn Korean but it was difficult without practice and the courses usually contented a lot of information, which I think when you've just started learning disturbs more than helps. While here you're introduced to the course gradually, in small steps. Tips and Notes are very easy to understand, not to mention all that work, you are doing to develop the "trees", which is hidden from us.
    I also appreciate Duo itself. Because of it's algorithms I finally memorised the English Present Tense (I don't why, but I had been constantly forgetting it). Wonderful forum and pleasing to the eye design.
    So I think that remarkable achievement which Duo reached now is a result of cooperation of the both sides.
    Hope that the hard work of these amazing volunteers will be rewarded.

    October 18, 2018

    https://www.duolingo.com/profile/DragonPolyglot

    My dad and I have conversations like this often. My dad works in IT and he understands software better than an average person because his past jobs were focused on security. He keeps telling me that computers will eventually be more accurate human-language translators than human translators. I will then ask him how the computers are going to get information to translate and who/what will update any databases or corpus used to translate between languages. In short, computers do a great job in translation, but only if they have the information on syntax, vocabulary and rules put in place by humans, because humans create human languages. Computer languages don’t work in the same way at all, because a computer language tells the computer how to run data to get a result. Human language is all about communication of situations, ideas, actions, thought and description. You can use one in favor of the other (which is exactly how Duolingo works) but at the end of the day, the goal between computers (especially algorithms) and human speakers are different.

    And just to be clear, I’m not saying that no one should use computer algorithms to further language education and translation, as well as documentation. Corpus linguistics is a field that has greatly benefited from computer technology and uses algorithms to show at what frequency a word is used and in what context of meaning. Algorithms are useful but they need to be “taught” (fed data) and certain algorithms only do so much.

    October 18, 2018

    https://www.duolingo.com/profile/compsognathus

    Yay for the real people! Robots don't admiration. I have the utmost respect for all of you for the work that you do, keep it up!

    October 18, 2018

    https://www.duolingo.com/profile/Songve

    Has Luis passed the Turing Test?

    October 18, 2018

    https://www.duolingo.com/profile/wyqtor
    • 1796

    My opinion is that Duolingo should offer some small % of the money it earns through ads and/or PLUS subscriptions as monthly payment to the volunteers contributing to the best courses (their fancy metrics and statistics should be able to easily determine which courses are the best at teaching, have the least number of reported missing translations, etc.).

    It will help make Duolingo a better platform. Free is good, but let's face it - most of the time, paid services are even better (Linux might be the only true exception)

    October 18, 2018

    https://www.duolingo.com/profile/garpike

    If you start paying volunteers, they cease to be volunteers, and all sorts of bureaucratic rules start to apply, dependent on jurisdiction.

    Which courses are 'better' at teaching according to the statistics also doesn't really give an accurate picture of the value of the volunteers, as some languages are intrinsically easier to teach than others. The Esperantists would be raking it in whilst the poor old Navajo contributors wouldn't get a brass farthing...

    October 18, 2018

    https://www.duolingo.com/profile/Larisa_L

    Garpike is right, paying volunteers might not be the best idea. Creating tools which would allow to assess effectiveness of teaching - that would be awesome, we don't have anything of this sort.

    October 19, 2018

    https://www.duolingo.com/profile/Claeb101

    AI is advancing the world one step at a time but who knows when it will take over. We are the creators of AI and it is nothing without us. It is like a machine but we are the operators. AI's need humans to run and update them, but as technology advances, they are becoming more efficient, requiring less work to be put in while producing a much greater output. But, as this is growing humans are growing too and teachers are finding new ways to use AI to teach their students. AI is a tool that teachers should have the ability to use, but at the end of the day, it should be their decision.

    October 19, 2018

    https://www.duolingo.com/profile/Toscaaa

    Being an AI developer this is a pretty interesting discussion for me. There is an incredible amount of opportunities for implementing AI on this platform and I'm sure some stuff has been done already, and it is pretty hyped at the moment. Simply seen from a PR stand point, what is the message you want to portray? Being up to speed with newest technologies or being a great community builder? In the end both of them have their own assets but I can imagine one can be driven to talk mainly about one oversimplified focus point if that's all a certain reporter wants to hear.

    October 20, 2018

    https://www.duolingo.com/profile/AndroidKanada

    While AI can find patterns in errors, and perhaps prescribe exercises, I'm not aware of any work in the last 30 years that would let it produce structured lessons in a complex real-world subject like language. I get the impression the reporter was guilty of producing tech-porn. ))) (i.e. trying to breathlessly pump up excitement for something that is not actually attainable.)

    I give Luis credit for allowing and responding to such discussions. However, I suggest that some tools for contributors would help a lot, such as algorithms (not AI heuristics) for incorporating new acceptable answers into the marking rules.

    I'd also like to add my appreciation for the volunteers, for setting up the lessons, fixing the bugs, and patiently answering questions. It's easy to rail against a faceless "them" who appear to be uncaring, another thing to realize volunteers are trying to do these things in their spare time, for free, out of a love for their language and a desire to help others.

    THANKS! XOXOX )))

    October 23, 2018

    https://www.duolingo.com/profile/lizsue

    I thought the volunteers build the courses in the first place, then the AI decides which question to ask next once someone takes the already-built course.

    Think about it: do you and other volunteers keep track of users' answers whenever they get questions from the course? Is there a volunteer on the other end of the connection monitoring my progress whenever I'm using Duolingo?

    October 18, 2018

    https://www.duolingo.com/profile/Larisa_L

    and? What is more important in your opinion, the course itself of grading? They are both important, but without the course materials no grading would be possible as there would be nothing to grade. And who do you think reads your reports? AI or volunteers?

    October 19, 2018

    https://www.duolingo.com/profile/lizsue

    No doubt that volunteers read the reports, and volunteers provide the course material the AI selects sentences from! :)

    Also, a few minutes ago, I practiced one of the courses. A few minutes ago, was a volunteer or staff member seeing which answers I got right and which ones I got wrong in order to decide which sentence Duolingo gave me next (not checking later to read any report I might have made, but reading my answers during my practice session)?

    I got the impression from the interview that Luis was talking about AI tracking lessons and practice sessions at the exact time a user takes them. So it's not AI vs. volunteers, it's volunteers and AI doing different things with the same course.

    October 19, 2018

    https://www.duolingo.com/profile/Larisa_L

    If you read my post carefully, I don't propose to get rid of AI :). I'm saying that success of Duolingo is defined by work of people not AI, at least for now, in the future that might change.

    October 19, 2018

    https://www.duolingo.com/profile/lizsue

    I'm not saying you proposed to get rid of AI! :)

    I'm just saying that I got the impression that any real-time personalization during the practice sessions themselves is done by AI instead of volunteers.

    After all, those practice sessions are available all day - not only when at least one volunteer is awake to watch and personalize them during the time the user is practicing.

    October 19, 2018

    https://www.duolingo.com/profile/Larisa_L

    Naturally, there are tons of things which are automated on Dulingo, not only in practice sessions, in any session as well - the order in which the exercises are given in the skill, the grading, giving points and lingots - all of that is done by algorithms.

    October 21, 2018

    https://www.duolingo.com/profile/FitSugarberg

    i love this post ! AI will never be as accurate and smart as man cause algorithms are not creative, and we are

    October 21, 2018

    https://www.duolingo.com/profile/garpike

    Whatever creativity you are are talking about is just another algorithm that we possess and AI doesn't yet. Don't expect that to remain the case; there's nothing mystical about human creativity—it's just a complicated data function.
    We shall probably be able to simulate the whole human brain within a decade or so (perhaps less). Such a simulation would provide output indistinguishable from that of a person; in theory we could simulate the brain of Leonardo Da Vinci.

    October 22, 2018

    https://www.duolingo.com/profile/Larisa_L

    Good luck with that, especially within a decade :))) In 5 years incubator exists we are nowhere near replacing people with AI and you're talking about replacing people on a much bigger scale.

    I also used to think that we are pretty close to human-like computers, but 2 things changed my mind:

    1) I work in IT, and I know how things are done - no, there is no way to create real intelligence quickly.
    2) I have a little son now. Seeing his development and experiencing all the complex parental emotions I conclude there is still a lot left to humanity, which simply doesn't need to be replaced by computers. We want to and can do those things ourselves.

    October 22, 2018

    https://www.duolingo.com/profile/garpike

    I was talking specifically about simulating the brain. Serious research into this is happening. I wasn't suggesting this was of direct and immediate relevance to the incubator (in a decade, however, who knows—if you work in IT, you'll undoubtedly appreciate how much can happen in ten years).

    I'm not suggesting that our complex emotions should be replaced by computers; merely pointing out that they could be. We biological computers are losing our monopoly over the algorithms of advanced cognition and decision-making. Yes, I really do think ten years.

    October 22, 2018

    https://www.duolingo.com/profile/FitSugarberg

    i dont know if my comment was posted ; so i agree and like what you say

    October 21, 2018

    https://www.duolingo.com/profile/Ontalor

    When he talks about rearranging skills, he doesn't mean the AI is doing that. The AI can figure out which topics people learn better in what order, and then the next version of the tree can be modeled with that in mind. The creation of the next tree still does take us to create, though.

    October 23, 2018

    https://www.duolingo.com/profile/ngraner42

    I am very greatful for the hard work of the volunteers making Duolingo possible.

    I can imagine how in the future the process could be taken over by an AI. Especially as more translated content becomes electronically available. The AI can pull translated sentences calcullate the word frequencies and make an algorithmic judgement of the gramatical complexity. Will it make mistakes, of course, but that is how the AI learns. It gives a sentence to beginning students and sees that they are struggling with it, so the sentence either gets pulled or moved to a later lesson. With just the single minded goal of providing lessons that result in student success (getting it right in subsequent attempts) it can refine what sentences to give at what point in the learning process. This dynamism can also make sure to stop or delay giving sentences or words to people who have mastered them, making the coures personalized to each student.

    October 19, 2018

    https://www.duolingo.com/profile/Larisa_L

    There are many issues with such approach, but it should be possible in the future. Currently the main problem within Duolingo system would be - who would translate all those sentences selected by AI? This is all done by humans and computer translators are not good enough to do that on their own.

    October 19, 2018

    https://www.duolingo.com/profile/ngraner42

    I was thinking that more and more translated content will become available on the internet. Certainly the availability of good translations is necessary and at least for the near term requires a human to do the translation.

    October 19, 2018

    https://www.duolingo.com/profile/Larisa_L

    The challenge with Duolingo is that each sentence has not just one but hundreds and sometimes even thousands of possible translations added. This approach allows all this amazing freedom which most users do not appreciate in any case, all they end up noticing is the few of those translations we didn't manage to add. :)))

    October 21, 2018

    https://www.duolingo.com/profile/ngraner42

    That is a good point that I was not thinking about. Lately I have been doing the pick a word option that basically leads you into one possible answer.

    October 22, 2018

    https://www.duolingo.com/profile/Larisa_L

    Yes, this option to build the sentence from a given set of words naturally leads you to only one answer. But any user can switch to a free writing at any moment (there is a button for that) and type in their own version.

    October 22, 2018

    https://www.duolingo.com/profile/Izim_M
    • 1653

    I understand the reaction, but I believe that in the long run AI is bound to take over all human tasks. There is no human task that a computer/robot won't be able to do. It's just a matter of creating sufficiently complex robots, which may take decades. After all, human beings and other animals are also just 'computers' made of organic material instead of silicon.

    This is (and will be more and more) a difficult ethical problem. It doesn't mean any human effort while the technology isn't there yet, is wasted.

    Automation is ethically difficult.

    October 19, 2018

    https://www.duolingo.com/profile/2348104534938

    how can I volunteer

    October 19, 2018

    https://www.duolingo.com/profile/Lolmander2006

    If you speak more than one language, you can use the incubator to teach the language you know to people that speak the other language you know. The website is here: https://incubator.duolingo.com/

    October 19, 2018

    https://www.duolingo.com/profile/mckaygerhard

    i totally agree with you.. seems, It seems that duolingo scammed you, used you and now it was nourished by you .

    October 20, 2018

    https://www.duolingo.com/profile/zyxwvutsrqpo

    I think you're sort of taking it a bit personal that AI is being called the future. AI being the future =/= "volunteers aren't important".

    And it doesn't necessarily mean the end of volunteers, it could mean that contributors could for example become coaches. Something I think teachers in schools should also be, instead of "tutors" they should become coaches.

    A strong and AI supported system to enable people and coaches to support you on your way is the ideal. Ultimately as AI becomes very advanced it might also partially take over the role of coach, but that's even a bit further out into the future.

    But I do also question a bit how it currently personalizes your learning, perhaps it's doing more than I realize though. I've sort of been trampling in place lately due to blablabla thus in general it's difficult to get a good sense of it especially without knowing how it works :P. In general I absolutely love the learning is being managed to optimize it for students :).

    One thing I very strongly disagree with is Luis's "the United States is fortunate enough to have a rather well-developed education system". I think that's very biased on personal experience. First off, I'm not from the US.

    But where I live we have a teacher rambling in front of the class and writing things on the blackboard. Sorry, but that's not teaching - at all, and I highly doubt it's any different in the US. It's so extremely ineffective, inefficient, students forget most shortly afterwards, it kills the mind of students, etc...

    The best experiences I've had with education were where there was a very high degree of autonomy, where you're being presented with material to ENABLE you to learn and are expected to complete exercises on your own and to figure everything out on your own using well built materials for that purpose, with the teacher being available to help whenever you get stuck (coaching role). This is currently a rare thing, but typically these teachers have a higher degree of students passing the class, you tend to remember better what you've learned (since you've figured it out mostly on your own instead of just quickly scrabbling down notes for the duration of the class) and it's a lot less boring / sleep provoking.

    Personally I'm more than done with traditional "education". Out of the classes I'm currently taking, only 1 teacher realizes it's 2018, all the others are stuck in the previous century. I'm currently skipping like half of all classes simply because they have absolutely 0 value, I'd even call them net-negative: they cost time to go to, you don't really learn anything while you're there because it's taught in a terrible way and they leave you too tired / bored to do anything productive when you get back home. Isn't it absolutely hilarious how education tries to teach you science, and how you should do it properly...while completely ignoring science when it comes to teaching? If you want to state something or do something, you have to prove its worth, but education? Nope, everything done in the name of education is supposedly true / good by default and because traditionalism (who ever first said or did it totally had a good a reason for sure, even if nobody knows what those reasons supposedly are, and will selectively seek / build "proof" for the premade conclusion that it's worthy). Sometimes they say that you should questions things, but don't you dare to question education!

    I'm pretty sure that Luis has an evolutionistic view on education: improve it using modern technology. But I think that's extreeeeeemely naive and especially long term it more likely hurts his own cause more than it helps it. Traditional education is stuck in the past, it will stay stuck there, barely questions or improves its methods, has tons of artificial barriers and it's extremely clunky to make it turn direction. Yeah, you can improve traditional education through modern technology, but it will stay a lot less effective compared to what is possible and will keep on being way more costly than necessary (both for students and society). If you want change, you must disrupt the cancer that traditional education really is, forcing it to improve itself or else to vanish as better means of teaching take over. What is needed is REVOLUTION, not evolution. Traditional education isn't able to keep up with technological advances, hasn't been for a long time and won't be anytime in the near future. Traditional education shouldn't be kept alive by evolution, or worse, keep being propped up as "the ideal". The only thing that will cause true and significant change is revolution, disrupt their educational model and force them to adapt to modern times or to vanish.

    Unlike companies, traditional education doesn't really have significant competition (yet). Just like companies in markets that lack any competitiveness, they barely innovate or improve themselves because they don't really have to. If you want general improvement in the education available to everybody: go for revolution instead of evolution and give traditional education a good reason to improve itself through competition.

    As inevitably some will think / say that education (universities) are the front-runners of science, is it really by virtue of them or in spite of? Science is monopolized by traditional education, with high barriers of entry for those who don't want to play by the rules of traditional education. Just look at the availability of scientific papers, happy hunting when you're not a scientist connected to a university. Knowledge being locked away within artificial walls and unmined potential locked out.

    October 18, 2018

    https://www.duolingo.com/profile/Larisa_L

    I'm not against AI by any means :)))) I'm a data analyst, I'm a big proponent of Online Education and Self-Driving cars. My point was that currently Duolingo's success is not generated by AI.

    I'm sorry to hear that you have had some negative educational experience. I got luckier, I've attended Harvard and Oxford online (saving money and paying from my own pocket) and it was worth all the money, time and effort (and I'm from Kazakhstan, so no advantages anywhere). Also my husband is a scientist, and he naturally has more access to scientific papers than I do, but if one really needs something, then there is always a way, scientific community is actually very open and provides lots of things for free (unlike businesses). I don't think that competition is the future for education, yes, you should have choice, but it shouldn't be run like a pure business, as it's a public good and would be under-produced in a free market. But this is something of a separate topic. It would take too long to discuss it here in detail.

    October 19, 2018

    https://www.duolingo.com/profile/zyxwvutsrqpo

    My point was that currently Duolingo's success is not generated by AI.

    Sure, true.

    I agree that competition in education absolutely isn't a must. The issue in my opinion there's an enormous lack of a desire for improvement in traditional education. At least where I live there are tons of teachers who are even heavily against using technology in education, something that will only go away as that generation of older teachers retires. Even at universities there's an enormous lack of drive for improved teaching methods. The one teacher that realized it's 2018 and build a great platform for teaching that has been adopted by several other universities is seemingly to some degree being shuffled aside by his colleagues who are stuck in the previous century. It's absolutely not impossible or difficult, as this one teacher has more than proven, it's because most others don't care enough (you can't really call it anything else). It's 2018, there's absolutely no excuse for even courses that involve programming to be "taught" by a teacher rambling in front of a class (whoever thinks you can learn programming like that is delusional). The good teacher makes sure you learn everything by having to solve increasingly difficult problems on your own but with a system that allows you to immediately check how well your solution works and also gives a bit of automated feedback (sort of building a strong foundation brick by brick). While for example in an other class, you're just handed a string of code during a "teacher rambling in front of a class" course, of which you can obviously learn what it does but it doesn't really make you capable of building similar ones on your own (sort of stacking a series of bricks on top of one another, it might keep on standing for a while but it's guaranteed to collapse after a while [forgetting it] - thus if you really want to learn it you have to invest extra time in looking for other ways to actually really learn it instead of just passing the test without really being able to do it). The poor teaching methods require you to invest a ton more time for a worse outcome (lesser knowledge / capabilities and forgetting it way sooner).

    Thus that's why I'm saying that competition is the only way to get traditional education out of the era of prehistoric teaching methods (since traditional education is currently the norm, thus to also get education in general out of prehistory). In general I absolutely don't think that competition in education is a must, there should be a drive for improvement and optimal teaching by default - currently there simply isn't and traditional education won't any time soon. It's a part of the "company" culture in education, something you can't change without drastic measures. Companies overcome such cultural difficulties by doing things like firing people stuck in the past or for example by creating a spin-off composed of more innovation / improvement / change minded people. You can't really create change solely by efforts aimed at getting everybody on the innovation train, as many people simply don't want to or simply don't have the skill set. That's why the only way to REALLY cause drastic improvement in education is by disruption, where traditional education would also be forced to follow suit or to die out.

    For example, it's really cool and all if you have a free or cheap educational platform to evolve education used by many traditional institutions, while the degrees are still solely being handed out by expensive traditional institutions and a lot of ineffective traditional classes still being given next to it. In that case, sure you've caused some improvement, but way less compared to the potential of the platform. And arguably a bigger issue: your system propped up the power of traditional education, they're still the ones with the most power and it will only become more and more difficult for you to cause real change. If you for example don't like that traditional institutions are still way too costly, way more costly than essentially necessary, what power do you still have? At that point if you threaten to pull out they'll just switch to a different platform and continue their BS. Meanwhile you've also lost of the disruptive potential that you had before - making it so much more difficult to cause real change. I don't know to what degree it's being understood what I'm trying to say, kinda difficult to explain.

    October 19, 2018

    https://www.duolingo.com/profile/benmaja

    Fantastic, elvper - please don't be discouraged by the downvotes (I've put my one upvote in to try to reduce them.) Most kids would agree with you - and they are the future. You are right that there is no real competition, because school is compulsory and adults - government, parents, teachers - would be terrified of making it voluntary. I'm a grandmother, and of course I have encouraged our family's kids to learn traditionally, but I'm really with you here.

    October 22, 2018

    https://www.duolingo.com/profile/zyxwvutsrqpo

    Thank you. I do think education in general should be compulsory but that it should be more oriented at enabling people to do what they like and/or are good at, which in theory should only yield advantages for both the kids and society. Especially with how accessible entertainment has become, computer games vs learning about things they don't care about is a battle you can't really win, certainly not without a cost. If you instead enable kids to pick their path to a fair extent so they can learn about what interests them most, there's a good chance they might actually start to learn eagerly about it and actually enjoy it. Plenty of people are willing to learn from nature out, if education would only support them in it instead of rigidly dictating what kids should learn about. With the added question of to what extent it really is "teaching" since it's just aimed at passing a test, after which you forget most again because of a poor teaching method.

    October 22, 2018

    https://www.duolingo.com/profile/zyxwvutsrqpo

    In addition: 81% (!!!) of students who have finished secondary education don't know what to study next (where I live). Does that sound even remotely like a successful education system? That while the education in my country is still supposed to be one of the better ones in world.

    Traditional education is terrible, and it's painfully obvious to anybody willing to truly investigate it. I know it's emotionally difficult for people to be willing to even consider that something that costs / costed them 25% of their lifetime and something they've build their identity out of was for a large part a waste of time (frustrates me too). People believing in traditional education is not much different from religious beliefs: people mostly only have that belief because it has been forced down their throats ever since a very young age, while also being forced to accept it without thinking by your environment.

    October 28, 2018
    Learn a language in just 5 minutes a day. For free.