Test Center - My Experience
I tried out the Duolingo Test Center and I wanted to share my experience as a native English speaker.
Though I initially had some technical difficulties , these were resolved and I was able to complete the test.
There were some challenging questions that I think even many English speakers would have had difficulty with. Nonetheless, I received a perfect score on the test (10.0/10), which my certificate indicates means I am an "Expert. Can understand virtually anything heard or read, even intellectually demanding material such as an academic lecture or a book on philosophy. Can use the language fluently and spontaneously in a way that can even be more advanced than an average native speaker." Given my profession as a teacher of English as a Second Language and my experience as a published writer of materials for teachers of literacy and a reviewer of academic texts, I would judge this to be a fair assessment of my grasp of the English language.
The length of the test I think lends both validity and reliability to its results, as it is long enough to pose questions at a range of skill levels and short enough that fatigue never played a factor.
The mandates not to look away from the screen and to keep my face completely in view of the camera were somewhat of a challenge, particularly for some of the more difficult questions. Sometimes when I have to concentrate on something I tend to unfocus my eyes and stare off into space, which I could not do during this test while guaranteeing that I kept my eyes pointed at the screen. Similarly, staying focused on where my face was within the camera's view distracted somewhat from my focus on the questions themselves. I understand these to be necessary precautions to insure the honesty of all test-takers, but it does raise questions about reliability of the exam for people who may need greater concentration on a second language.
Both times I started the test, the first question was a "write-what-you-hear" type of question. I was wise enough to turn the sound up beforehand the second time, but the first time I had to turn up my volume and have the app repeat the question before I could answer it. Since there is a time limit for each question, this could cause problems. I suggest that there be a reminder before beginning the test to make sure that the device's volume is turned up (particularly since headphones are forbidden).
As with all other features of Duolingo, the test lacks a conversational component. The test also lacks other features present in the TOEFL to demonstrate language production, such as the written component. The cloze questions on the test measure production to a small degree, although this is restricted as well by the multiple-choice nature of those questions. Similarly, the spoken questions are useful for demonstrating fluency of pronunciation, but not much beyond that.
Overall, I would say that this test is an excellent measure of receptive language skills, but it lacks rigorousness in its measure of productive skills. A written and/or conversational component would help remedy this problem, but would also require more attention by the staff than simply monitoring the eye-tracking software that is presumably being applied, and would therefore be likely to drive up the price. For an affordable option for people that need to demonstrate language competence, I think it is an excellent tool and I will be recommending it to the coordinator of the adult ESL program at the center where I work.
This looks like a fair assessment of the test. Composition or the spoken component will always be hard to evaluate without a human reading the test results. That is at least until an AI system is good enough to evaluate such things.
Given that academic institutions or the industry requires average production skills I think this may be something that Duolingo needs to evaluate and attempt to improve.
Anyway, how would you rate the test on a scale of 1-10, and taking into account its limitations in evaluating the productive skills?
Thanks so much for the feedback, revdolphin! We are working to improve both the UI experience and the test itself, and will definitely take all of these points into account!
I'm concerned by revdolphin's description of receiving what sounds to be a single score at the end of the test based on responses to both spoken and written questions. Giving a single score like that guarantees that deaf test takers will receive poor scores even if they read and write English flawlessly. I suggest providing separate scores for spoken and written comprehension. I'm deaf, and I freely acknowledge that my comprehension of spoken English is terrible, but would hate to be barred from jobs requiring WRITTEN English on the basis that I can't hear!
This is an excellent point DuoLib, and something we considered, although for simplicity we went for an "all-skills" assessment for the initial version of the English test. In future versions of Test Center, we plan to find ways of accommodating all potential test takers!
It's good to hear that future versions of Test Center will more accurately measure the skills of deaf test takers. In many places it's extremely difficult for deaf people to obtain education or skilled employment, and misleadingly-low test scores will provide an additional barrier.
By the way, SOME people with limited hearing will do okay with listening questions if they wear hearing aids. Since headphones are forbidden during tests, it would be good to ensure that the test monitors are trained in recognizing the variety of hearing aids on the market, and understand that they're used for hearing, not cheating.
Was it all graded by AI? Based on the initial launch information, I was always under the impression that someone would be grading the results.
Of course this would allow for some flexibility in staring at the screen (perhaps closing your eyes to avoid signs of cheating) and more conversational and larger written portions.
No human grades the test, as far as I know. As it would need competent examiners (and more expensive) to do so. The only human participation is a person that ensures that you're not cheating by checking the video.
That's the biggest challenge with this or any kind of assessment, it takes hours and committed examiners to mark tests, and even they make mistakes or make random decisions about what is right or wrong sometimes.
I read the thread and I understand what you mean. The experts are the ones who designed the test, and tested it internally before allowing external researchers to conduct their test. That is probably the extent of their participation, they will not in any way review or conduct the test. It wouldn't be reasonable anyway. Imagine if 1 million of Duolingo's users take that test, it would require many more examiners to mark in any reasonable time frame.
Nice review revdolphin. I always wanted to take an English language assessment like this but could not afford the time and money for the TOEFL or IELTS. Their quite expensive here in the Philippines and is equal to roughly my pay for a month and are administered it just a few locations. I believe the Duolingo tests would be easier to validate for universities if they supply the proctored video along with the exam results. I talk a lot in the video. hehe The videos can serve as the Spoken component and then the written will be the answers you gave on the app.