Test Center - My Experience
I tried out the Duolingo Test Center and I wanted to share my experience as a native English speaker.
Though I initially had some technical difficulties , these were resolved and I was able to complete the test.
There were some challenging questions that I think even many English speakers would have had difficulty with. Nonetheless, I received a perfect score on the test (10.0/10), which my certificate indicates means I am an "Expert. Can understand virtually anything heard or read, even intellectually demanding material such as an academic lecture or a book on philosophy. Can use the language fluently and spontaneously in a way that can even be more advanced than an average native speaker." Given my profession as a teacher of English as a Second Language and my experience as a published writer of materials for teachers of literacy and a reviewer of academic texts, I would judge this to be a fair assessment of my grasp of the English language.
The length of the test I think lends both validity and reliability to its results, as it is long enough to pose questions at a range of skill levels and short enough that fatigue never played a factor.
The mandates not to look away from the screen and to keep my face completely in view of the camera were somewhat of a challenge, particularly for some of the more difficult questions. Sometimes when I have to concentrate on something I tend to unfocus my eyes and stare off into space, which I could not do during this test while guaranteeing that I kept my eyes pointed at the screen. Similarly, staying focused on where my face was within the camera's view distracted somewhat from my focus on the questions themselves. I understand these to be necessary precautions to insure the honesty of all test-takers, but it does raise questions about reliability of the exam for people who may need greater concentration on a second language.
Both times I started the test, the first question was a "write-what-you-hear" type of question. I was wise enough to turn the sound up beforehand the second time, but the first time I had to turn up my volume and have the app repeat the question before I could answer it. Since there is a time limit for each question, this could cause problems. I suggest that there be a reminder before beginning the test to make sure that the device's volume is turned up (particularly since headphones are forbidden).
As with all other features of Duolingo, the test lacks a conversational component. The test also lacks other features present in the TOEFL to demonstrate language production, such as the written component. The cloze questions on the test measure production to a small degree, although this is restricted as well by the multiple-choice nature of those questions. Similarly, the spoken questions are useful for demonstrating fluency of pronunciation, but not much beyond that.
Overall, I would say that this test is an excellent measure of receptive language skills, but it lacks rigorousness in its measure of productive skills. A written and/or conversational component would help remedy this problem, but would also require more attention by the staff than simply monitoring the eye-tracking software that is presumably being applied, and would therefore be likely to drive up the price. For an affordable option for people that need to demonstrate language competence, I think it is an excellent tool and I will be recommending it to the coordinator of the adult ESL program at the center where I work.
This looks like a fair assessment of the test. Composition or the spoken component will always be hard to evaluate without a human reading the test results. That is at least until an AI system is good enough to evaluate such things.
Given that academic institutions or the industry requires average production skills I think this may be something that Duolingo needs to evaluate and attempt to improve.
Anyway, how would you rate the test on a scale of 1-10, and taking into account its limitations in evaluating the productive skills?
I'm concerned by revdolphin's description of receiving what sounds to be a single score at the end of the test based on responses to both spoken and written questions. Giving a single score like that guarantees that deaf test takers will receive poor scores even if they read and write English flawlessly. I suggest providing separate scores for spoken and written comprehension. I'm deaf, and I freely acknowledge that my comprehension of spoken English is terrible, but would hate to be barred from jobs requiring WRITTEN English on the basis that I can't hear!
Was it all graded by AI? Based on the initial launch information, I was always under the impression that someone would be grading the results.
Of course this would allow for some flexibility in staring at the screen (perhaps closing your eyes to avoid signs of cheating) and more conversational and larger written portions.
No human grades the test, as far as I know. As it would need competent examiners (and more expensive) to do so. The only human participation is a person that ensures that you're not cheating by checking the video.
That's the biggest challenge with this or any kind of assessment, it takes hours and committed examiners to mark tests, and even they make mistakes or make random decisions about what is right or wrong sometimes.
Nice review revdolphin. I always wanted to take an English language assessment like this but could not afford the time and money for the TOEFL or IELTS. Their quite expensive here in the Philippines and is equal to roughly my pay for a month and are administered it just a few locations. I believe the Duolingo tests would be easier to validate for universities if they supply the proctored video along with the exam results. I talk a lot in the video. hehe The videos can serve as the Spoken component and then the written will be the answers you gave on the app.