I’m a fan of technology. I believe that most things can be made better via automation, virtualization and all the other *-tions that introduce technology into equation. In most cases, if something is preceded by “electronic” or “online” it means better. There are exceptions of course. One exception is called shitty implementation and unfortunately happens roughly 70% of the time. It happens to every one – that’s why we have websites like The Daily WTF.
Then there are things that should not be automated, mechanized or put online. One of these things is test taking. I dislike taking electronic tests, and as a rule I avoid giving them. I simply think they do not work as well as most people think they do. Sure, they are easy to administer, easy to grade, and easy to generate and randomize. You can build a database of questions, rate them by challenge level, and randomly generate unique tests of equal difficulty ad infinitum. But I believe they are not entirely fair to the students who take them.
For one, students are trained to take paper based tests. In most schools and universities, nearly all exams are administered using traditional methods such as blue books, or Scantron sheets or simple printouts that students fill out by hand and turn in. When you ask them to take an online test, you are working against years of conditioning and test taking habits. They are faced with unfamiliar format, unfamiliar input methods and a new interface. In fact a lot of test taking engines introduce distracting or stressful features that may negatively impact final scores.
For example, some electronic tests like to display the test score, or notify the taker whether they got the question right or wrong immediately after they have submitted the answer. I believe this is a mistake as it may easily create a negative feedback loop. A student who gets many questions in a row wrong, gets progressively more and more stressed. Each wrong answer adds to the pressure to the point of panic where most of the concentration is lost.
Displaying a ticking clock, elapsed time, or time left for the test may have similar effect. Students often start to do a lot of random guessing when they notice they are behind the clock. This goes double for engines which require that questions are answered within some specific time limit. Especially if test penalizes you for leaving blank answers.
These distractions and negative feedback loops do not exist in paper based tests. Students have unlimited amount of time to work on each question, and can keep track of the elapsed time at their own measure. They also do not know whether or not they got a given question right or wrong so they do not have to think about their score until after the test. Naturally, over time they could learn to ignore these types of distracting feedback. But this takes practice, and requires a uniform interface. But as I mentioned above, most students do not get to take online exams very often and thus lack the familiarity with the medium.
Electronic tests require a different test taking strategy. For example, one of the fundamental test taking tips is to skip difficult questions and then get back to them at the end. This prevents you from getting stuck on a difficult problem, and loosing too much time. Good test takers will tackle problems in an increasing order of difficulty to maximize the number of correct questions. Is this possible with electronic tests?
It really depends on implementation. Some test engines do not provide this functionality. Others do it by looping the questions around. When you deplete the list of questions, it starts showing you your skipped questions. Unfortunately most implementations do not allow for the level or prioritization allowed by a paper based test.
Electronic tests also do not allow you to back-track and “fix” the previously answered question. Test takers are often discouraged from doing this, implying that their first instinct is usually correct. Of course one has to remember most tests are flawed in such way that one question may give hints for solving a previous one. I will give you an example. Let’s say a test taker is asked for a definition of a buzzword, given 4 choices. He eliminates two of them right off the bat but not knowing the definition he makes a guess. Later on, the buzzword comes up again in one of the answers and through elimination the test taker is able to extrapolate that the have answered the previous question wrong.
Some people may say this is irrelevant, because the test taker did not know the answer to begin with. But the ability to connect the two questions, and correct the answer shows that the test taker is good at logically reasoning out hard problems based on given information. It may also indicate deeper understanding of the subject at hand. This is more valuable than mere ability to regurgitate memorized material and should be rewarded.
Electronic tests hardly ever allow you to change an answer that was already given. In fact, test engines that immediately indicate success or failure to answer a question make this type of reasoning impossible.
Not to mention that the test takers attention span may be different depending on the medium. It has been shown by countless studies that people tend to get bored and distracted with electronic, online and on-screen media much faster than with paper based media. That’s why most Youtube videos are short. That’s why blog posts tend to be short, and digestible in one sitting. So using medium that is commonly associated with fast paced, short attention span, burst communication to administer a long, drawn out test is not the best idea. Because of this attention span dissonance, a lot of electronic tests tend to seem more draining and exhausting than they should be.
These are the reasons why I dislike taking, and/or giving tests using electronic medium. I prefer good old scantron technology which combines a paper based test with a machine assisted grading. You just need to make sure you bring #2 pencils for everyone.
Of course, if you ignore the attention span problem, it would be possible to design an electronic test that would aim to emulate paper based experience. Make it look like an actual test booklet with several questions per page. Each question would have an active area where the student would check or type in answers. The engine would allow the student to scroll up and down, and flip pages, answer questions in random order even read the whole test without answering any questions without penalty. It would also have to allow the student to go back and fix previous answers at any time. In fact, such an interface could be easily designed using the all-present Web 2.0 design style and philosophy. But I haven’t seen anything like that in use yet. Have you?
Besides, a multiple choice tests can be machine-graded whether they are administered online or not as long as you use Scantron or similar scannable test sheets. Essay questions on the other hand must be graded “by hand” regardless of the medium. Online versions remove the issues with deciphering illegible handwriting, but other than that offer the same grading experience. So why not give students the paper based tests they are used to. If for nothing else, just for the tactile experience.
RSS Feed
/dev/random