I’m a fan of technology. I believe that most things can be made better via automation, virtualization and all the other *-tions that introduce technology into equation. In most cases, if something is preceded by “electronic” or “online” it means better. There are exceptions of course. One exception is called shitty implementation and unfortunately happens roughly 70% of the time. It happens to every one – that’s why we have websites like The Daily WTF.
Then there are things that should not be automated, mechanized or put online. One of these things is test taking. I dislike taking electronic tests, and as a rule I avoid giving them. I simply think they do not work as well as most people think they do. Sure, they are easy to administer, easy to grade, and easy to generate and randomize. You can build a database of questions, rate them by challenge level, and randomly generate unique tests of equal difficulty ad infinitum. But I believe they are not entirely fair to the students who take them.
For one, students are trained to take paper based tests. In most schools and universities, nearly all exams are administered using traditional methods such as blue books, or Scantron sheets or simple printouts that students fill out by hand and turn in. When you ask them to take an online test, you are working against years of conditioning and test taking habits. They are faced with unfamiliar format, unfamiliar input methods and a new interface. In fact a lot of test taking engines introduce distracting or stressful features that may negatively impact final scores.
For example, some electronic tests like to display the test score, or notify the taker whether they got the question right or wrong immediately after they have submitted the answer. I believe this is a mistake as it may easily create a negative feedback loop. A student who gets many questions in a row wrong, gets progressively more and more stressed. Each wrong answer adds to the pressure to the point of panic where most of the concentration is lost.
Displaying a ticking clock, elapsed time, or time left for the test may have similar effect. Students often start to do a lot of random guessing when they notice they are behind the clock. This goes double for engines which require that questions are answered within some specific time limit. Especially if test penalizes you for leaving blank answers.
These distractions and negative feedback loops do not exist in paper based tests. Students have unlimited amount of time to work on each question, and can keep track of the elapsed time at their own measure. They also do not know whether or not they got a given question right or wrong so they do not have to think about their score until after the test. Naturally, over time they could learn to ignore these types of distracting feedback. But this takes practice, and requires a uniform interface. But as I mentioned above, most students do not get to take online exams very often and thus lack the familiarity with the medium.
Electronic tests require a different test taking strategy. For example, one of the fundamental test taking tips is to skip difficult questions and then get back to them at the end. This prevents you from getting stuck on a difficult problem, and loosing too much time. Good test takers will tackle problems in an increasing order of difficulty to maximize the number of correct questions. Is this possible with electronic tests?
It really depends on implementation. Some test engines do not provide this functionality. Others do it by looping the questions around. When you deplete the list of questions, it starts showing you your skipped questions. Unfortunately most implementations do not allow for the level or prioritization allowed by a paper based test.
Electronic tests also do not allow you to back-track and “fix” the previously answered question. Test takers are often discouraged from doing this, implying that their first instinct is usually correct. Of course one has to remember most tests are flawed in such way that one question may give hints for solving a previous one. I will give you an example. Let’s say a test taker is asked for a definition of a buzzword, given 4 choices. He eliminates two of them right off the bat but not knowing the definition he makes a guess. Later on, the buzzword comes up again in one of the answers and through elimination the test taker is able to extrapolate that the have answered the previous question wrong.
Some people may say this is irrelevant, because the test taker did not know the answer to begin with. But the ability to connect the two questions, and correct the answer shows that the test taker is good at logically reasoning out hard problems based on given information. It may also indicate deeper understanding of the subject at hand. This is more valuable than mere ability to regurgitate memorized material and should be rewarded.
Electronic tests hardly ever allow you to change an answer that was already given. In fact, test engines that immediately indicate success or failure to answer a question make this type of reasoning impossible.
Not to mention that the test takers attention span may be different depending on the medium. It has been shown by countless studies that people tend to get bored and distracted with electronic, online and on-screen media much faster than with paper based media. That’s why most Youtube videos are short. That’s why blog posts tend to be short, and digestible in one sitting. So using medium that is commonly associated with fast paced, short attention span, burst communication to administer a long, drawn out test is not the best idea. Because of this attention span dissonance, a lot of electronic tests tend to seem more draining and exhausting than they should be.
These are the reasons why I dislike taking, and/or giving tests using electronic medium. I prefer good old scantron technology which combines a paper based test with a machine assisted grading. You just need to make sure you bring #2 pencils for everyone.
Of course, if you ignore the attention span problem, it would be possible to design an electronic test that would aim to emulate paper based experience. Make it look like an actual test booklet with several questions per page. Each question would have an active area where the student would check or type in answers. The engine would allow the student to scroll up and down, and flip pages, answer questions in random order even read the whole test without answering any questions without penalty. It would also have to allow the student to go back and fix previous answers at any time. In fact, such an interface could be easily designed using the all-present Web 2.0 design style and philosophy. But I haven’t seen anything like that in use yet. Have you?
Besides, a multiple choice tests can be machine-graded whether they are administered online or not as long as you use Scantron or similar scannable test sheets. Essay questions on the other hand must be graded “by hand” regardless of the medium. Online versions remove the issues with deciphering illegible handwriting, but other than that offer the same grading experience. So why not give students the paper based tests they are used to. If for nothing else, just for the tactile experience.
Before I start – i agree with your post completely.
But it did lead my thoughts onto some related things.
For a recent job they asked me to do an online test – basic literacy, numeracy and IT skills.
These were tests administered by an external company which, presumably, specialises in these sorts of testing – they weren’t something knocked up by a secretary or anything silly like that.
The IT skills was things like making a page in word landscape, printing via print preview and so on.
These were a series of screenshots with the pre-XP styling, which were set up with what were essentially hot spot links. The only feedback you got was the screen changing (to another predefined screenshot) if you hit the right spot, there was no negative feedback, and you couldn’t get lost in the wrong menus as they didn’t work.
There was also no final feedback – So when you clicked ‘print’ or similar before clicking ‘done’ you had no idea whether your last click had been registered (which it often wasnt since the button tended not to cover the whole button).
Due to the nature of the job, far more advanced computer skills and literacy were requried – and this was by no means entry level, /they/ had approached /me/.
So entirely useless tests, administered in a broken manner. I ended up telling the head hunter that I didn’t think that testing these things bode well for the position and withdrew.
My other experience of electronic testing was the UK driving theory test which has to be passed before you can take a practical driving test.
This worked very well. It was a series of multiple choice questions, which didn’t requrie any working out (I’m sure you all know the type of question). Followed by a hazard awareness test where you watched a film and had to click the mouse whenever you saw a potential hazard.
This test is obviously very different from the sort of testing you are talking about. For a start the multiple choice aspect is allot easier to do online.
It was done in a controlled environment and increased throughput – rather than having to marhsall a room of 20/50 people all in one go, at a set time – they staggered the starts, walking people to and from the computers as neccersary.
The people taking the test are very different – they are a complete cross section of society – all ages (above 17), all levels of education etc etc.
Many of these people, rather than having been trained to take paper tests are more likely to have developed a fear/apprehension of anything resembling an exam. By doing it online, in a smaller room it becomes far less daunting for many people.
And obviously the hazard perception would have been difficult on paper. ;)
I guess testing is like voting – there are distinct advantages of using a paper based system and these advantages cannot just be thrown away.
I can’t even imagine doing a maths exam without a pencil and piece of paper.
I’m afraid I’ll have to disagree with your post totally, this time…:)
As a former high-school teacher, and now a university student (and teaching assistant), I find electronic testing superior in most ways. Perhaps it’s because the systems I have used from both sides do not appear to suffer from the limitations you outlined above.
The majority of the online testing I have been involved in has been multiple-choice. I’m not the biggest fan of these – the whole recognition/recall assessment dichotomy has me sitting on the recall side. You really know something when you don’t need to see the word to remember it.
The system I have most involvement with now is called “Blackboard”, and whilst I have some issues with the implementation and performance (it’s a fairly clunky Java applet), it allows you to:
1. revisit questions and change them later.
2. see quickly which questions you have and haven’t answered.
3. do not provide immediate feedback. In fact, grading is not provided until the test has been “released”, which means other students cannot be given solutions by a student who has completed the task
The only thing that is (according to your post) a disadvantage is that it has a countdown timer. Which every examination room I have ever been in has had in a sense: a clock on the wall showing the time, and notifications of how long remains. The online systems just mean that everyone sees this all of the time.
As for the design: it’s a single scrollable list, with multiple-choice (or true-false, or short-answer) questions listed on after another. There is a fixed panel on the right with a table containing which questions have been answered, which have been changed and not saved, and which have not yet been attempted.
The advantages of this system – obviously the marking time is reduced. Also, errors during the marking process are also reduced. It is possible to have a large pool of questions, which can be used to make up many practice tests, without necessarily giving away all of the real questions. The answers can be placed in random order for each student, meaning it is harder to cheat off one another (“A-B-A-C-A-B-D”).
Granted, here in Australia we are not trained like the US to take tests in the same way. Examinations are often short-answer, which are by definition much harder to mark. And our pencils have names like “H”, “HB”, “2B” and so on. Not sure how they match up with your #2…
:)
@James Heaver: The theoretical driving test here in US is a bit similar, without the road awareness part. It is all multiple choice, and it is possibly one of the worst offenders.
You can’t skip questions without being penalized, you can’t go back and you can’t correct the answer. Each question has a short time limit, and you are told which one you got right or wrong. You also see your score so far, and allegedly the exam stops when your score falls below some threshold.
I also took GRE’s which are a standardized test and it was a bit better but still fairly clunky. It wouldn’t let you move back and forward through questions but it would loop them back if you passed them. But, you couldn’t move between sections of the exam. And it had a big ticking clock in the corner which was unnerving – and you had limited time per section.
I remember that taking SAT’s was much more pleasant experience thanks to the paper based format – at least they were paper based when I took them. I’m not sure how it is done now. The electronic GRE’s on the other hand were nerve wrecking.
@Matt Schinckel: Ah, good point about Blackboard. When I wrote this I was for some reason thinking more among the lines of the more controlled environments such as you see in standardized testing centers, DMV and etc. Ones that sort of take over the whole machine, and don’t let the test taker open other windows or fiddle with the OS. Especially since these things stood out in my memory as worst offenders.
I’m actually familiar with Blackboard but I never used their testing features. In general I have a love-hate relationship with that system. I find it rather awkward – it loves to do things in roundabout way, always making sure even the simplest action will require you to make several clicks and wait till the page reloads few times.
Last time I asked, most students said they preferred to use scantron rather than blackboard. So there is still the familiarity of the medium, the attention span thing and the convenience factor.
My other concern with blackboard is that (at least my university) it is open to the whole internet (meaning absent students could access it from home) and usually conducted on general purpose lab computers with unmonitored internet access (which means a student could potentially IM/Email questions to someone. Naturally these things can be controlled but it introduces new variables into the equation.
I’ve found Blackboard to be miles better than Microsofts alternative – Scholaris – which I was involved with as a pilot trial for the Education Department of South Australia a few years back. There is lots of surplus clicking, agreed. But at least it works on non-MS machines and browsers.
Blackboard is open to everywhere, which I think is a bonus in most cases. However, it is possible to limit access to a particular assessment task to a range of IP addresses. I don’t know how to do this, but can put you in contact with someone who does…it was done for an examination for one topic I took last semester. You can also put passwords on tests, and then give them out at the class/exam time.
I’ll also admit that I am not most students – what I like is probably quite different to what most students like…
my driver’s licence is 25 years in the past, there were no electronic tests then ;).
When I started in the consulting business I had to fulfill Tivoli certification requirements so that my company could keep its business partner status (they required at least x % employees with basic certifications and y % employees with advanced certifications, don’t remember the exact numbers).
All certification is “outsourced” to Prometric, and their electronic system is quite nice. I don’t know if it’s homebrew or a standard system they bought. You can skip questions without penalty, and at the finishing screen you can revisit every question. You can “mark” questions for revisiting, and you have a check mark for answered questions, so you can very simply revisit all yet unanswered questions. Imho the countdown timer is very convenient, and I didn’t feel pressed. Usually I don’t wear a wristwatch, so I noticed this as a positive thing ;). The result is presented after you press the “finish” button, and you immediately receive your “diploma”. This was running on very old Windows 3.1 PCs which were totally blocked. Of course you should not be able to cheat by calling Google for the answers ;)
We use electronic testing in law school for our finals. Basically you install a program on your laptop, register it, come test day you download the test through the program, it forces a reboot, and comes up with basically wordpad & disables nearly every keyboard shortcut (except cut/paste) and auxiliary device (e.g. no wifi). It also kills all unknown processes and prevents practically everything from starting up. After the test is over, the computer reboots again, uploads the results as soon as it gets a connection, and you’re done. There is also a manual upload function.
It won’t run in a VM (I tried), it is Windows only, and it doesn’t mind a dual boot. Both the test itself & answers are encrypted in some way. The system is pretty good assuming you’ve got cooperative test-takers (no one wants to risk their course grade over tinkering). I’ve gone into linux by mistake after the second reboot, and looked at the test & such, it seems pretty good.
Our exams are almost exclusively essays, but the system supports multiple choice & short answer too. Blue books are still an option if your computer won’t play nice, or you prefer it. Virtually no one uses them.
Given how slowly I handwrite (compared to typing) it is a no brainer for me. Plus you can edit on a computer, which is great. There is no immediate feedback (though I know what you’re talking about with that, and I hate it), just a clock telling you when you started, how long the test is, and how much time you’ve got left. One day some dlls were missing on my laptop and I had to take my contracts exam by hand. I got a decent grade, but I know editing would’ve made it much better – I always think of things to add partway through the exam.
I’d die if I had to handwrite my finals. Given some essays spill over 7,500 words in 3 hours (lots of writing), whereas others are limited to 200 words in 3 hours (lots of editing) – typing for most students is the only option.
I tend to prefer paper for exams… not sure why, just prefer putting ink to paper over clicking stuff onscreen for that one use (conversely I’ll open up a Notepad window instead of grabbing a post-it, so paper-love isn’t a general thing for me)
I know one part of our exams in Philosophy this summer is a long essay, which we plan for with 1000 words of written notes, but then type up under supervised conditions… not sure how locked down the computers are, but they’re college PCs which already have fairly tight controls on them, so blocking off the internet or any unwanted programs wouldn’t be hard.
Pingback: Arriverderci AP Italian? « an academic at work
you are a dumb loser children should not test on paper because take in consideration the great improvement it would make on the world’s climate and how it would help the human race YOU STUPID IDIOT!!!!!!>:(
@ Angie:
Angie, please remember you always capitalize the first letter of the sentence. Otherwise it looks like it is a sentence fragment – an unfinished thought if you will.
Secondly, punctuation is your friend. If you skip commas, you change the meaning.
Thirdly, why do you insist that “dumb loser children should not test on paper”? Do you mean only smart children should be allowed to partake in paper based tests? That doesn’t seem fair. Also, I don’t think as educators we are allowed to discriminate against kids based on their intelligence. So yeah, I don’t agree with this point. Or maybe you meant something else… I don’t know. Maybe if there was some sort of punctuation in your sentence I could figure out what you meant.
Also, I noticed you still have an AOL email. This must mean you are a time traveler. I haven’t seen an AOL email in at least a decade. When you go back in time to the 90’s please warn them to fix the housing market so we don’t have a recession now.
Finally, I personally believe that printing tests on paper has negligible effect on the climate compared to corporate emissions. I fully support being environmentally conscious but we should concentrate our efforts on not voting for science-phobic republicans, climate change deniers and urging our government to regulate and enforce Co2 emission control for corporations and subsidize research into alternate, clean energy sources. Then we can talk about harassing educators for using too much paper.
You forgot a capital letter and a comma but I’m sure you mean it
@ MEh:
Angie, you forgot a terminating period again. Yeah, I know it’s you because you’re using the same IP. Also, hotmail? Really? That’s even worse than AOL!
Also, I think you need to work on your tenses. “I’m sure you meant it.” is probably what you have wanted to say here.
Are you the one child they left behind because you should probably you should stop worrying about my grammatical errors and start worrying about you’re horrible life because face the facts your only accomplishment is probably this idiotic blog oh yeah and only losers have blogs you piece of toxic waste
And you most likely look like gary busey
YOure A MENTALLY DISABLED LOSER!!!! HAHAHHAHAHHAHHAHHAHAHHAHAHAHHAHH AHA>:(
@ Pata:
Dear Angie,
I could easily delete your comments or IP block you from this website, but your comments amuse me. One would think that after being mocked for your lack of punctuation, you would at least attempt to get it right in the subsequent comments, but no. No such luck.
Thank you for asking about my accomplishments. I happen to be an educator (I teach a few courses at a local university) and a software engineer. What do you do? Other than posting poorly spelled comments on technology blogs that is?
Anyways, how is Cox Communications treating you? Just had a nice chat with their abuse department. How is the weather in Phoenix?