Trials and Triumphs of Testing Online:
Use of Online Tests in Relation to Student Attitudes
Towards and Experience With Computer

Daniel Arkkelin*

Valparaiso University

Poster Presented at Syllabus2001, Santa Clara, CA

July, 2001

*Psychology Department, Valparaiso University, Valparaiso, IN 46383

e-mail:  Daniel Arkkelin

Introduction

This paper discusses the author’s experiences with two online testing programs (Thomson Learning’s“World Class Testing Tools” and Pearson Education Group’s“Test Manager”) in three different courses over three semesters. The paper also presents the results of an empirical investigation relating students’ attitudes towards and experience with computers to their attitudes towards online testing before and after experiencing online testing.

Both testing programs were obtained at no cost from textbook publishers upon adoption of a relevant text. Both programs could be installed to the campus network for creating, administering and grading tests. Test Manager (TM) can only be used on a local network. World Class Testing Tools (WCTT) supposedly could also be used via the WCTT website, but the author learned quickly that this was not a viable option due to a serious bandwidth problem. Thus, an obvious limitation of these programs is that they require students to be on campus/connected to the intranet.

The Instructor's P.O.V.

Both programs have an intuitive interface and good security features. Creating and publishing tests to the intranet was easy if the instructor used the texts’ computer test banks. However, neither program allowed for simple importing of original tests or test items from a text file (these had to be “cut-and-pasted” line by line into the program). Saving and accessing test files was not intuitive for either program. Inexperienced faculty would need considerable help in understanding how and where to retrieve test files.

Both programs scored and stored test results into a grade book. There are serious problems with both of these grade books. Neither program allows the instructor to edit online scores nor to add “extra credit” points to the students’ grades. WCTT grade book files frequently became corrupted and inaccessible, and TM files could not be saved to a PC hard drive or floppy disk. The author ended up manually entering test scores into an Excel file, since neither electronic grade book could be trusted.

Both programs have pedagogical features that allow the instructor to reference appropriate text page numbers for questions, as well as for adding explanations/comments about individual response choices to questions. Students’ test scores are displayed immediately after completing the exam, and students can review their choices, the correct answers to questions and the accompanying explanations/comments if the instructor selects this option.

Both programs permit the instructor to generate “reports” of the class test results, including a variety of descriptive statistics and individual item analyses. These features can be useful in assessing the psychometric properties of tests as well as for test development and validation. Using these features can be very time-consuming, however, and generating appropriate report print outs is confusing and frustrating. Generating individual student reports for an exam is not possible with TM. This can be done with WCTT, but not if the instructor selects the option to present test items in random order for each student.

The Student's P.O.V.

Most students found navigating the test programs fairly intuitive (apart from the typical login/password problems). The most common problem encountered involved accidentally exiting the test before finishing. A less frequent, but more serious problem, was that both programs occasionally “froze” without sending the results to the gradebook. This required the instructor to republish the exam and the student to retake it.

Most students liked the immediate feedback of their test results and the ability to review the individual questions, their response choices and the correct answers/explanations. Some complained that not having a hard copy of the test interfered with their performance because they were used to writing notes on the exam. In the first two semesters, the instructor scheduled exams in a computer lab, resulting in some students complaining of distractions. In the third semester, the instructor allowed students to take the exam at any time during a one-week period. Virtually all students liked this option, though there are obvious security risks with this approach.

Student Attitudes & Online Testing

Students in the target courses completed a questionnaire at the beginning of the semester (prior to the first online test) and again at the end of the semester (after having completed at least three online tests). This instrument assessed students’ usage of computers, general attitudes towards computers, self-perceived knowledge of computers, fear of computers, and attitudes toward online exams.

Pearson correlations were computed to assess the interrelationships among these variables (see Table 1). These analyses indicate a statistically significant positive correlation between high self-rated knowledge of computers and positive attitudes towards computers and online exams, as well as a significant negative correlation between high knowledge and low fear of computers. Also, a significant positive correlation was obtained between frequent daily use of a computer and positive attitudes towards computers and online tests. A significant positive correlation was obtained between favorable attitudes towards computers and favorable attitudes towards online tests, as well as a significant negative correlation between favorable computer attitudes and low fear of computers. Interestingly, fear was unrelated to daily use, knowledge or attitudes towards online exams.

Table 1. Pearson Correlations Among Variables

*Significant Correlation (p < .01); ^nsNonsignificant Correlation (p > .05)

Paired-sample t-tests were conducted to compare mean scores on the measured variables before and after experiencing online testing (see Table 2). These analyses revealed that students had significantly more favorable attitudes towards online exams after (M = 4.65) than before experiencing online exams (M = 4.09). The other differences were only marginally significant, but the comparisons are in the direction of greater knowledge, less fear and more favorable attitudes towards computers after experiencing online testing than before.

Table 2.  Mean Pre-Post Scores and t-tests

^ns: Nonsignificant Difference (p > .05); *Significant Difference (p < .05).

Discussion

The moral of the story from the instructor’s perspective is that you get what you pay for. Both of these programs were “freebies,” and the author began using them two years ago. Course delivery software such as Blackboard and WebCT may have overcome some of the technical problems encountered with TM and WCTT in their testing/gradebook software (indeed, Thomson Learning has recently discontinued their web-based testing software, referring previous WCTT users to “WebTutor Toolbox on WebCT 3.1”).

The author has not yet used any of these new course delivery systems, but other potential users of these programs might bear the issues discussed here in mind as they consider the choice of an online testing program. Of course, any program should be tested before implementation in a course. The author recommends giving students a “practice exam” to familiarize themselves with the program before taking the first exam. Both WCTT and TM have practice quiz options, where the scores on the quiz are not entered in the course grade book. It is still recommended to try out the exam option so that the instructor can familiarize him/herself with the grade book software.

Apart from the problems, both software programs generally worked well as viable substitutes for traditional testing methods. Further, they offer many psychometric and pedagogical  features that are difficult, if not impossible, to implement with traditional paper-and-pencil testing. Of course, completely online courses necessitate the use of online testing. Last, but not least, online testing is much more environmentally-friendly and less expensive than paper testing.

From the students’ perspective, most students adapted quite well to online testing, and only a few phobic students consistently had problems. Many were anxious at first, but most came to like online testing. Most encouraging to the author is that despite some problems, there was a statistically significant increase in favorableness of attitudes towards online testing at the end of the semester, as well as trends toward greater knowledge, less fear and more favorable attitudes towards computers. The author believes that it was a good learning experience for students, and it may  encourage them to further explore educational technology.

Future research should collect empirical information on actual learning outcomes. For example, investigations might examine the impact on exam performance of student usage of practice quizzes and time spent following up on exam review feedback (e.g., re-reading cited text page numbers).

More data is needed to elaborate the causal connections relevant to the correlational data presented here. While the present data do not permit such causal inferences, the patterns are intriguing. Self-perceived knowledge of computers emerged as a central correlate of all other variables except daily use. Both daily use and knowledge were  positively related to both computer attitudes and online attitudes, so perhaps these are independent causes of such attitudes (e.g., use is a behavioral component, while knowledge is a cognitive component of attitudes).

Regarding the affective component of attitudes, fear was negatively correlated with general computer attitudes, but unrelated to attitudes towards online exams. This suggests that fear may be an indirect cause of the more specific attitude towards online tests, exerting its effect through general computer attitudes. Since fear was also negatively correlated with self-rated knowledge of computers, it may also indirectly affect online test attitudes via the path of self-perceived knowledge. Causal modeling techniques would be necessary to test such hypotheses. The results of this study provide some clues as to the cognitive, affective and behavioral processes underlying student online test attitudes, and may help in understanding student responses to online testing.