Putting Prometheus' Feet to the Fire:
Student Evaluations of Prometheus in Relation to Their Attitudes Towards and
Experience With Computers,Computer Self-Efficacy and Preferred Learning Style

Daniel Arkkelin, Ph.D., Valparaiso University

Introduction

Most empirical research evaluating online course delivery systems has compared features of various systems or assessed IT staff or faculty evaluations. Little research exists on evaluations of the most important client in distance learning – the student. Even less research exists concerning important psychological processes (e.g., computer attitudes, preferred learning style) that may mediate or moderate students’ assessment of IT in online learning. Such empirical research is needed to evaluate the effectiveness of instruction using this relatively new technology.

This paper reports the results of an investigation of student evaluations of Prometheus in an online course (Research Methods in Psychology). In addition, we assessed students’ experience with computers and attitudes towards computers and technology, as well as computer self-efficacy and preferred learning style (visual, auditory, tactile modes), as well as the interrelationships among these variables. At the time the data was collected, Valparaiso University had just acquired  Prometheus, so an opportunity to assess first-time users of a courseware system presented itself. The author developed the Research Methods course as a hybrid course for the data collection period (i.e., most material and activities were online, with occasional in-class and work-team meetings). Students enrolled in this course were not expecting a significant online component, so this avoided the problem of participant self-selection in assessing reactions to online course systems.

Students in this course were tested early in the semester and again at the end of the semester to measure changes in evaluations over time and experience with Prometheus, as well as to assess changes in the psychological variables. Last, at the end of the semester these students were given a series of tasks to perform (e.g., find an Assignment or a posting on a Discussion Topic) in a completely new course created with Prometheus. These students’ responses to these tasks were compared to those of a second group of students from two different courses (Social Psychology and Statistical Methods) who had not previously taken an online course with Prometheus. This served as a control group to allow assessment of the effect of a great deal experience with the courseware (in the online Research Methods course) compared to performance and evaluations of students with minimal or no experience with Prometheus (in the traditional Social Psychology and Statistics courses).

Methods

Participants

There were 19 students in the online course (6 men, 13 women) and 25 students in the traditional courses (9 men, 16 women), for a total of 44 participants. The majority of participants were sophomore level and of traditional college age. The sample consisted almost entirely of upper-middle class Caucasians. Participants received extra credit for their participation in the study.

Materials

Table 1 summarizes the Prometheus Evaluation items. A 7-point Likert scale was placed below each criterion for students to indicate their evaluations.

Table 2 summarizes the psychological variables assessing cognitive, affective and behavioral components of attitudes towards computers and IT. With the exceptions of the Computer Self-Efficacy Scale and the Learning Style Scales, all other variables were measured using individual Likert-type scales.

The Computer Self-Efficacy scale was developed by Simon Cassidy and Peter Eachus, and can be found at the following URL: http://www.chssc.salford.ac.uk/healthSci/selfeff/selfeff.htm. It consists of 30 individual items, each measured using 7-point Likert Scales. Individual scores are summed to yield a total score ranging from 30 to 210. 

The Learning Styles Inventory was developed at Honolulu Community College, and can be found at the following URL: http://www.hcc.hawaii.edu/intranet/committees/FacDevCom/guidebk/teachtip/vark.htm. The scale consists of 24 items measured on 5-point Likert scales. The Visual, Auditory and Tactile Learning Style subscale scores are obtained by summing the ratings given to each of the 8 items relevant to each subscale, resulting in scores ranging from 8 to 40.

Participants in the online course were administered all instruments during the first  month of the semester. They were informed that their responses would be anonymous and would be used for research purposes only. It was emphasized that they were being asked to evaluate Prometheus as an online course delivery system, rather than to evaluate the course itself or the instructor.

Near the end of the semester the students in the online course were readministered the instruments and given the Prometheus tasks to complete. The students in the two traditional courses were administered the instruments at this time and were also given the Prometheus tasks to complete. It was originally planned to employ the Prometheus tracking feature to measure length of time to complete the tasks, but the tracking feature was too unreliable to yield meaningful data. Instead, students were instructed to keep track of how long it took them to complete the tasks and to report the time in minutes on their response sheets.

Results and Discussion

Prometheus Evaluations

Table 3 presents descriptive statistics of the evaluations of all participants in both classes. It can be seen that Prometheus faired pretty well, receiving median ratings above the midpoint of the Likert scales on six of the twelve items, and receiving no median ratings the midpoint of the scales. Thus, the overall evaluation of Prometheus by students ranged from fair to excellent on all features.

However, the fact that six of the twelve items were only rated fair suggests that the students were not overly-impressed with the communications features of Prometheus.

This does not necessarily mean that there are problems with these features of Prometheus per se. Indeed, it was the instructor’s own assessment that while students quickly and easily accessed the information/course content, it was quite difficult getting them to use the communications features in working together on small group projects. Students simply resisted posting and responding to Discussion topics, and almost never met in the team chat rooms.

This may have been due to the fact that students were all on campus anyway and preferred to meet in person. However, the course has been taught twice since then completely online and populated by students at different locations. The instructor has still experienced difficulties encouraging students to engage in either synchronous or asynchronrous communication using the Prometheus courseware. Small group communication in team projects is often difficult to promote in traditional classes as well, so the problem may not lie with Prometheus, but rather with students’ lack of experience with and understanding of effective online collaboration. Further, since none of the features of Prometheus received extremely low ratings, it is reasonable to conclude that students’ overall evaluation of Prometheus was favorable.

To investigate whether or not relative experience with Prometheus affected evaluations, independent-groups t-tests were conducted to compare the mean ratings of students in the online vs. the traditional courses. No significant differences were obtained on any item. This may suggest that first-time and experienced Prometheus users had essentially the same evaluation of Prometheus. However, the small sample size in the two groups may not have yielded enough statistical power to detect meaningful differences. In fact, comparisons between groups on the psychological variables listed in Table 2 also yielded no signficant differences. As is always the case with retaining the null hypothesis with a small sample, it is not possible to determine whether the failure to find a difference actually reflects a true lack of difference or is due to the lack of statistical power. Obviously, this question could be addressed by obtaining larger sample sizes in future research.

Significant differences were obtained between the two groups on self-reported understanding of Prometheus ( t = 3.38, p < .01, df = 39) and time to complete the Prometheus Tasks given at the end of the semester (t = 2.34 , p < .03, df = 39). Students in the online course reported a significantly higher level of understanding of the Prometheus courseware (M = 5.53) than did students in the traditional courses (M = 3.92). Online students also reported taking fewer minutes to complete the Prometheus tasks (M = 15.67) than did the traditional students (M = 22.44). These differences are not surprising, and reflect the differences in relative experience with Prometheus between the two groups. They also indicate that students in the online course were successful in learning to use and understanding the courseware.

Pre-post paired sample t-tests conducted on the two sets of responses of the online students (early vs. late in the semester) yielded no significant differences on any of the Prometheus evaluation items. Again, the small sample size and resulting lack of statistical power could be the reason for failure to obtain significant differences, or it may be that the evaluations remained consistent over time and experience with Prometheus.

Pre-post comparisons on the other psychological variables did yield a significant difference on the measure of computer self-efficacy ( t = 3.72, p < .01, df = 17), indicating that students reported significantly greater self-efficacy at the end of the semester (M = 123.22) than early in the semester (M = 88.78). If this is a reliable change, this would suggest that completing an online course using Prometheus has an important beneficial impact on computer self-efficacy. The importance of student self-efficacy will be seen in the next section.

Intercorrelations Among Variables

Pearson correlation coefficients were computed to assess the interrelationships among all the measured variables. Table 4 presents a summary of the significant correlations that were obtained from this analysis. Blank cells in this table indicate that the correlation among a given pair of variables was not significant (p > .05). The listing below the matrix provides the definition of the abbreviated variable names in the matrix.

It can be seen from this correlation matrix that the cognitive/behavioral variables of experience with computers, computer self-efficacy and knowledge of computers emerged as central among the pattern of interrelationships. Strong positive relationships between all three of these variables were obtained, and all three correlated strongly with several of the other affective/attitudinal variables.

Self-rated experience with computers was strongly and positively related to high self-efficacy and self-rated knowledge of computers. Strong positive correlations were also obtained between all three of these variables and positive affect towards both technology and computers, indicating that high levels of experience, self-efficacy and knowledge are associated with positive feelings towards computers and technology.

Strong negative correlations were also obtained between all three of these variables and negative affect towards computers and technology, indicating that low-levels of experience, self-efficacy, and knowledge are related to increased negative affect towards computers and technology. Thus, using computers, understanding them and having technological self-confidence seem to be closely and predictably related to both the positive and negative affective dimensions of attitudes towards computers and technology.

It remains for future research to determine the direction of causation (if a causal relationship exists) between these variables. That is, it may well be that the cognitive/behavioral factors influence positive and negative feelings towards technology. However, the reverse causation might exist: it may be that positive affect towards technology leads one to gain experience, knowledge and self-efficacy regarding technology, and negative affect causes one to avoid computers, thereby not learning about them or developing perceived control over technology. In the end, the causation is probably bi-directional, with cognitive/behavioral factors influencing affect and vice-versa.

Further, weak positive correlations were obtained between self-efficacy and favorable attitudes towards online courses as well as favorable attitudes towards Prometheus. Thus, self-efficacy may be a critical psychological determinant of attitudes towards educational technology. This suggests that interventions designed to increase self-efficacy could lead to a greater acceptance of information technology.

Not surprisingly, the affect towards computers and towards technology scales were strongly intercorrelated with one another. That is, positive affect towards technology and was related to favorable attitudes towards computers, and was negatively correlated with unfavorable attitudes towards computers. Thus liking computers goes hand-in-hand with liking for information technology, and fear of computers is related to negative feelings about information technology.

Here also, if a causal relationship among these variables exists, it would be interesting to attempt to determine the direction of the relationship. For example, it might be that providing pleasant experiences with computers could lead to increased liking for information technology in general. Further, increasing appreciation for information technology could lead to feeling better about the role of computers in the learning process.

Both positive and negative feelings for computers and technology were weakly related to attitudes towards Prometheus. That is, high levels of negative affect were associated with unfavorable Prometheus attitudes, and high levels of positive affect were related to favorable attitudes towards Prometheus. This is also not surprising, since positive/negative affect towards attitude objects are important components of overall attitudes towards the target. Again, it may be that experiences associated with positive affect with technology may contribute to favorable attitudes towards online course delivery systems. Indeed, a weak positive correlation was obtained between positive affect towards computers and attitudes towards online learning in general.

Few significant correlations were obtained between the three learning style modes and other variables. However, some interesting weak, but significant relationships were observed. A preference for both visual and auditory modes of learning was related to positive affect towards technology, and a weak positive correlation was obtained between preference for the visual mode and favorable attitudes towards online courses. This suggests that perhaps those who have a preference for visual learning (and to a lesser extent, for auditory learning) may facilitate appreciation for computers and technology in learning.

A weak positive correlation was obtained between preference for both the visual and auditory modes and preference for the tacticle mode of learning style. Beyond that, preference for tactile learning was significantly related to only one other variable, computer self-efficacy. A weak negative correlation was obtained, suggesting that preference for a tactile learning style is related to low computer self-efficacy. Thus, those who learn best by tactile means may be not possess self-confidence in their ability to use abstract information technology. Once again it should be cautioned that a larger more random sample with greater variability in learning styles should be investigated before firm conclusions can be drawn about these relationships or their implications.

Last, a strong positive correlation was obtained between attitudes towards Prometheus and attitudes towards online courses. This is also to be expected, in that favorable attitudes towards online courses goes hand-in-hand with appreciation for online course delivery systems. Thus, this relationship would probably also exist for other types of courseware (e.g., Blackboard).

Conclusions

One purpose of the present study was to gain empirical evidence of student reactions to online courses and courseware systems. The evaluations of Prometheus reported here suggest that students who were initially novices to this then-new system generally evaluated it as somewhere between fair and excellent. This suggests that students as a whole are generally open to using such course delivery systems in their education, and at least did not have strongly negative attitudes towards either Prometheus or online courses.

The lack of significant changes in evaluations of online students between early and late semester may be due to the small sample size, so the study should be replicated with a larger and more diverse group than the one employed here. However, it may simply be the case that the initial reactions to Prometheus remained stable over time and with further experiences with the course delivery system. One intriguing significant pre-post difference was obtained: students reported a significant increase in computer self-efficacy from the beginning to the end of the semester. If this is a reliable relationship, then participation in online courses may be one way of increasing self-confidence in technology, perhaps leading to subsequent increased interest in technology-based learning.

The comparisons between the experimental (online course) group and the control (traditional course) group revealed only two differences, indicating that the online group had a better understanding of and facility with Prometheus than did the traditional group – an expected effect of greater experience with the program by the former group. The lack of differences between the experimental and control groups in evaluations may again reflect the small sample sizes, but it may also support the conclusion above that students form attitudes towards courseware with relatively little experience, and that further experience with the system does not change the initial impression.

The correlational results suggest several interesting relationships that bear further exploration. In particular, the results suggest that the cognitive/behavioral variables of experience, self-efficacy and knowledge may be critical factors in determining students’ affective reactions to computers and technology as well as their overall attitudes towards technology. In turn, positive and negative affect in response to computers and technology also appear to be related to overall attitudes. Thus, efforts to give students positive emotional experiences with computers which teach them about computers and technology and which give them a sense of mastery may contribute to favorable attitudes towards and openness to online education.

Although it was expected that different learning styles might be related to attitudes towards computers and technology, not many relationships emerged. Again this may have been due to the small sample and lack of variability in learning style scores, but at least the current data suggest that a preference towards visual learning mode is the only learning style associated with favorable attitudes towards online courses, although preference for both visual and auditory learning style were associated with positive affect towards technology.

Clearly more research is needed regarding such individual difference variables in student responses to information technology. While there was fairly general consensus in student evaluations of Prometheus, the correlational results suggest that individual differences in cognitive, affective and behavioral determinants of reactions to technology may well mediate or moderate student attitudes towards online learning. Anecdotal evidence of the instructor suggests that the much-vaunted technological sophistication of the latest generation of college students is not universal.

The author has observed a shrinking, but still significant, minority of students who are adamantly opposed to information technology and online courses in particular. Similarly, each semester there are at least a few people who really struggle with learning and using the technology. Further, almost all students still seem to have difficulty in online communication and collaboration in small-group work. While these are just informal observations of the author, they underscore the need to continue gaining empirically-based answers to questions about student responses to information technology. And such investigations into psychological mediator and moderator variables (e.g., self-efficacy and positive/negative affect) might yield useful suggestions for the successful implementation of distance education. More importantly, such research could help to promote the goal of improved attitudes and learning outcomes for the student in distance education.

Acknowledgments

The author wishes to express appreciation to Kevin Festerling and Jeremy Cole for their assistance in material preparation, data collection and data analyses.