John Turpin
5350 Assignment 11

Thoughts on Distance Educational System Evaluation

The evaluation of non-traditional learning systems is an issue that can create both healthy debate and considerable angst. Considering the increasing distance education migration that is being made around the world-large in both terms of finances and effort-it is critically important to ensure that that programs are effective, that systems are cost effective and that material is being adopted by learners. Undoubtedly, program evaluation must take place and it must be closely examined.

This evaluation process, however, introduces the proverbial two-edged sword. While results are insightful to system designers, the results might also be used to question or discredit the very effectiveness of that organizations' transition toward distance education. The stakes involved here can be quite high. While we can ignore the managerial and political impact of evaluation results for a moment, we should not lesson the importance of developing assessment tools that can be trusted by the organization.

This issue is further complicated by the credibility assigned to traditional evaluation measurements. Those authors who wrestle with the issue of distance education evaluation often seem to intimate that traditional education evaluation methods are universally effective and trustworthy. While traditional students can be reached more easily for assessment, this does not necessarily mean that the evaluation method chosen was inherently more trustworthy or that the measurement was effectively performed. Definitively measuring the success of any instructional system, either local or distance learning, is as much an art as a science. Let us not rush to dismiss a distance education evaluation simply because it was acquired remotely. The traditional evaluation results used to contrast non-traditional results could be equally questionable.

So then, how can we develop evaluations that can be trusted by stakeholders and used to improve instruction? That tends to be more difficult to answer than it might seem. The articles included for reviewed in this assignment testify to the void that exists in this area. It seems that evaluation techniques used for decades in the traditional classroom have been replaced with-well-anything and everything. We're not short on theories.

In Evaluation Is Not Necessarily the Solution, by Michael Theall, the author takes issue with an article by Hmielseki and Champagne that proposes certain evaluation methods. Theall states that the article authors are not familiar with the theory, literature, and reality of evaluation and student ratings. Additionally, Theall indicates that the authors considered only one aspect of the process and are "proposing an overly simple, unilateral, and potentially dangerous response to a complex problem." Presumably, Hmielseki and Champagne would disagree with this analysis and make a spirited defense of their evaluation methods. Jamie Morley also addresses this topic in In Methods of Assessing Learning in Distance Education Courses. Morely discusses the advantages and disadvantages of both synchronous and asynchronous methods in the assessment of distance learning. Both obviously have proponents and each would have many successes. Morley admits that both can be used effectively and stresses that the institution much choose the method that matches the organization the best. Makes sense.

Lastly, in On the Necessity of Grassroots Evaluation of Educational Technology: Recommendations for Higher Education, Ehrmann asks educators to reexamine educational goals and perform "grassroots" studies to improve outcomes. Ehrmann also calls for the public sector to invest in methods to prevent course designers and educators from "flying blind" when it comes to the evaluation of their course offerings. We all sympathize with the distance-learning instructor groping for feedback.

Each of these articles, to some degree, demonstrates a science in flux. All of the methods detailed in these articles are attempting to deal with a huge paradox-being that distance education is a form of "arms length" instruction that can best be evaluated from an intimate distance. Thus our dilemma.

The good news is that much thought and effort is currently going into this topic. This field is expanding and many projects are highly visible. Distance Education is under close scrutiny and researchers clearly understand the need for effective evaluation. This will result in better methods that can be implemented by program designers. The only questions that remain relate to the form of the methods, their effectiveness, and the timing of their development. They cannot come too soon.

In retrospect, is the non-traditional educator completely in the dark and flying blind? Are we making this more difficult that it really is? It might be simplistic, but the answer seems to lie in the relationship between the instructor and the learner. For centuries, teachers tracked the success of their instruction by examining the responses and behavior of their students. The closer their relationship; the better the feedback. Given the preponderance of collaboration and communication technologies that are out there, is a close relationship between instructor and student so difficult? While a great deal of research is forthcoming and many new assessment methods will be debated, a great deal of insight can be gained today by just picking up a telephone and talking to our students. While this might not produce the measurements needed by scholars, administrators and statisticians, it can provide the front-line instructor with a wealth of information and prevent the blind flying discussed by Ehrmann. Besides, we might just be amazed at what we learn.