Lesson 11: Some Thoughts on the Evaluation of Technology's Effectiveness

Katherine L. Squires
Comm. 5350, Dr. Ingle

There is an excellent resource at http://www.ala.org/acrl/guides/distlrng.html that was developed by the Association of College and Research Libraries and provide Guidelines for Distance Learning Library Services. I thought this important in that libraries at institutions of higher education play an important role in the distance learning process. They have established "guidelines that reflect a greater emphasis on outcomes assessment and [that were] approved by the ACRL Board of Directors and the ALA Standards Committee in fall 2000."

Another excellent resource is located at http://www.chea.org/Commentary/distance-learning-2.cfm. This Site, which is a part of the Council for Higher Education Accreditation, has some important guidelines for assessment criteria and tools in distance learning. "The Institute's report, What's the Difference? A Review of Contemporary Research on the Effectiveness of Distance Learning in Higher Education, reviewed the research on distance learning since 1990. In addition to noting that much of the original research was of questionable quality-which rendered many of the findings inconclusive-The Institute identified several gaps in the research that need to be addressed. The report stated that the research:

• has tended to emphasize student outcomes in individual courses rather than for total academic programs;
  
• does not take into account differences among students;
  
• does not adequately explain why the drop-out rates of distance learners are higher;
  
• does not take into consideration how the different learning styles of students relate to the use of particular technologies;
  
• focuses mostly on the impact of individual technologies rather than on the interaction of multiple technologies;
  
• does not include a theoretical or conceptual framework; and
  
• does not adequately address the effectiveness of digital libraries.

The report concluded that while technology is having a profound impact on colleges and universities across the nation and around the globe, the higher education community still has much to learn regarding how and in what ways technology can enhance the teaching/learning process. Nevertheless, technology has helped to renew the academy's focus on pedagogy and, as a result, either implicitly or explicitly, the fundamental question of how best to teach students is being asked." 1

Another important Site is located at http://www.users.csbsju.edu/~tcreed/adept/ and provides access to Project ADEPT (Assessment of Distance Education Pedagogy and Technology). This was conducted at the University of Wisconsin and states that "in looking at students' level of satisfaction with distance education, we want to assess what factors contribute to that satisfaction; the extent to which delivery format affects whether they will be more or less likely continue to pursue higher education and be life-long learners; and how they will likely go about learning in the future.

Some of the characteristics of a distance format that we are interested in looking at are:

• What balance of synchronous/asynchronous content delivery is best for what kind of learners?
  
• What components of a distance format most enhance the experience for both students and professors?
  
• What components of a distance format most detract from the experience for both students and professors?
  
• What of value to the students did they miss most by a distance format?
  
• What of value to the students did they gain most by a distance format?
  
• How can the benefits of cooperative learning best be promoted at a distance?
  
• Is there some role/value to face-to-face promotive interaction in distance education?" 2

Finally, (although I could continue - there are so many excellent resources) a report on "Current Research on Web Based Distance Learning: Web Resources on Distance Learning http://www.bgsu.edu/departments/tcom/webnet98.html presented by Dr. Bruce C. Klopfenstein (and others) at WebNet 98 Orlando, Florida provides some valuable insights on many methodologies and approaches to assessing the effectiveness of distance learning.

1 CHEA, Update No. 2 http://www.chea.org/Commentary/distance-learning
2 The report is very detailed and is very useful in evaluating successful assessment strategies.

LESSON 11 - ADDITIONAL ASSIGNMENT
THOUGHTS ON SUPPLEMENTAL READING MATERIALS
Katherine L. Squires

Comm. 5350, Dr. Ingle

I found the Assessment Article, dated November/December 20001 and the Letter to the Editors to be of particular interest. I, personally, am very interested in the use of technology in higher and continuing education programs as a delivery tool and resource. I am concerned that we need to mentor traditional, live faculty about how to best integrate and use technology in instructional and course development. I am in full agreement with the statement, "using the Web to support a distance learning program ought to be accompanied by a reexamination of educational goals, a fresh look at students, and a reexamination of support for staff and students."2

We cannot just use technology for technology's sake or so inundante online leaners with all the technology options that we lose sight of the educational goals and instructional purpose. We have many educational technology options available to us, and many more are rapidly emerging. A successful course is not one that has a Webcast, a synchronous chat, an asynchronous chat, online testing, streamed media, interactive questions/quizzes, etc. but one that uses technology tools wisely to enhance and supplement the learning process.

How do we evaluate the effectiveness of these tools? How do we choose which tools are appropriate for certain instructional environments? I, believe, it is this reexamination that is the focal point of this article. Faculty mentoring and workshops are critical if we are to prevent or lessen the "flying blind" problem. How do we evaluate if we cannot assess and how do we assess if we do not seek input from learners. I have had the privilege to teach online courses and be a student in online courses. I have also had the privilege to teach in graduate programs for over 15 years. There is a vast difference between teaching in a live, F2F environment and teaching online. My student status in online programs has made me sensitive to the educational technology methodologies I adopt in my teaching of online courses.

Ehrmann suggests that we "develop specialized investigative tools that educators need to study and [use] to improve their own instructional programs."3 I agree. I work with faculty in helping them understand how to best transition a live course into an online course and how to use what educational technologies are best for a particular course. Ehrmann suggests interviews and surveys to gather the information necessary to develop the investigative tools. I agree. There needs to be a facilitator who can assist in this development process, so it is not the "blind leading the blind."

1 Ehrmann, Stephen, On the Necessity of Grassroots Evaluation of Educational Technology: Recommendations for Higher Education, Assessment, November/December 2000.
2 Ibid.
3 Ibid.