Assessment of the impact of CIELO-G
An independent external evaluation of the project will be provided by Kirk Knestis Ph.D., Principal of Washington, DC-area research firm Evaluand LLC. Dr. Knestis has been a professional education researcher for 20 years and has led evaluation contracts or research partnerships of approximately 35 NSF projects across seven different NSF programs. The evaluation will be designed to provide both formative and summative reporting to the project team, independent of the team managing the project but applying a collaborative evaluation approach to optimize support for the project. |
Project implementation and evaluation design are guided by a logic model (Figure above), developed in a collaboration between project designers and the external evaluator. This conceptual model is a representation of “how the model works”—its theory of change—in terms of how the CIELO-G model is expected to realize specific outcomes for the five stakeholder groups targeted by the effort. The logic model guides design and implementation of data collection and analysis to test the theoretical model embedded in the program design (McLaughlin & Jordan, 1999), including the processes by which results are realized through the five conditions of the Collective Impact framework.
Analysis will apply a concurrent triangulation mixed design (Creswell et al., 2003) to answer a series of five questions guiding the evaluation:
- With what quality and in what quantities are grant-funded CIELO-G activities implemented?
- To what extent do faculty and researcher stakeholders realize outcomes anticipated for them in their roles supporting the project?
- To what extent do educators and graduate/postdoc students realize the specific geoscience-related awareness, understanding, and skill outcomes intended for the project?
- How do outcomes for the above stakeholder groups translate into geoscience-related engagement, awareness, and understanding outcomes for local NGO staff and the general public in communities where the model is implemented?
- What progress are proposed project activities making progress in terms of the NSF merit criteria of intellectual merit and broader impacts?