The purpose of this post is to introduce the concept of research design and evaluation design to people with little or no research training or experience. There are many types of evaluation designs, but two types are key to planning impact evaluation: experimental designs and quasi-experimental designs. The characteristics of each type are described.
LINK… Research design intro
Click on this link for an outline for an evaluation design document…http://evalfrank.com/2017/10/outline-evaluation-design-document/
I have argued that in general impact evaluation is not appropriate for community development programs. The primary purpose for impact evaluation is to judge if the evaluated program caused the documented outcomes. But collecting sufficient rigorous evidence to support that claim is complex and unreasonably expensive. I advocate using available evaluation resources for rigorous monitoring and other types of evaluation that provide valid and useful evaluation at reasonable cost.
LINK to Evaluating transformational development outcomes; includes discussion of difficulties in using impact evaluation
LINK to Creating a project monitoring system
Recently I have done more study on the nature of causation across disciplines. I believe now that there are ways we can examination causation in community development programs while practicing stewardship of meager resources for monitoring and evaluation.
Scriven’s (2008) description of General Elimination Methodology (GEM) includes the basic logic. There are two features:
- Causation is directly and reliably observable. A causal connection is a complex pattern of perceptions (observations) that have been registered in the brain.
- The benchmark for certainty regarding a causal connection in all disciplines is “beyond a reasonable doubt.”
Employing the GEM approach is a process of systematically ruling out plausible causes by:
- Creating a wide ranging list of plausible causes for the effect of interest. For TE this requires deep understanding of the nature of transformation and knowledge of settings in which transformation has occurred.
- Seeking rigorous relevant triangulated evidence for the validity of each cause on the list. Those causes that are judged invalid beyond a reasonable doubt based on rigorous evidence are eliminated. Criteria for “beyond a reasonable doubt” need to be agreed by the evaluation team and stakeholder representatives as a component of the evaluation design.
Note that experimentation is not logically required to establish a causal connection; random controlled trials (RCTs) are not required. Future posts will explore Scriven’s views on RCTs.
Scriven, Michael, A Summative Evaluation of RCT Methodology: & An Alternative Approach to Causal Research, Journal of MultiDisciplinary Evaluation, Volume 5, Number 9, ISSN 1556-8180, March 2008, pp. 11-24. Retrievable from http://www.jmde.com. (search for RCT)