The attached file describes nine obstacles to critical thinking; failure to overcome them can lead to flawed reasoning. Train yourself to spot these obstacles as you do evaluation work, and train yourself in overcoming them. Your work will have a better chance of leading to changes in community development planning and implementation that make the world a better place.
Robert J. Gula, Nonsense: Red Herrings, Straw Men and Sacred Cows: How We Abuse Logic in Our Everyday Language, Axios Press, 2007.
The author intended to write a “timeless handbook of verbal logic.” Using everyday language and examples, he succeeded. Spending a few minutes twice a week reading and thinking about a specific logical error has improved my writing and my ability to identify nonsense quickly in news stories. I recommend this inexpensive but valuable paperback.
I have argued that in general impact evaluation is not appropriate for community development programs. The primary purpose for impact evaluation is to judge if the evaluated program caused the documented outcomes. But collecting sufficient rigorous evidence to support that claim is complex and unreasonably expensive. I advocate using available evaluation resources for rigorous monitoring and other types of evaluation that provide valid and useful evaluation at reasonable cost.
LINK to Evaluating transformational development outcomes; includes discussion of difficulties in using impact evaluation
LINK to Creating a project monitoring system
Recently I have done more study on the nature of causation across disciplines. I believe now that there are ways we can examination causation in community development programs while practicing stewardship of meager resources for monitoring and evaluation.
Scriven’s (2008) description of General Elimination Methodology (GEM) includes the basic logic. There are two features:
- Causation is directly and reliably observable. A causal connection is a complex pattern of perceptions (observations) that have been registered in the brain.
- The benchmark for certainty regarding a causal connection in all disciplines is “beyond a reasonable doubt.”
Employing the GEM approach is a process of systematically ruling out plausible causes by:
- Creating a wide ranging list of plausible causes for the effect of interest. For TE this requires deep understanding of the nature of transformation and knowledge of settings in which transformation has occurred.
- Seeking rigorous relevant triangulated evidence for the validity of each cause on the list. Those causes that are judged invalid beyond a reasonable doubt based on rigorous evidence are eliminated. Criteria for “beyond a reasonable doubt” need to be agreed by the evaluation team and stakeholder representatives as a component of the evaluation design.
Note that experimentation is not logically required to establish a causal connection; random controlled trials (RCTs) are not required. Future posts will explore Scriven’s views on RCTs.
Scriven, Michael, A Summative Evaluation of RCT Methodology: & An Alternative Approach to Causal Research, Journal of MultiDisciplinary Evaluation, Volume 5, Number 9, ISSN 1556-8180, March 2008, pp. 11-24. Retrievable from http://www.jmde.com. (search for RCT)
The attached paper describes the concept of working logic. This is the logic evaluators use to support conclusions. Applying the concepts in the paper will strengthen your evaluation reporting. I illustrate the various elements of working logic by inviting you to consider some evaluation questions for transformational development. I welcome additional illustrations and comments.
Click this link → Working Logic
The essence of program evaluation involves sound reasoning about the linkages between good evidence and good conclusions. Internalizing the material in these books will provide a basic evaluative mindset grounded in sound reasoning regarding transformational development. Then you can read any text on evaluation theory and methods to increase your knowledge about evaluation within a Christian worldview. Or you can review evaluation texts you have read from a more holistic perspective.