The attached file includes seven principles for reporting TE findings. Critical comments welcome.
Objectivity is an essential feature of evaluation. It is more complex in social inquiry than people unfamiliar with relevant literature may realize.
In this post I make the case for objectivity in TE, which is social inquiry, as a process that involves multiple objective observers who actively engage in rigorous dialogue concerning appropriate data collection, analysis, interpretation, and reporting throughout the inquiry process. The process itself is thoroughly documented and submitted to qualified persons for critique. Bias control and scope of cultural differences, subjected to competent independent verification, are the essential elements of objectivity in TE.
My case for this description is in the attached file. Constructive criticism is welcome.
I have argued that in general impact evaluation is not appropriate for community development programs. The primary purpose for impact evaluation is to judge if the evaluated program caused the documented outcomes. But collecting sufficient rigorous evidence to support that claim is complex and unreasonably expensive. I advocate using available evaluation resources for rigorous monitoring and other types of evaluation that provide valid and useful evaluation at reasonable cost.
LINK to Evaluating transformational development outcomes; includes discussion of difficulties in using impact evaluation
LINK to Creating a project monitoring system
Recently I have done more study on the nature of causation across disciplines. I believe now that there are ways we can examination causation in community development programs while practicing stewardship of meager resources for monitoring and evaluation.
Scriven’s (2008) description of General Elimination Methodology (GEM) includes the basic logic. There are two features:
- Causation is directly and reliably observable. A causal connection is a complex pattern of perceptions (observations) that have been registered in the brain.
- The benchmark for certainty regarding a causal connection in all disciplines is “beyond a reasonable doubt.”
Employing the GEM approach is a process of systematically ruling out plausible causes by:
- Creating a wide ranging list of plausible causes for the effect of interest. For TE this requires deep understanding of the nature of transformation and knowledge of settings in which transformation has occurred.
- Seeking rigorous relevant triangulated evidence for the validity of each cause on the list. Those causes that are judged invalid beyond a reasonable doubt based on rigorous evidence are eliminated. Criteria for “beyond a reasonable doubt” need to be agreed by the evaluation team and stakeholder representatives as a component of the evaluation design.
Note that experimentation is not logically required to establish a causal connection; random controlled trials (RCTs) are not required. Future posts will explore Scriven’s views on RCTs.
Scriven, Michael, A Summative Evaluation of RCT Methodology: & An Alternative Approach to Causal Research, Journal of MultiDisciplinary Evaluation, Volume 5, Number 9, ISSN 1556-8180, March 2008, pp. 11-24. Retrievable from http://www.jmde.com. (search for RCT)
The ten seed technique is a participatory monitoring and evaluation tool that documents a group’s perceptions about a wide range of topics and issues. The attached file describes and illustrates the technique. Click the link → Ten Seed Technique
Search the site using keyword “ten seed” to see examples of TE indicators based on TST.
Review the post on evaluating TD outcomes to keep in mind the essential characteristics of TE. Click the link → TE
Evaluation conclusions and recommendations often do not inspire imaginative thinking about the future. A provocative proposition describes what the organization would look like if it were designed to maximize the best practices, the peak experiences, in a sustainable way. I have prepared a hypothetical case study to illustrate how formulating provocative propositions can be included in an evaluation exercise. The process can encourage organizational transformation that can be used by God for social transformation. Click LINK Provocative Propositions
How good is this program, really?
An evaluation rubric “paints a picture of what the evidence should look like at different levels of performance” (Davidson, 2013, p. 24). It is a set of descriptions, ideally created by key stakeholders working together, of what particular conditions will look like when a program is performing excellently, satisfactorily, or poorly.
Click link for information and an example for transformation development → Evaluation Rubrics
The essence of program evaluation involves sound reasoning about the linkages between good evidence and good conclusions. Internalizing the material in these books will provide a basic evaluative mindset grounded in sound reasoning regarding transformational development. Then you can read any text on evaluation theory and methods to increase your knowledge about evaluation within a Christian worldview. Or you can review evaluation texts you have read from a more holistic perspective.
Meditate on Luke 18:31-34
Jesus steadfastly set his face toward Jerusalem, for the journey to Jerusalem was God’s purpose for his life. “The great thing to remember is that we go to Jerusalem to fulfill God’s purpose, not our own… in the Christian life we have no aim of our own (Chambers).”
Transformational development program stakeholders have a general purpose: to enable individual and social transformation by achieving the program goals. A key objective of transformative evaluation is to assist program stakeholders to understand at a deeper level what really matters in development. As an evaluator I will examine evidence related to accomplishment of program goals. But as a transformative evaluator I will work with stakeholders to facilitate humble exploration of their expectations in the light of God’s word.
A conclusion is the answer to a specific question included in the evaluation design, or an interpretation of outcomes related to a particular information need.
In general discernment is the ability of an evaluator to see what is not obvious to the typical stakeholder. For transformative evaluation discernment goes beyond this to see what is right and wrong, good and evil, in the program from God’s perspective.
For a more detailed discussion go to page 19 in Evaluating Transformational Development Outcomes on this site. I encourage you to comment, perhaps starting a dialogue that will help me and others deepen understanding of this topic/issue.
Revised June 23, 2014
I define a recommendation as a statement offered as worthy of acceptance or approval by stakeholders. Based on available evidence, knowledge and experience the evaluator is saying that it is reasonable for stakeholders to adopt the action included in the statement.
In transformative evaluation there are two types of recommendations. The first type is what you expect to see in any program evaluation report: description (based on evidence) of changes to implement to improve the chances that program goals and objectives will be achieved efficiently and effectively.
The second type is description of changes to increase the likelihood that the program will enable individual or social transformation regardless of the program goals and objectives.
In-depth prayerful reflection will enhance both types of recommendations.