The value of including stakeholders in the evaluation team has various dimensions.
- It can increase the usefulness of evaluations if their views and expertise are considered and integrated whenever appropriate. This requires a skilled evaluation facilitator and stakeholder commitment to substantial participation, particularly in analysis and interpretation activities.
- Participatory evaluation methods can be used to create consensus and ownership in relation to the development activities.
- Dialogue with stakeholders can help improve understanding and responsiveness to their needs and priorities.
In evaluation work “triangulation” is a fancy word that stands for using multiple methods to collect data, data sources, perspectives and evaluators to develop a more in-depth understanding of whatever is being studied or evaluated. Independent corroboration of a result strengthens its utility for decision making as well as extending our knowledge.
See post on triangulation … Introduction to triangulation
The triangulation dimension is not given the same degree of attention in the participatory community development evaluation literature. Participation by stakeholders can be a critical way of revealing and dealing with bias, and uncovering complexity in how the evaluated program is affecting participants and others.
Triangulation is not evaluation magic. Two common assumptions about the value of triangulation need to be examined closely.
- Does it eliminate bias?
The first assumption is that bias will be eliminated in a multimethod design. Although different methods can yield different understandings of the object of investigation it is difficult to conclude that those different understandings somehow neutralize any biases present. Each may not compensate for the limitations.
- Does it reveal true propositions?
The second common assumption is that use of triangulation will lead to convergence upon true propositions. Conflicting findings is a typical outcome of using different methods for collecting information especially if there is both quantitative and qualitative information. The evaluator must be prepared to wrestle with ambiguity creatively and to encourage others to do so. Exploration of possible explanations for differences in findings may lead to valuable conclusions that otherwise would not be included. Patton (Qualitative Evaluation Methods, 1980, pp. 329-332) recommends triangulation during analysis of the information, where different teams of evaluators or different members of the same evaluation team use different analysis approaches. Exploration of differences in conclusions may lead to additional insights about the object of evaluation.
Triangulation is not magic, but it can lead to better informed conclusions and evaluation advice.
See post on evaluation advice… Evaluation Advice
Blending principles of constructivist inquiry with principles of holistic thinking creates a spiritual lens for collecting and analyzing information about spirituality. This link describes the lens and makes a case for using discussion protocols instead of focus group questions.
Interview or Discussion
An indicator elaboration worksheet has three main functions.
- Filling out the sheet helps planners to define properly the indicator and relate it to relevant development hypotheses or the theory of change underpinning the project.
- The precise description reveals the specific information that is required and the means for collecting and processing it. This allows the feasibility of using the indicator to be determined in terms of time, money and expertise required.
- Other people can understand the indicator and use it in other settings and at other times in ways that allow the readings to be comparable.
The attached file is such a worksheet. Comments and suggestions welcome.
LINK → Indicator Elaboration
The purpose of a focus group in an evaluation is to determine perceptions, feelings and ways of thinking about various aspects of the evaluand. Unfortunately this important method for collecting data appears to be relatively easy to use. As a consequence the required rigor for preparing the data collectors, selecting participants and implementing the group discussion may not be employed. This will reduce substantially the value of data collected.
The attached file is a set of guidelines for using this important method more rigorously. Comments and suggestions welcome.
LINK → Focus Group Guidelines
© World Vision International 2006, All Rights Reserved. No part of this document or any of its contents may be reproduced, copied, modified or adapted, without the prior written consent of the author.
The heart of transformational development strategy is in the way development workers interact with community people. Reflection on the story of Zacchaeus, Luke 19:1-10, suggests four indicators of transformational development.
Time spent looking for people in trees
One very important indicator of transformational development is the amount of time that facilitators spend looking for people in the trees that are watching them. Such people want to be involved, but for some reason they think they are too small, or too big, or too shy, or too old, etc. They look on from the sidelines. In our frantic efforts to do impossible things with unrealistic deadlines, how much time do we spend looking for the people in the trees?
Number of people called by name
Another indicator is the number of people that a development worker knows by name, and has spent time in their home. Not to convert them, but to invite them by name to engage developmentally in their community.
Amount of sharing with the poor
Another indicator is the amount of sharing with the poor that is taking place in the community. Transformational development is all about deepening love for God and neighbor, which has an outcome of learning what the poor need and being instrumental in seeing that it is provided for them.
Another is the type of repentance that is visible as people come to know that in their condition, whatever it is, they are loved by God.
The attached paper describes the concept of working logic. This is the logic evaluators use to support conclusions. Applying the concepts in the paper will strengthen your evaluation reporting. I illustrate the various elements of working logic by inviting you to consider some evaluation questions for transformational development. I welcome additional illustrations and comments.
Click this link → Working Logic
Reasoning involves thinking through how pieces of information logically support an idea. More formally, reasoning is the process of forming conclusions, judgments, or inferences from facts or premises.
The quality of an evaluation is directly related to the quality of the evaluator’s reasoning skills. This brief post describes five areas where many of us could improve our reasoning skills. It is based on Rog (1995); read the article for more information. Continue reading
Professional evaluation literature includes much wisdom for the Christian evaluator. I have developed a secular framework for comparing different evaluation approaches described in the literature. Perhaps it will be useful to you as you broaden your knowledge of evaluation approaches. Click this link → Evaluation Model as a Concept
I have identified several approaches that I think are most helpful for a transformative approach to evaluation: responsive evaluation, deliberative democratic evaluation, constructivist evaluation, and utilization focused evaluation. Continue reading
A transformative evaluator must understand the nature of impact evaluation very well. The fundamentals of impact evaluation should be applied to transformational development programs when assessment of evaluation feasibility justifies the cost.
The transformative evaluator must also understand technical and ethical limitations of impact evaluation applied to TD programs. Go to page 8 in Evaluating Transformational Development Outcomes for some thoughts about this.
Future posts will examine impact evaluation in more detail along with possible alternative approaches that may be more illuminating. Your comments about points to explore are welcome.
The evaluation wheel is a graphic device for comparing evaluands on specific criteria. (Remember that an evaluand is a fancy word for whatever is being evaluated.) It is a versatile tool that can be used in a variety of ways:
- Individuals can use it to indicate strengths and weaknesses of their performance on some task over a period of time.
- As a group exercise it can illuminate differences in perceived strengths and weaknesses of something by the group members. A composite wheel is constructed from wheels completed by individuals.
- As a multi-group exercise it enables comparison of perceived strengths and weaknesses of something as perceived by the different groups.