Monthly Archives: January 2020

Program Evaluation Policy and Procedures

An agency should have a policy or framework that applies to all program evaluation work it does. This is a set of broad statements about the minimum requirements for each evaluation, regardless of diversity in the evaluands and evaluation objectives.

A framework is a set of high-level standards. The framework in this post is based on a participatory approach to evaluation. There are eight elements in the framework.

Planning an evaluation

A Feasibility Study should be done to determine if the anticipated benefits of doing an evaluation will justify the estimated costs. See posts on Feasibility Study for details.

Assuming that an evaluation feasibility study supports doing a program evaluation, planning begins by preparing an evaluation design that includes the purpose of the evaluation, evaluation objectives, stakeholder groups, primary needs for information, methodology, reporting to different audiences, evaluators, budget, and a time line for each stage of the evaluation.

When the design document is consistent with this framework, generally the evaluation design is the document approved by the appropriate representatives of management, partners, and program participants. The evaluators are accountable for using the design to plan and complete evaluation activities that achieve the evaluation purpose and objectives.

A detailed work plan evolves as activities are scheduled and completed to achieve the evaluation objectives. Each evaluation plan will be reviewed against this framework. Exceptions to elements in this framework will be explained in the evaluation plan.

1. Values – Characteristics of an evaluation that are valued most.
  1. In addition to typical methods for collecting and analyzing information to achieve evaluation objectives, the evolving plan shall include designated time for reflection and discernment.
  2. Participatory methods shall be used throughout all aspects of the evaluation exercise. These methods include involving stakeholders in developing questions and engaging in analyzing and interpreting collected data.
  3. The goodness of a program shall be defined by notions of goodness described by different groups of stakeholders as described in the evaluation design. Evidence will be collected for each notion included in the approved evaluation design.
2. Utilization of findings.
  1. The evaluation design will describe the primary audience for using the evaluation findings, and how that audience intends to use them. Other audiences can use at least some of the findings but they may have limitations. The evaluation report will describe limitations in using findings for other purposes.
  2. The evaluation design will also describe other audiences and the probable means of reporting to them.
3. Theory of Social Change (ToC) within the surrounding context.
  1. Each evaluation will examine the appropriateness of the implicit and explicit theory of change undergirding the program design.
  2. Each evaluation will document the process followed to develop the program design. The evaluator will comment on the process regarding the role of ToC in the process.
  3. Each evaluation will document achievements and how the interactions between project persons, partners and participants reflect Christian values. If achievement or interactions are unsatisfactory the evaluation report will include recommendations regarding investigating theories of change that may guide future programing that will have better results.
4. Knowledge of assets in the context that strengthen program results.
  1. The evaluation will examine the assessments that guided the program design to determine if assets were considered; if so, the evaluation will document how the design included assets to strengthen the program.
  2. The evaluation will examine how the program monitored assets in the context and how management responded to opportunities to use them.
5. Knowledge of obstacles in the context that could reduce program effectiveness or efficiency.
  1. The evaluation will examine the identification of assumptions that were considered which if valid would have major negative consequences and how that affected the design of the program.
  2. The context for each program result will be examined to identify obstacles to achieving maximum results.
6. Assumptions about evaluation approach.
  1. When program objectives can be achieved by applying knowledge based on cause-effect relationships, the evaluation shall use appropriate methods to document and analyze significance of achievements. This is outcome or impact evaluation.
  2. Generally cause-effect methods are not appropriate for documenting change in spiritual dimensions of reality. To evaluate such change an evaluation will include rigorous documentation of information collected through spiritual practices and qualitative methods of inquiry.
  3. If both types of evaluation are desired for a program, key stakeholders should agree on whether there will be two separate evaluations or one mixed methods evaluation. If the decision is to do one mixed methods evaluation, then the evaluation team needs to include an experienced cause-effect evaluator and an experienced spiritual-qualitative evaluator who respect each other’s expertise.
7. Program implementation monitoring.

Various aspects of program implementation should be monitored at least quarterly. An evaluation design should include an objective to examine monitoring results for the period covered by the evaluation. Typical topics to analyze: adequacy of the indicators used, validity and reliability of indicator results, use of monitoring information by management; accuracy of reporting, etc.

8. Framework revision.

Every five years this framework will be reviewed by the agency and partners to determine its relevance and usefulness. A participatory process will be used to determine modifications.

Getting Program Evaluation Right

This post is an outline of the article: “Getting evaluation right: a five point plan,” by Dr. Jyotsna Puri, Deputy Executive Director and Head of Evaluation of the International Initiative for Impact Evaluation (3ie), October 25, 2012. Retrieve from https://oxfamblogs.org/fp2p/getting-evaluation-right-a-five-point-plan/

The 5-point plan is a prototype for an organization’s evaluation policy that embeds evaluation exercises throughout the program planning, implementation, and follow-up processes.

This is an outline of the five points; go to the article for details. The audience is an International Non-governmental Organization.

Point 1: Have a good theory of change/causal pathway/impact pathway or whatever you want to call it.

Theories of change are good for understanding the program, for schematics and great communication tools too. Additionally an evidence-based theory of change can help you decide where you need most investigation, where a process evaluation is sufficient, where a counterfactual analysis of outcomes is required and where a simple tracking of indicators is useful.

Point 2: Put in place monitoring and information systems. Track process and process/output and some outcome indicators across program areas.

Put together a set of detailed standard operating procedures for collecting information on process indicators. Train persons in doing the procedures and periodically verify that they are doing them correctly. At least one full-time skilled person should manage data collection and analysis.

Point 3: Think about measuring attributable change.

[FGC note. Consult with an expert about the costs and benefits of doing this right. Don’t agree to include this in an evaluation design unless the benefit is worth the cost.]

Point 4: Undertaking cost and cost effectiveness studies.

What are the priced and non-priced inputs in the project? Think about whether you want to use these projects in other places? Scale them up?

Put together a standardized template with cost categories and measurement methods. (E.g. how will you measure the cost of using good seeds for the farmer? It’s not just the cost of procurement or transportation but also the cost of additional manure, the cost of storage for seed and post-harvest produce.)

Point 5: Focus on implementation research as an important part of your program design.

Systematically document implementation factors, and put together a protocol which contains questions that are relevant to informing all stages of the evaluation. This is where participatory methods, focus groups, observational scrutiny, process research should come in, and also inform your theory of change.

Types of Program Results

A program result is the difference between the status of some condition prior to program implementation and the status of that condition after program implementation. There are several types of results. I am using terminology that is common in evaluation literature.

I am doing this exercise because I believe good evaluation involves using language precisely as different viewpoints are explored. For example, saying that a program achieves results is not precise use of language. Achievement is something that is a consequence of human action. So program implementation, the actions of human actors, achieves results.

Program effects are results directly related to expectations for the program, often in the form of aims, goals, and objectives.

Program impact are effects that are practically significant and sustained over a relatively long period of time. The evaluation needs to rule out plausible non-program factors that could have led to each effect; the evaluation reporting should include the conclusions of these efforts.

Side effects are results that are not directly related to program expectations.

Undesirable consequences are results that are harmful in someway.

I welcome comments.

Biblical Characteristics of Excellence

These five characteristics are from  http://greaterseas.com/2011/05/biblical-excellence/, a Christian business blog. Years ago I obtained this material; today I cannot find it at this site. Please share with me materials you consult on this topic.

Excellence begins in the heart; you cannot be ordinary in your heart and extraordinary in your life. Similarly you will not be ordinary in your life if you are extraordinary in your heart. As you seek to pursue excellence, consider the following five characteristics of excellence that are often overlooked.

1. Based In Humility

Excellence cannot occur without humility. A proud man will hide his inadequacies, but a humble man will expose them and seek to excel in his weakness through the perfection of Christ. James 4:6 says, “God is opposed to the proud, but gives grace to the humble.” Only through the grace of God and recognition of the fact that we can do nothing apart from Christ can we succeed in the pursuit of excellence.

2. Developed Over Time

No one is born excellent. Instead, excellence is developed over time by experience and the testing of faith. James 1:2-4 tells us that trials produce perseverance and perseverance produces faith – thus making you mature and complete.

Furthermore, excellence is developed in the details. If you do the little things diligently every day to improve, excellence becomes attainable. It’s all about implementation. Plan to be excellent, and then take every step necessary to fulfill that plan.

3. Grounded In Skeptical Inquiry

1 John 4:1 says, “Do not believe every spirit, but test the spirits to see whether they are from God…” Excellence demands a skeptical inquirer because it is not easily blown and tossed by the wind (James 1:6), therefore enabling you to make sound decisions, pursue only things worth pursuing, and function as a trusted resource in every matter of life. By being a skeptical inquirer, you reduce wasted thoughts, actions, and ideas and focus completely on pursuits of excellence.

4. Complete [Pervasive]

Excellence is not a skill – it is an attitude (Ralph Marston; author of the blog, ‘The Daily Motivator’, http://greatday.com “Create Excellence,” July 23, 2007). Excellence is an attitude that touches every area of your life. You are called to be excellent in everything you do (2 Corinthians 8:7). True excellence means you dedicate yourself fully to everything you do, as working for the Lord (Colossians 3:23).

5. Benchmarks Against Itself

Excellence does not compare itself to others. Instead, people who are excellent benchmark their performance against themselves because they know God judges us not by our harvest but by our seed (Luke 12:48). Don’t look outward to measure your excellence; instead look inward and compare your growth to where you were in the past (Galatians 6:4).

Excellence in TE

Meditate on Philippians 1:9-11

It is my prayer that your love may abound more and more, with knowledge and all discernment, so that you may approve what is excellent, and may be pure and blameless for the day of Christ, filled with the fruits of righteousness which comes through Jesus Christ, to the glory and praise of God.

 

Excellence has to do with love and righteousness more than technical prowess. This is the proposition that I explore in this meditation.

We can discern what is best only to the extent that we have deep knowledge of God’s ways. We can act righteously only to the extent that we follow the ways of Jesus. We must be centered on God, not anything else, to do what is good. These principles apply to Transformative Evaluation as well as any other vocational occupation.

In the business leadership literature there is much said about excellence that falsely claims scriptural authority for it. This is one passage that is cited. What does the passage mean?

William Barclay makes these points about this passage; I have elaborated them. For another perspective on excellence see Philippians 4:8-9.

  • Love is always the way to knowledge. The prayer is that you will become more insightful and perceptive in your relationships with others so that you will not inadvertently injure them. As you learn more about them you will affirm all that is righteous in them in the eyes of Christ. Through their relationship with you they will learn more about what is right and what is wrong, and desire to do what is right.

For TE, relational love is a “methodology” that leads to deeper insight about program implementation and stakeholder perceptions about the merit and worth of the program.

  • Love is sensitive to the mind and heart of the one loved. You seek to understand what the other desires, and to please them in your relationship with them. If I love Jesus, then I will seek to understand and do what Jesus desires.

In an evaluation exercise, seek to understand what program participants desire, and their perceptions of the program outcomes. How does an evaluator please people through his interactions with them? An evaluator that demonstrates willingness to listen and document accurately, and to submit conclusions to participants for critique may please them.

  • A pure person is one who does not cause someone else to stumble or fall from righteous living. In our relationships with others we love them when we support them in following Jesus; their technical skill is irrelevant. Our focus should be on their life style more than their technical abilities. Our own “continuing education” should be focused on identifying our character flaws and overcoming them, so that we may be more helpful to others in building character. When others see us they should be attracted to Christianity, not repelled from it.
  • The goal for living is to win praise and glory for God, not ourselves. Whatever excellence we may have should point others to the reality of the grace of God. To the extent that our behavior focuses attention on our qualities or the qualities of things that we produce, it is impure.

Christian love is not blind, despite what worldly wisdom says about the nature of love. Christian love overflows in knowledge and depth of insight. God wants us to love wisely in truth. Use your head and test your feelings; love is not sentimentalism.

Evangelist Selwyn Hughes develops the theme, “the pursuit of excellence,” in his reflections on Jeremiah. Excellence is defined as “doing the work of God faithfully, industriously, and without cutting corners.” Hughes does not include the concept of meeting high standards in this definition. The emphasis is on perseverance in all situations, finishing the race rather than winning it. An implication for TE is that the evaluator regularly reflects on the relationship between what has been done in the evaluation exercise, and what God is doing in the area where the exercise is taking place.

Every person is inadequate for the tasks that God asks. God works through us. Let feelings of inadequacy lead you to greater dependence on God. Excellence depends on our response to God’s ability as God works through us, not on our ability. May this meditation contribute to your professional journey.