All posts by Evaluator Frank

Program Evaluation Policy and Procedures

An agency should have a policy or framework that applies to all program evaluation work it does. This is a set of broad statements about the minimum requirements for each evaluation, regardless of diversity in the evaluands and evaluation objectives.

A framework is a set of high-level standards. The framework in this post is based on a participatory approach to evaluation. There are eight elements in the framework.

Planning an evaluation

A Feasibility Study should be done to determine if the anticipated benefits of doing an evaluation will justify the estimated costs. See posts on Feasibility Study for details.

Assuming that an evaluation feasibility study supports doing a program evaluation, planning begins by preparing an evaluation design that includes the purpose of the evaluation, evaluation objectives, stakeholder groups, primary needs for information, methodology, reporting to different audiences, evaluators, budget, and a time line for each stage of the evaluation.

When the design document is consistent with this framework, generally the evaluation design is the document approved by the appropriate representatives of management, partners, and program participants. The evaluators are accountable for using the design to plan and complete evaluation activities that achieve the evaluation purpose and objectives.

A detailed work plan evolves as activities are scheduled and completed to achieve the evaluation objectives. Each evaluation plan will be reviewed against this framework. Exceptions to elements in this framework will be explained in the evaluation plan.

1. Values – Characteristics of an evaluation that are valued most.
  1. In addition to typical methods for collecting and analyzing information to achieve evaluation objectives, the evolving plan shall include designated time for reflection and discernment.
  2. Participatory methods shall be used throughout all aspects of the evaluation exercise. These methods include involving stakeholders in developing questions and engaging in analyzing and interpreting collected data.
  3. The goodness of a program shall be defined by notions of goodness described by different groups of stakeholders as described in the evaluation design. Evidence will be collected for each notion included in the approved evaluation design.
2. Utilization of findings.
  1. The evaluation design will describe the primary audience for using the evaluation findings, and how that audience intends to use them. Other audiences can use at least some of the findings but they may have limitations. The evaluation report will describe limitations in using findings for other purposes.
  2. The evaluation design will also describe other audiences and the probable means of reporting to them.
3. Theory of Social Change (ToC) within the surrounding context.
  1. Each evaluation will examine the appropriateness of the implicit and explicit theory of change undergirding the program design.
  2. Each evaluation will document the process followed to develop the program design. The evaluator will comment on the process regarding the role of ToC in the process.
  3. Each evaluation will document achievements and how the interactions between project persons, partners and participants reflect Christian values. If achievement or interactions are unsatisfactory the evaluation report will include recommendations regarding investigating theories of change that may guide future programing that will have better results.
4. Knowledge of assets in the context that strengthen program results.
  1. The evaluation will examine the assessments that guided the program design to determine if assets were considered; if so, the evaluation will document how the design included assets to strengthen the program.
  2. The evaluation will examine how the program monitored assets in the context and how management responded to opportunities to use them.
5. Knowledge of obstacles in the context that could reduce program effectiveness or efficiency.
  1. The evaluation will examine the identification of assumptions that were considered which if valid would have major negative consequences and how that affected the design of the program.
  2. The context for each program result will be examined to identify obstacles to achieving maximum results.
6. Assumptions about evaluation approach.
  1. When program objectives can be achieved by applying knowledge based on cause-effect relationships, the evaluation shall use appropriate methods to document and analyze significance of achievements. This is outcome or impact evaluation.
  2. Generally cause-effect methods are not appropriate for documenting change in spiritual dimensions of reality. To evaluate such change an evaluation will include rigorous documentation of information collected through spiritual practices and qualitative methods of inquiry.
  3. If both types of evaluation are desired for a program, key stakeholders should agree on whether there will be two separate evaluations or one mixed methods evaluation. If the decision is to do one mixed methods evaluation, then the evaluation team needs to include an experienced cause-effect evaluator and an experienced spiritual-qualitative evaluator who respect each other’s expertise.
7. Program implementation monitoring.

Various aspects of program implementation should be monitored at least quarterly. An evaluation design should include an objective to examine monitoring results for the period covered by the evaluation. Typical topics to analyze: adequacy of the indicators used, validity and reliability of indicator results, use of monitoring information by management; accuracy of reporting, etc.

8. Framework revision.

Every five years this framework will be reviewed by the agency and partners to determine its relevance and usefulness. A participatory process will be used to determine modifications.

Getting Program Evaluation Right

This post is an outline of the article: “Getting evaluation right: a five point plan,” by Dr. Jyotsna Puri, Deputy Executive Director and Head of Evaluation of the International Initiative for Impact Evaluation (3ie), October 25, 2012. Retrieve from

The 5-point plan is a prototype for an organization’s evaluation policy that embeds evaluation exercises throughout the program planning, implementation, and follow-up processes.

This is an outline of the five points; go to the article for details. The audience is an International Non-governmental Organization.

Point 1: Have a good theory of change/causal pathway/impact pathway or whatever you want to call it.

Theories of change are good for understanding the program, for schematics and great communication tools too. Additionally an evidence-based theory of change can help you decide where you need most investigation, where a process evaluation is sufficient, where a counterfactual analysis of outcomes is required and where a simple tracking of indicators is useful.

Point 2: Put in place monitoring and information systems. Track process and process/output and some outcome indicators across program areas.

Put together a set of detailed standard operating procedures for collecting information on process indicators. Train persons in doing the procedures and periodically verify that they are doing them correctly. At least one full-time skilled person should manage data collection and analysis.

Point 3: Think about measuring attributable change.

[FGC note. Consult with an expert about the costs and benefits of doing this right. Don’t agree to include this in an evaluation design unless the benefit is worth the cost.]

Point 4: Undertaking cost and cost effectiveness studies.

What are the priced and non-priced inputs in the project? Think about whether you want to use these projects in other places? Scale them up?

Put together a standardized template with cost categories and measurement methods. (E.g. how will you measure the cost of using good seeds for the farmer? It’s not just the cost of procurement or transportation but also the cost of additional manure, the cost of storage for seed and post-harvest produce.)

Point 5: Focus on implementation research as an important part of your program design.

Systematically document implementation factors, and put together a protocol which contains questions that are relevant to informing all stages of the evaluation. This is where participatory methods, focus groups, observational scrutiny, process research should come in, and also inform your theory of change.

Types of Program Results

A program result is the difference between the status of some condition prior to program implementation and the status of that condition after program implementation. There are several types of results. I am using terminology that is common in evaluation literature.

I am doing this exercise because I believe good evaluation involves using language precisely as different viewpoints are explored. For example, saying that a program achieves results is not precise use of language. Achievement is something that is a consequence of human action. So program implementation, the actions of human actors, achieves results.

Program effects are results directly related to expectations for the program, often in the form of aims, goals, and objectives.

Program impact are effects that are practically significant and sustained over a relatively long period of time. The evaluation needs to rule out plausible non-program factors that could have led to each effect; the evaluation reporting should include the conclusions of these efforts.

Side effects are results that are not directly related to program expectations.

Undesirable consequences are results that are harmful in someway.

I welcome comments.

Biblical Characteristics of Excellence

These five characteristics are from, a Christian business blog. Years ago I obtained this material; today I cannot find it at this site. Please share with me materials you consult on this topic.

Excellence begins in the heart; you cannot be ordinary in your heart and extraordinary in your life. Similarly you will not be ordinary in your life if you are extraordinary in your heart. As you seek to pursue excellence, consider the following five characteristics of excellence that are often overlooked.

1. Based In Humility

Excellence cannot occur without humility. A proud man will hide his inadequacies, but a humble man will expose them and seek to excel in his weakness through the perfection of Christ. James 4:6 says, “God is opposed to the proud, but gives grace to the humble.” Only through the grace of God and recognition of the fact that we can do nothing apart from Christ can we succeed in the pursuit of excellence.

2. Developed Over Time

No one is born excellent. Instead, excellence is developed over time by experience and the testing of faith. James 1:2-4 tells us that trials produce perseverance and perseverance produces faith – thus making you mature and complete.

Furthermore, excellence is developed in the details. If you do the little things diligently every day to improve, excellence becomes attainable. It’s all about implementation. Plan to be excellent, and then take every step necessary to fulfill that plan.

3. Grounded In Skeptical Inquiry

1 John 4:1 says, “Do not believe every spirit, but test the spirits to see whether they are from God…” Excellence demands a skeptical inquirer because it is not easily blown and tossed by the wind (James 1:6), therefore enabling you to make sound decisions, pursue only things worth pursuing, and function as a trusted resource in every matter of life. By being a skeptical inquirer, you reduce wasted thoughts, actions, and ideas and focus completely on pursuits of excellence.

4. Complete [Pervasive]

Excellence is not a skill – it is an attitude (Ralph Marston; author of the blog, ‘The Daily Motivator’, “Create Excellence,” July 23, 2007). Excellence is an attitude that touches every area of your life. You are called to be excellent in everything you do (2 Corinthians 8:7). True excellence means you dedicate yourself fully to everything you do, as working for the Lord (Colossians 3:23).

5. Benchmarks Against Itself

Excellence does not compare itself to others. Instead, people who are excellent benchmark their performance against themselves because they know God judges us not by our harvest but by our seed (Luke 12:48). Don’t look outward to measure your excellence; instead look inward and compare your growth to where you were in the past (Galatians 6:4).

Excellence in TE

Meditate on Philippians 1:9-11

It is my prayer that your love may abound more and more, with knowledge and all discernment, so that you may approve what is excellent, and may be pure and blameless for the day of Christ, filled with the fruits of righteousness which comes through Jesus Christ, to the glory and praise of God.


Excellence has to do with love and righteousness more than technical prowess. This is the proposition that I explore in this meditation.

We can discern what is best only to the extent that we have deep knowledge of God’s ways. We can act righteously only to the extent that we follow the ways of Jesus. We must be centered on God, not anything else, to do what is good. These principles apply to Transformative Evaluation as well as any other vocational occupation.

In the business leadership literature there is much said about excellence that falsely claims scriptural authority for it. This is one passage that is cited. What does the passage mean?

William Barclay makes these points about this passage; I have elaborated them. For another perspective on excellence see Philippians 4:8-9.

  • Love is always the way to knowledge. The prayer is that you will become more insightful and perceptive in your relationships with others so that you will not inadvertently injure them. As you learn more about them you will affirm all that is righteous in them in the eyes of Christ. Through their relationship with you they will learn more about what is right and what is wrong, and desire to do what is right.

For TE, relational love is a “methodology” that leads to deeper insight about program implementation and stakeholder perceptions about the merit and worth of the program.

  • Love is sensitive to the mind and heart of the one loved. You seek to understand what the other desires, and to please them in your relationship with them. If I love Jesus, then I will seek to understand and do what Jesus desires.

In an evaluation exercise, seek to understand what program participants desire, and their perceptions of the program outcomes. How does an evaluator please people through his interactions with them? An evaluator that demonstrates willingness to listen and document accurately, and to submit conclusions to participants for critique may please them.

  • A pure person is one who does not cause someone else to stumble or fall from righteous living. In our relationships with others we love them when we support them in following Jesus; their technical skill is irrelevant. Our focus should be on their life style more than their technical abilities. Our own “continuing education” should be focused on identifying our character flaws and overcoming them, so that we may be more helpful to others in building character. When others see us they should be attracted to Christianity, not repelled from it.
  • The goal for living is to win praise and glory for God, not ourselves. Whatever excellence we may have should point others to the reality of the grace of God. To the extent that our behavior focuses attention on our qualities or the qualities of things that we produce, it is impure.

Christian love is not blind, despite what worldly wisdom says about the nature of love. Christian love overflows in knowledge and depth of insight. God wants us to love wisely in truth. Use your head and test your feelings; love is not sentimentalism.

Evangelist Selwyn Hughes develops the theme, “the pursuit of excellence,” in his reflections on Jeremiah. Excellence is defined as “doing the work of God faithfully, industriously, and without cutting corners.” Hughes does not include the concept of meeting high standards in this definition. The emphasis is on perseverance in all situations, finishing the race rather than winning it. An implication for TE is that the evaluator regularly reflects on the relationship between what has been done in the evaluation exercise, and what God is doing in the area where the exercise is taking place.

Every person is inadequate for the tasks that God asks. God works through us. Let feelings of inadequacy lead you to greater dependence on God. Excellence depends on our response to God’s ability as God works through us, not on our ability. May this meditation contribute to your professional journey.

Formulating Recommendations in TE

This post consolidates and expands previous discussions of recommendations on this site. This is my current thinking (November 2019) about this central element of TE.

Defining recommendation

I define a recommendation as a statement offered as worthy of acceptance or approval by stakeholders.  Based on available evidence, knowledge and experience the evaluator is saying that it is reasonable for stakeholders to implement the action/change included in the statement.

Recommendations should be developed in consultation with various stakeholder groups. Ideally, conflicts that emerge should be resolved through reflection as described below. If they cannot be resolved the core of the disagreement should be included in the report respectfully and fairly.

It is essential to keep in mind, however, that as stakeholders consider the recommendation in light of other factors they may decide reasonably not to implement the recommendation. If stakeholders are involved in the interpretation of evaluation results before the report is prepared, the report is less likely to contain recommendations that are not implemented.

In transformative evaluation there are two types of recommendations.

  • The first type is what you expect to see in any program evaluation report: description (based on evidence) of changes to implement to improve the chances that program goals and objectives will be achieved efficiently and effectively.
  • The second type is description of changes to increase the likelihood that the program will enable individual or social transformation regardless of the program goals and objectives.

Primary features of a recommendation

There are four features of a sound recommendation in TE reporting.

  1. The proposed change is based on verifiable, credible evidence as reported to stakeholders.
  2. A recommendation is actionable if it describes change that can implemented to improve the evaluated program.
  3. A recommendation is feasible if relevant stakeholders agree that the resources needed to implement the proposed change are available or can be acquired in a reasonable period.
  4. A TE recommendation is a proposed change that is a consequence of prayerful critical reflection. In-depth prayerful reflection will enhance both types of recommendations.

Using reflection to formulate recommendations that can make a difference.

In transformative evaluation the recommendations should be an outcome of prayerful reflection regarding all conclusions along with insights about how God is at work in the communities. In TE both individual and group reflection are important. In either case the primary purpose is to discern what God is calling forth in this situation from those who seek to live as they were created to live. In brief, reflection is an empowering activity that is essential for learning how to facilitate transformation through development work with communities and partners.

Facilitating a reflection event is much more than chairing a meeting.

Mutual trust and respect among the participants is required. This usually means that they have had positive interactions prior to the reflection event.

It is best if someone not involved with the program facilitates the event so that all key partners can participate fully.

Ideally the facilitator understands reflection from a variety of perspectives, such as

Freire’s (1996) approach to pedagogy (discussed below) and action learning. The sections that follow unpack this brief description.

Reflection is a dialogical activity.

Paulo Freire’s description of critical reflection is an excellent guide to planning reflection activities that lead to recommendations that really matter. Richard Schaull in the introduction to Freire’s Pedagogy of the Oppressed (1996) summed up Freire’s concept of critical reflection as comprised of dialogical activity, vocationally focused discussion and critical thinking that empowers individuals and groups to initiate humanizing change.

Freire’s starting point for understanding reflection is:

A person’s ontological vocation is to be a Subject who acts upon and transforms his world, and in so doing moves towards ever new possibilities of a fuller and richer life individually and collectively. In other words, meaningful “being” beyond merely existing involves making a positive difference in the world as an individual and in concert with others.

Don’t let “dialogical” distract or puzzle you. It simply means that reflection is a learning activity that involves talking and acting with other people on something that is important to everyone engaged in the reflection activity. The learning activity is much more than storing information. It involves identifying what is helpful and unhelpful in changing one’s personal and social context so as to experience living at its fullest.

Reflection is focused discussion.

Freire views the world as a problem to be worked on and solved. The world is the material that a person uses to create history and to overcome that which is dehumanizing at any particular time and place. Reflection is discussion that has such a vocational focus, or a focus on what God created us to do and be.

Focused discussion becomes reflection as emphasis is placed on identifying barriers to individual and social transformation. Program reflection is focused on specific information about the program rather than individual impressions or concerns. Documented quantitative and qualitative data are important for effective program reflection. Acknowledge and appreciate the good things that are happening, but concentrate on resolving issues or concerns.

For the Christian transformation occurs as God works within a particular context to redeem persons and groups from the consequences of sin. God often works through people to create opportunities for transformation, but transformation itself is God’s work. Focused discussion becomes reflection as the participants move beyond discussing what they can do to solve a problem to discussing their perceptions of the activity of God in their situation and how they can align their actions with it.

Reflection involves critical thinking.

Time should be given to enhancing critical thinking skills in each reflection event. Freire believed strongly that every person is capable of looking critically at his world in a dialogical encounter with others. Given the proper tools a person can gradually perceive his or her personal and social reality as well as the contradictions in it. He or she can become conscious of personal perceptions of that reality and deal critically with them. Each person, through such a process, wins back the right to say his own word, to name the world.

Critical thinking involves probing ideas from different perspectives looking for different understandings of what is real in the context. The probing must be done respectfully, assuming that each person’s perspective has value as the group describes the parts of reality that they have in common.

Reflection can lead to empowerment.

When people are able to name the world from the perspective of being fully human they are empowered with a new sense of dignity and hope. They are more likely to take upon themselves the struggle to change the structures of society which until now have served to oppress them.

Prayer throughout the reflection exercise is very important. For the Christian empowerment is based on understanding the will of God for the individual or group and intentionally aligning behavior with it.

Essence of Measurement

The usual definition of measurement is the assignment of numbers to something according to specified rules, where empirical evidence can be used to validate the results (Magnusson, 1966, see p.1).

But at least 40 different ways of using the term “measure” have been identified (Lorge, 1951). The same term can mean the act of weighing, the balance by which weighing is done, the grams that are used to balance an object, or the numeral that expresses the result of the weighing process. It can be used to refer to any instrument used as a basis for comparison, even when that comparison involves processes of estimation or judgment.

“Measure” is used more frequently to refer to acts of subjective estimates than to precise objective determination.

It is not likely that consensus will emerge regarding one meaning for “measure” or associated terms. Most people, however, expect some quantity will be used to express the outcome of some measurement process. In general, measurement involves assigning a class of numerals to a class of objects by applying a specific set of rules or procedures. (See Numbers and Numerals All measurement involves a person’s use of perceptual faculties, either unaided or extended by instrumentation of some sort.

Thus the classes of objects, the conceptual organization of discourse about those objects, the nature of instruments used, and the training of the observer all influence the results of a measurement process. Some observations are made directly — for example, the observer lays the ruler on the face of an object and compares the ends of the face with the markings on the ruler. The property of interest is observed directly.

Other observations are indirect — for example, the length of a column of mercury in a thermometer has been correlated with variations in temperature. In this case the effects of temperature on the column of mercury are observed, and then an inference is made about temperature by observing the column of mercury.

In scientific measurement, regardless of the field, the conditions for observation are carefully specified in terms of time, place, and circumstance. Observations are independently verified under the same conditions, with the results reported in terms of probable error.

Lorge (1951) makes the important observation that what is observed depends upon man’s conceptual equipment to translate sensory experiences into the notion of a property. Frequently the notion of a property will change as attempts are made to measure it. Detailed description of the property is essential.

Statements about a property are empirical in that they depend on what is experienced. Science demands that observations be reproducible — measurements must be reproducible. Assuming the property remains constant, measurement of the property under the same conditions by different observers should yield the same result.

Lorge, I. (1951). The fundamental nature of measurement. In E. F. Lindquist (ed.), Educational measurement. Washington, D.C.: American Council on Education. (pp. 533-559).

Magnusson, D. (1966). Test theory. Reading, MA: Addison-Wesley.

Numbers and Numerals

Number is a mathematical concept that is used to develop various formal systems of mathematical reasoning. Generally, the concept rests at the tacit level of understanding in the sense that we assume that we know what it means, and that others have the same meaning in mind. But the concept becomes more complex as we play with it.

Consider the formal number systems.

  • There are natural numbers, which is the set [1, 2, 3, …].
  • There are whole numbers, which is the set of natural numbers with the added element 0.
  • Within the set of whole numbers there is the subset : [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]. Each of the ten elements in this subset is called a digit. Digits are combined in various ways with other symbols to represent numbers from a variety of number sets.
  • There are integers, which is based on the property of symmetry around 0: the set is […, -3, -2, -1, 0, 1, 2, 3, …]. We say that integers can be negative or positive, depending on their location with respect to 0.
  • There are rational numbers, which is the set of integers plus all of the ratios of two integers, p/q, where q is not 0. (Thus, ½ is a rational number, as is 312/311.)
  • Then there are numbers that cannot be expressed as the ratio of integers, such as the ratio of the circumference of a circle to its diameter, called pi. These are called irrational numbers.
  • The set of rational numbers combined with the set of irrational numbers is called the set of real numbers.

There are larger sets of numbers, but the point is made that number is a mathematical concept that is a building block for formal systems of mathematical reasoning.

For most purposes, especially in the social sciences, in measurement discourse numerical references are to elements in the set of real numbers. What does that mean?

Usually we use the term “number” to refer to the result of counting specific units, or the result of performing some group of mathematical operation on that result. For example, six (6) cars went through the red traffic light at this intersection on Monday between 7:00 a.m. and 9:00 a.m. The unit is a car that has certain characteristics (passed through the intersection in the two-hour period when the traffic light was red). The number “6” represents the result of someone counting the defined units. (The same principle applies if the counting was done by a set of instruments set up at the intersection.)

Suppose the results of five days of counting were 6, 7, 8, 10, and 5. What does the statement mean: In the five days an average of 7 1/5 cars went through the red light. The number “7 1/5” is the result of the mathematical operations of adding the five numbers and dividing the total by the number 5 (which is the result of counting the number of days that cars were counted). This is a rational number (it can be expressed as the ratio of integers 36/5). The process for deriving the number 7 1/5 is clear, but what does it mean?

Mathematically speaking, it means that 7 1/5 cars going through the red light every day for five days is the same total number of cars that were actually observed going through a red light in that period. Without knowing the actual counts for each day, we usually assume that about 7 cars were observed each day, maybe a few more on a particular day and a few less on another day. Presumably the observer did not see a part of a car go through the intersection.


A numeral is a symbol that represents a number. Numerals have been devised to facilitate mathematical reasoning. The result of counting six cars can be represented by the Arabic numeral “6” or the Roman numeral “vi”. Arabic numerals are much easier to work with.

In measurement discourse, it is essential that we clearly define the unit that is being counted, the counting process and the mathematical operations performed. That is, it is essential that we define the numbers that are represented by the numerals in a particular situation. Otherwise, confusion lurks in the background.

For fascinating accounts of some relationships between numerals and numbers see Isaac Asimov,  (1977). Asimov on numbers. Garden City, NY: Doubleday.

Managing Evaluator Anxiety

6Do not be anxious about anything, but in every situation, by prayer and petition, with thanksgiving, present your requests to God. 7And the peace of God, which transcends all understanding, will guard your hearts and your minds in Christ Jesus. (Philippians 4:6-7)

Do not be anxious… Do not be apprehensive or fearful. Do not worry about something that might harm you in some way at some time.

I cannot prevent feeling anxious, but I can manage anxiety with the actions listed here by Paul. I can talk about the situation with God, and ask for help. That helps me realize that alone I cannot do what matters most.

Anxiety reminds me that I am not self sufficient in being what I was created to be. I am dependent on my loving Creator for knowing what that is, and then acting on that knowledge. In this sense anxiety is a friend.

In every situation… This includes everything that happens, or does not happen, in an evaluation. Stakeholders quarrel as the evaluation is being planned, torrential rain makes it impossible to travel to villages to interview and observe per the data collection plan, evaluation team members become ill, etc. In these and innumerable other circumstances instead of fretting about how an evaluation might be flawed I can, and should, pray for guidance to move forward.

With thanksgiving… in every situation… I commit to giving my anxieties to God.

The Big Picture for TE

TE is located at the intersection of a Christian worldview, the profession of evaluation, and approaches to community development.

Christian worldview

  • The ways of Jesus are a guide to practicing TE.

Evaluation profession

  • Within the profession program evaluation examines relationships between resources, processes and outcomes of a program, taking into account the surrounding context and various stake holder groups, as well as comparisons with alternative options.

Approaches to community development

  • Transformational approaches to community development seek sustainable changes among community residents related to loving God and neighbor, and empowerment of the community to influence social structures, systems and institutions for the common good.

Create a Venn diagram, TE Venn, that shows the intersections of the three circles that represent the three elements. Think about the features of each intersection; how does this inform you about the nature of TE?

This Venn diagram shows my thinking about the intersections. Venn Diagram for TE

Go to this post for six slide shows that provide an orientation to essential features of TE. Transformative Evaluation Overview–6 Slide Shows