All posts by Evaluator Frank

Formulating Recommendations in TE

This post consolidates and expands previous discussions of recommendations on this site. This is my current thinking (November 2019) about this central element of TE.

Defining recommendation

I define a recommendation as a statement offered as worthy of acceptance or approval by stakeholders.  Based on available evidence, knowledge and experience the evaluator is saying that it is reasonable for stakeholders to implement the action/change included in the statement.

Recommendations should be developed in consultation with various stakeholder groups. Ideally, conflicts that emerge should be resolved through reflection as described below. If they cannot be resolved the core of the disagreement should be included in the report respectfully and fairly.

It is essential to keep in mind, however, that as stakeholders consider the recommendation in light of other factors they may decide reasonably not to implement the recommendation. If stakeholders are involved in the interpretation of evaluation results before the report is prepared, the report is less likely to contain recommendations that are not implemented.

In transformative evaluation there are two types of recommendations.

  • The first type is what you expect to see in any program evaluation report: description (based on evidence) of changes to implement to improve the chances that program goals and objectives will be achieved efficiently and effectively.
  • The second type is description of changes to increase the likelihood that the program will enable individual or social transformation regardless of the program goals and objectives.

Primary features of a recommendation

There are four features of a sound recommendation in TE reporting.

  1. The proposed change is based on verifiable, credible evidence as reported to stakeholders.
  2. A recommendation is actionable if it describes change that can implemented to improve the evaluated program.
  3. A recommendation is feasible if relevant stakeholders agree that the resources needed to implement the proposed change are available or can be acquired in a reasonable period.
  4. A TE recommendation is a proposed change that is a consequence of prayerful critical reflection. In-depth prayerful reflection will enhance both types of recommendations.

Using reflection to formulate recommendations that can make a difference.

In transformative evaluation the recommendations should be an outcome of prayerful reflection regarding all conclusions along with insights about how God is at work in the communities. In TE both individual and group reflection are important. In either case the primary purpose is to discern what God is calling forth in this situation from those who seek to live as they were created to live. In brief, reflection is an empowering activity that is essential for learning how to facilitate transformation through development work with communities and partners.

Facilitating a reflection event is much more than chairing a meeting.

Mutual trust and respect among the participants is required. This usually means that they have had positive interactions prior to the reflection event.

It is best if someone not involved with the program facilitates the event so that all key partners can participate fully.

Ideally the facilitator understands reflection from a variety of perspectives, such as

Freire’s (1996) approach to pedagogy (discussed below) and action learning. The sections that follow unpack this brief description.

Reflection is a dialogical activity.

Paulo Freire’s description of critical reflection is an excellent guide to planning reflection activities that lead to recommendations that really matter. Richard Schaull in the introduction to Freire’s Pedagogy of the Oppressed (1996) summed up Freire’s concept of critical reflection as comprised of dialogical activity, vocationally focused discussion and critical thinking that empowers individuals and groups to initiate humanizing change.

Freire’s starting point for understanding reflection is:

A person’s ontological vocation is to be a Subject who acts upon and transforms his world, and in so doing moves towards ever new possibilities of a fuller and richer life individually and collectively. In other words, meaningful “being” beyond merely existing involves making a positive difference in the world as an individual and in concert with others.

Don’t let “dialogical” distract or puzzle you. It simply means that reflection is a learning activity that involves talking and acting with other people on something that is important to everyone engaged in the reflection activity. The learning activity is much more than storing information. It involves identifying what is helpful and unhelpful in changing one’s personal and social context so as to experience living at its fullest.

Reflection is focused discussion.

Freire views the world as a problem to be worked on and solved. The world is the material that a person uses to create history and to overcome that which is dehumanizing at any particular time and place. Reflection is discussion that has such a vocational focus, or a focus on what God created us to do and be.

Focused discussion becomes reflection as emphasis is placed on identifying barriers to individual and social transformation. Program reflection is focused on specific information about the program rather than individual impressions or concerns. Documented quantitative and qualitative data are important for effective program reflection. Acknowledge and appreciate the good things that are happening, but concentrate on resolving issues or concerns.

For the Christian transformation occurs as God works within a particular context to redeem persons and groups from the consequences of sin. God often works through people to create opportunities for transformation, but transformation itself is God’s work. Focused discussion becomes reflection as the participants move beyond discussing what they can do to solve a problem to discussing their perceptions of the activity of God in their situation and how they can align their actions with it.

Reflection involves critical thinking.

Time should be given to enhancing critical thinking skills in each reflection event. Freire believed strongly that every person is capable of looking critically at his world in a dialogical encounter with others. Given the proper tools a person can gradually perceive his or her personal and social reality as well as the contradictions in it. He or she can become conscious of personal perceptions of that reality and deal critically with them. Each person, through such a process, wins back the right to say his own word, to name the world.

Critical thinking involves probing ideas from different perspectives looking for different understandings of what is real in the context. The probing must be done respectfully, assuming that each person’s perspective has value as the group describes the parts of reality that they have in common.

Reflection can lead to empowerment.

When people are able to name the world from the perspective of being fully human they are empowered with a new sense of dignity and hope. They are more likely to take upon themselves the struggle to change the structures of society which until now have served to oppress them.

Prayer throughout the reflection exercise is very important. For the Christian empowerment is based on understanding the will of God for the individual or group and intentionally aligning behavior with it.

Essence of Measurement

The usual definition of measurement is the assignment of numbers to something according to specified rules, where empirical evidence can be used to validate the results (Magnusson, 1966, see p.1).

But at least 40 different ways of using the term “measure” have been identified (Lorge, 1951). The same term can mean the act of weighing, the balance by which weighing is done, the grams that are used to balance an object, or the numeral that expresses the result of the weighing process. It can be used to refer to any instrument used as a basis for comparison, even when that comparison involves processes of estimation or judgment.

“Measure” is used more frequently to refer to acts of subjective estimates than to precise objective determination.

It is not likely that consensus will emerge regarding one meaning for “measure” or associated terms. Most people, however, expect some quantity will be used to express the outcome of some measurement process. In general, measurement involves assigning a class of numerals to a class of objects by applying a specific set of rules or procedures. (See Numbers and Numerals All measurement involves a person’s use of perceptual faculties, either unaided or extended by instrumentation of some sort.

Thus the classes of objects, the conceptual organization of discourse about those objects, the nature of instruments used, and the training of the observer all influence the results of a measurement process. Some observations are made directly — for example, the observer lays the ruler on the face of an object and compares the ends of the face with the markings on the ruler. The property of interest is observed directly.

Other observations are indirect — for example, the length of a column of mercury in a thermometer has been correlated with variations in temperature. In this case the effects of temperature on the column of mercury are observed, and then an inference is made about temperature by observing the column of mercury.

In scientific measurement, regardless of the field, the conditions for observation are carefully specified in terms of time, place, and circumstance. Observations are independently verified under the same conditions, with the results reported in terms of probable error.

Lorge (1951) makes the important observation that what is observed depends upon man’s conceptual equipment to translate sensory experiences into the notion of a property. Frequently the notion of a property will change as attempts are made to measure it. Detailed description of the property is essential.

Statements about a property are empirical in that they depend on what is experienced. Science demands that observations be reproducible — measurements must be reproducible. Assuming the property remains constant, measurement of the property under the same conditions by different observers should yield the same result.

Lorge, I. (1951). The fundamental nature of measurement. In E. F. Lindquist (ed.), Educational measurement. Washington, D.C.: American Council on Education. (pp. 533-559).

Magnusson, D. (1966). Test theory. Reading, MA: Addison-Wesley.

Numbers and Numerals

Number is a mathematical concept that is used to develop various formal systems of mathematical reasoning. Generally, the concept rests at the tacit level of understanding in the sense that we assume that we know what it means, and that others have the same meaning in mind. But the concept becomes more complex as we play with it.

Consider the formal number systems.

  • There are natural numbers, which is the set [1, 2, 3, …].
  • There are whole numbers, which is the set of natural numbers with the added element 0.
  • Within the set of whole numbers there is the subset : [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]. Each of the ten elements in this subset is called a digit. Digits are combined in various ways with other symbols to represent numbers from a variety of number sets.
  • There are integers, which is based on the property of symmetry around 0: the set is […, -3, -2, -1, 0, 1, 2, 3, …]. We say that integers can be negative or positive, depending on their location with respect to 0.
  • There are rational numbers, which is the set of integers plus all of the ratios of two integers, p/q, where q is not 0. (Thus, ½ is a rational number, as is 312/311.)
  • Then there are numbers that cannot be expressed as the ratio of integers, such as the ratio of the circumference of a circle to its diameter, called pi. These are called irrational numbers.
  • The set of rational numbers combined with the set of irrational numbers is called the set of real numbers.

There are larger sets of numbers, but the point is made that number is a mathematical concept that is a building block for formal systems of mathematical reasoning.

For most purposes, especially in the social sciences, in measurement discourse numerical references are to elements in the set of real numbers. What does that mean?

Usually we use the term “number” to refer to the result of counting specific units, or the result of performing some group of mathematical operation on that result. For example, six (6) cars went through the red traffic light at this intersection on Monday between 7:00 a.m. and 9:00 a.m. The unit is a car that has certain characteristics (passed through the intersection in the two-hour period when the traffic light was red). The number “6” represents the result of someone counting the defined units. (The same principle applies if the counting was done by a set of instruments set up at the intersection.)

Suppose the results of five days of counting were 6, 7, 8, 10, and 5. What does the statement mean: In the five days an average of 7 1/5 cars went through the red light. The number “7 1/5” is the result of the mathematical operations of adding the five numbers and dividing the total by the number 5 (which is the result of counting the number of days that cars were counted). This is a rational number (it can be expressed as the ratio of integers 36/5). The process for deriving the number 7 1/5 is clear, but what does it mean?

Mathematically speaking, it means that 7 1/5 cars going through the red light every day for five days is the same total number of cars that were actually observed going through a red light in that period. Without knowing the actual counts for each day, we usually assume that about 7 cars were observed each day, maybe a few more on a particular day and a few less on another day. Presumably the observer did not see a part of a car go through the intersection.

Numeral

A numeral is a symbol that represents a number. Numerals have been devised to facilitate mathematical reasoning. The result of counting six cars can be represented by the Arabic numeral “6” or the Roman numeral “vi”. Arabic numerals are much easier to work with.

In measurement discourse, it is essential that we clearly define the unit that is being counted, the counting process and the mathematical operations performed. That is, it is essential that we define the numbers that are represented by the numerals in a particular situation. Otherwise, confusion lurks in the background.

For fascinating accounts of some relationships between numerals and numbers see Isaac Asimov,  (1977). Asimov on numbers. Garden City, NY: Doubleday.

Managing Evaluator Anxiety

6Do not be anxious about anything, but in every situation, by prayer and petition, with thanksgiving, present your requests to God. 7And the peace of God, which transcends all understanding, will guard your hearts and your minds in Christ Jesus. (Philippians 4:6-7)

Do not be anxious… Do not be apprehensive or fearful. Do not worry about something that might harm you in some way at some time.

I cannot prevent feeling anxious, but I can manage anxiety with the actions listed here by Paul. I can talk about the situation with God, and ask for help. That helps me realize that alone I cannot do what matters most.

Anxiety reminds me that I am not self sufficient in being what I was created to be. I am dependent on my loving Creator for knowing what that is, and then acting on that knowledge. In this sense anxiety is a friend.

In every situation… This includes everything that happens, or does not happen, in an evaluation. Stakeholders quarrel as the evaluation is being planned, torrential rain makes it impossible to travel to villages to interview and observe per the data collection plan, evaluation team members become ill, etc. In these and innumerable other circumstances instead of fretting about how an evaluation might be flawed I can, and should, pray for guidance to move forward.

With thanksgiving… in every situation… I commit to giving my anxieties to God.

The Big Picture for TE

TE is located at the intersection of a Christian worldview, the profession of evaluation, and approaches to community development.

Christian worldview

  • The ways of Jesus are a guide to practicing TE.

Evaluation profession

  • Within the profession program evaluation examines relationships between resources, processes and outcomes of a program, taking into account the surrounding context and various stake holder groups, as well as comparisons with alternative options.

Approaches to community development

  • Transformational approaches to community development seek sustainable changes among community residents related to loving God and neighbor, and empowerment of the community to influence social structures, systems and institutions for the common good.

Create a Venn diagram, TE Venn, that shows the intersections of the three circles that represent the three elements. Think about the features of each intersection; how does this inform you about the nature of TE?

This Venn diagram shows my thinking about the intersections. Venn Diagram for TE

Go to this post for six slide shows that provide an orientation to essential features of TE. Transformative Evaluation Overview–6 Slide Shows

Widow’s Offering Mark 12:38 – 13:2

The widow’s offering stimulated one of my earliest reflections on scriptural guidance for thinking about evaluation. I saw Jesus observing the activity at the temple treasury demonstrating key features of evaluation work. See Widow’s Offering Mark 12:41-44

Recently I read Practicing Resurrection, Janet Wolf’s commentary on the Gospel of Mark. She points out that this brief story is situated between stern warnings about judgment that will be made on those who put more value on religiosity than on living righteously. I had not examined the larger context (Mark 12:38 – 13:2).

Evaluators, explore the larger context surrounding the evaluand. You may discover significant findings that otherwise will be overlooked.

References

Rice, Jim. “Where is Thy God?” November 8, 2009. Sojourners Preaching the Word [online]

Smith, Geoffrey. “A Closer Look At The Widow’s Offering: Mark 12:41-44,” 1997;2002. Journal of the Evangelical Theological Society (Vol. 40, Page 27-36).

Wolf, Janet. Practicing Resurrection: The Gospel of Mark and Radical Discipleship, 2019. (Mission Study) United Methodist Women. Kindle Edition.

Widow’s Offering Mark 12:41-44

The parable about the widow’s offering illustrates several important aspects of evaluation that are consistent with following Jesus.

Jesus quietly watched people as they went about their normal activity. His observations did not disrupt people, nor cause them to act in unusual ways. Useful evaluation takes place on ordinary days, not just during special events.

Jesus understood the culture. He knew the ways of the people that allowed him to understand what he was seeing. The evaluator must have relevant experience within the community’s culture to understand the significance of observations and information.

Jesus chose a life-style activity to observe, an activity that provided information about people’s central values. For the religious folk, contributions to the temple treasury were related to the degree to which they trusted in God’s commitment to provide for their own needs. The evaluation should include observations of behavior related to core values, especially those that indicate love for God and neighbor in that culture. This can be very complex, but it is essential for the community development evaluator.

Jesus related one fact to another to describe a relationship. Evaluation works with patterns or relationships, not isolated facts. Jesus said that rich people put large amounts into the treasury, while the widow put in two small coins. Jesus did not describe the exact amounts for the rich people, because the comparison was not between the amounts given by the rich people and the widow. The important comparison was between the relative amounts that the rich and the widow had left over.

Jesus knew when not to be too precise. The strength of the conclusion in this situation does not depend on knowing the exact amounts that the rich put into the treasury. Quantitative analysis can be useful, but the appropriate precision is determined by the aim of the analysis. Avoid spending time collecting and analyzing information that is too precise.

Jesus interpreted what he saw by relating the observations to significant principles, and stating his conclusion. The widow had put more into the treasury than all the others, even though the actual amount was only two small coins. And he gave the evidence to support the conclusion: The others gave only a part of their wealth, while she gave everything she had to live on. A conclusion describes more than the facts you observe; it describes what the facts mean in relationship with each other.

In this parable Jesus made no recommendation. The disciples were left to ponder his conclusion, and to decide for themselves what they should do. Recommendations have a place in evaluation, but there is also a place for encouraging users of the results of an evaluation to think through recommendations for themselves.

Lord, I resolve to have the courage and humility to reconsider my understanding of evaluation regularly as I meditate on your teachings. Amen.

Defining Transformation

I have described my understanding of “transformation” in an earlier post. This post is a brief description from a slightly different perspective for a secular audience, and two brief descriptions for a Christian audience. The third description was added in May 2019.

First description…Transformation is change in the deepest part of one’s being that puts loving Creator of all universes and every neighbor, known and unknown, at the center of one’s worldview.

Personal Transformation is initiated by the Creator of all universes. Engaging in transformation involves:

  • Becoming aware of how my beliefs, attitudes, ways I interact with others are harmful to facilitating transformation in others.
  • Replacing obstructive practices with non-obstructive ones. Typically this will require dependence on the Creator for guidance and strength to sustain various changes in beliefs, attitudes and practices. A Spiritual Director can help with this.
  • Taking care of my own sets of needs for the purpose of being better prepared to enable and encourage transformation in others (putting on their new self and living out of it minute by minute).

Second description…Transformation is much more than change. Transformation is profound holistic change at the root of being. A transformed person is sustained by kingdom values. A transformed society is sustained by kingdom values. Any individual or social change that does not involve living by kingdom values is not transformation.

One outcome of Transformation is more clarity on one’s true vocation as children of God. That vocation is to be more loving and just in their relationships with self, with each other, with the community, with God, and with their environment.

Another outcome is willingness to confront evil in their personal lives and in the community. Suffering that restores right relationships is an integral part of transformational development; Jesus on the cross is an essential image for understanding transformational development.

Third description… Selwyn Hughes, British evangelist, described transformation as a lifelong continuous process of being released from the deadly disease of unbalanced preoccupation with self (Every Day With Jesus, January 02, 2006).

God’s goal for those who accept God’s invitation to be fully human is that they should become like Christ in their character. (See 1 John 3:2.) This is the essence of transformation. This is the work of God in the world; to restore us fallen beings to the original image of God; to transform us.

Transformation turns you inside out; outside in

Meditate on Ephesians 4:17-28

This is a message I spoke to community development staff in Uganda in November 1998.

As an evaluator I search for truth. As a facilitator I help others search for truth, accept truth, live by truth.

In searching for truth, you need to watch out for lies, especially clever lies. In the restaurant at Laston Hotel in Masaka there is a poster that says:

Pilsner Lager
Extra Strong
It’s got what you want

This poster tells a lie. A lie that is believed as true by many. Those who live by this lie cause harm and destruction to others and themselves. In my own family I have seen the harm and destruction that addiction to alcohol can cause. If this poster were to tell the truth, it might look like this:

Pilsner Lager
Extra Strong
If you want to cause harm and destruction to yourself and others,
It’s got what you want

Now hear the word of the Lord: Ephesians 4:17-28

Verse 28 describes the fundamental indicator for transformational development. The person who steals is to stop stealing. Not only that, the person is to produce useful things with his or her own hands, that he or she might share them with others. This is transformation, from stealing to gratify the self, to being productive on behalf of others.

This passage describes the key to planning development programs that are transformational, not just enhancements to living in the world. This passage describes the key to planning programs that will inspire hope for those who are in despair. It describes the key to bringing joy to those who are in very difficult circumstances. Transformational development is based on the truth as it is taught by Jesus.

To understand the truth that is in Jesus, we need to face the fact that each and every one of us in a sinner. As told in the early chapters of Genesis, we have inherited the tendency to act as if we are God rather than being a child of God. God, the omnipotent, loving Creator of all there is, was, and ever will be, allowed us the freedom to disobey, while longing deeply that we would not disobey. But we did, and so we are sinners.

The truth that is in Jesus is that God, through boundless grace, invites us as sinners to be in His presence as forgiven sinners. If we accept this invitation, if we turn our backs on the deceitful ways of the world, we can live in God’s emerging kingdom right now. Cleansed, purified, made new in attitudes of mind, free to put on a new self, free to be like God in true righteousness and holiness.

Free to stop stealing, free to work with our hands to make something useful so that we can share it with others. Free to turn inside out and outside in, free to say “no” to the lies all around us, free to live by the truth that is in Jesus

ToR, Evaluation Design, Evaluation Plan

There are different types of plans that guide an evaluation exercise. I define three of them in this post.

Terms of reference (ToR)… Stakeholder expectations to be met by the evaluator, and evaluator expectations to be met by the client. The document is like a contract.

Evaluation design… The purpose and objectives of the evaluation exercise, and the means for achieving them. The primary function of the design is to anticipate possible alternative explanations for program results, and describe data collection and analysis methods to render implausible as many of them as feasible, leaving a stronger case that results can be attributed to program implementation.

Evaluation plan… Day-by-day assignments for each member of the evaluation team for each day of the exercise.

Comments welcome.