Michael Scriven, a renowned evaluator for decades, developed the Key Evaluation Checklist (KEC) to guide evaluators in planning and implementing an inquiry that is a legitimate program evaluation. The KEC, which has gone through many revisions, is described in the Evaluation Thesaurus (1991, pp. 204-211). Each item on the checklist also has a separate entry in the Thesaurus. An outline is on p.83 of Foundations (1991). Another version is in Evaluation Models (1983).
The checklist identifies the multiple dimensions which must be explored prior to the final synthesis in an evaluation. Failure to do so increases the risk dramatically that the evaluation will include invalid arguments. Many iterations of the KEC are involved in the typical evaluation, generating a process of successive approximation. It is used in preparing the design, but it is also used during data collection and analysis and report preparation.
Another use for the checklist is to critique an evaluation report or a description of a completed evaluation inquiry. This link, KEC charts, gives my brief description of 14 features of legitimate program evaluation based the references to Scriven’s work. Study the references carefully if you decide to use this approach to test the value of your evaluation work.
For more information
Scriven, Michael. 1983. Evaluation Ideologies. In Madaus, George F.; Scriven, Michael; Stufflebeam, Daniel L. (Eds.). Evaluation Models: Viewpoints on Educational and Human Services Evaluation. 258-260. Sage.
Scriven, Michael. 1991. Evaluation Thesaurus (fourth edition). Sage.
Shadish, William R., Jr.; Cook, Thomas D.; Leviton, Laura C. 1991. Foundations of Program Evaluation: Theories of Practice. Sage.
A common program evaluation tool is a list of criteria or indicators of merit. Michael Scriven has given some helpful guidelines for constructing such a tool in his Evaluation Thesaurus, fourth edition, 1991, page 80-81. Examples of checklists are a checklist for program implementation and a checklist for program logic.
Here is a checklist for evaluation checklists. I welcome comments and alternative checklists.
* = required characteristic.
A checklist that is missing one required characteristic is a poor checklist. One that is missing two or more characteristics is a psuedochecklist.
( ) * The list includes all significant relevant dimensions of value.
( ) * Absolute requirements are marked and placed at the beginning of the list.
( ) * The criteria or indicators are measurable or objectively observable.
( ) * Weights are assigned to the criteria or indicators. The procedure for using weights is clearly described.
( ) There is minimal overlap between items. Overlap is clearly described. 0-3 points.
( ) A long list groups items into meaningful categories. 0-3 points.
( ) The checklist is easy to remember, understandable and easy to use. 0-3 points.
Assuming the required characteristics are satisfactory, the other three items have equal weights. The point totals can be interpreted as follows:
7-9 = good checklist
4-6 = fair checklist; think about spending some time improving it
0-5 = definitely should be improved
Transformative Evaluation (TE) is a blend of two perspectives on reality: the Christian perspective and the science perspective. It is difficult for any evaluator to do competent evaluation work on community development from the science perspective. Speaking from professional experience it is qualitatively more than twice as difficult to plan and implement legitimate TE.
I take two approaches to describing legitimate TE. First, there are sets of characteristics that guide the evaluator and stakeholder groups as they craft a particular evaluation.
Second, there are different types of flawed inquiries that are not legitimate evaluation work. Examining a particular inquiry against the features of a legitimate evaluation and the features of flawed inquiries that inappropriately may be called evaluations should lead to improved TE.
In the coming weeks I will be sharing posts about TE from both approaches. I welcome constructive criticism as well as your own thoughts about what makes any inquiry a legitimate evaluation.