How to ask the Right Evaluation Questions? OECD-DAC criteria helps!

The term evaluation is synonymous with research in its broadest sense; and thus, the research questions posed and the methods used are as varied as social sciences, humanities, and natural sciences fields can be! However, many often stumble at the first step – asking the “right evaluation questions”.

Organization for Economic Co-operation and Development – Development Assistance Committee (OECD-DAC) has developed six evaluation criteria. These criteria offer a flexible yet “guided” framework to ensure at least the right evaluation questions are asked. Evaluating all six evaluation criteria might not be suitable for every program. Therefore, decisions on:
(a) which criteria to use, and
(b) what questions to ask
…must be based on the context and purpose of the program.

The Right Evaluation Questions that can be asked under the OECD-DAC Criteria

Below, I explain which questions can be asked under the OECD-DAC criteria.

Relevance

Here, we assess the situation and needs of the programme population and evaluate whether the programme was suitably designed and targeted for maximizing benefit. So, the questions to ask are:

• Have we identified the right target population for an intervention?
• Is the intervention indeed a right one for the given population?
• Is/was the programme the best among other alternatives to address the development needs?

Coherence

Here, we recognize, appreciate, and assess a “causal package” that must exist for delivering impacts. A programme rarely delivers impacts on its own and must do so under the context of or in collaboration with other programs, government policies, communities’ capacity. The questions to ask include”

• Does the program create duplication of efforts?
• Does it undermine or supplement the effects of any existing programs or policies?
• Can programme succeed/fail if other programmes or support-system falters?

Effectiveness

This criterion is often confused with the impact criteria because sometimes “effect and impacts” are used as synonyms. Put simply, this criterion evaluates whether the program delivered the planned or committed outcomes; and does not evaluate higher-order, intended, or unintended impacts. The questions explored under this criterion can be:

• Can/did the program meet stated objectives or outcomes?
• Are there any unintended outcomes of the program?
• Did the program deliver the most “relevant” objectives?
• What is the relative importance of programme results?

Efficiency

Here, we evaluate if the program should or did result in maximum benefits in an economic and timely manner. It is not a financial and human resource audit but an inquiry into whether inputs are converted to outputs in the most efficient way. Some examples of questions to ask are:

• What technology and systems did the programme use to ensure efficiency?
• Is/was there any other implementation strategy which was more efficient?
• How does cost-effectiveness of the programme compare with other programmes?
• Did the programme leverage other programmes working in the area? (here, this criterion can overlap with the coherence criterion)

Impact

Here, we evaluate the extent to which the programme will/has delivered higher-level outcomes (known as impacts). (Read our blog that explains the difference between impact evaluation, impact assessment and programme evaluation). Proving causality is not a prime intent of impact assessments but evaluating the theory of change is. Impact assessment can be done before the programme begins and can assess unintended or negative outcomes. Some example questions are:

• What is the attributable impact of the programme on reducing maternal mortality? (a causal impact evaluation question)
• Has the programme negatively affected non-beneficiaries? (an example of seeking out unintended negative impact)
• In what ways did the program transform the lives of the beneficiaries? (open-ended inquiry)

Sustainability

Here, we assess whether the programme has created systems and capacities to sustain the program benefits in future once the implementers exit. The potential for sustainability is evaluated from social, economic, financial, environmental, and institutional perspectives. The questions can include:

• Is there capacity and intent in the community to continue the programme activities on their own?
• Is there any post-project monitoring system developed to track slippage and take corrective action?

I hope the list above can help you get started on asking the “right” evaluation questions.

You can also schedule a free micro-consulting to get any clarity on evaluation methods.