Evaluation is the determination of the value, nature, character, or quality of something or someone (Merriam-Webster, n.d.). Evaluation is a broad term that encapsulates a variety of activities that are often used to determine the success of an evaluand against pre-determined plans and objectives. It supports evidence-based decision and is most effective when used end to end but can add value as an end-on pursuit. The Organisation for Economic Co-operation and Development (OECD) established evaluation criteria in 1991, since then the criteria have been used widely and following a review, an adapted set of criteria were released in 2019; these are as follows:
- Relevance
- Coherence
- Effectiveness
- Efficiency
- Impact
- Sustainability (OECD, 2019).
These criteria are supported by two guiding principles, firstly, that the criteria should be contextualised to support high quality, useful evaluation and second, that the criteria is not used mechanistically as there are various factors which can reasonably impact whether a criterion is covered (OECD, 2019). The criteria application depends on the scope, purpose and limitations, which can influence how and if the criterion are used (OECD, 2019). Figure 1 displays how these criteria sit within boarder standards and guidelines.
Figure 1: How the Criteria Fit in with other Norms and Standards (OECD, 2021)
Evaluators should use these criteria as a framework along with other professional standards, guidelines and best practice to gather findings and determine merit on the intervention (usually a program, project or activity) (OECD, 2021). A good consultant will help clients gather and analyse data, measure outcomes, and draw conclusions about the overall performance and impact of their initiatives through evaluative reasoning on the evidence surfaced.
Evaluations often consider the different contexts at play which may not be immediately apparent, we personally like the Rog (2012) model shown in Figure 2. The vertical line of circles shows a linear and simplistic approach to evaluation. Oftentimes, a problem exists, an intervention is designed (in the form of a program, project or activity) and then an evaluation is conducted to determine the impact of the intervention. Rog takes this a step further by suggesting evaluation should include the decision-making context, the broader environment, and consider how the listed dimension might apply to each of the contexts. This will often yield richer evaluation findings than the standard linear approach.
Figure 2: Areas of Context That Affect Evaluation Practice (Rog, 2012)
A good evaluator will spend time analysing and reflecting on the different contexts and dimensions at work while working through the agreed methodology for your evaluation. One aspect of doing this well is having an interdisciplinary and diverse team who bring their own skills, experience, perspectives and lens to ensure any bias and groupthink are minimised.
We aim to provide our stakeholders with engaging and relevant content and will expand on many future topics. We hope that through this blog, we can provide a bit of education to our clients, partners, new evaluators and community, while advocating and sharing best practice and achievements across the industry. We look forward to sharing some successful case studies with you as well.
References
Merriam-Webster. (n.d.). Evaluation. In Merriam-Webster.com dictionary. Retrieved
May 13, 2024, from https://www.merriam-webster.com/dictionary/evaluation
OECD (2019), Better Criteria for Better Evaluation: Revised Evaluation Criteria Definitions and Principles for Use, OECD Publishing, Paris, https://doi.org/10.1787/15a9c26b-en.
OECD (2021), Applying Evaluation Criteria Thoughtfully, OECD Publishing, Paris, https://doi.org/10.1787/543e84ed-en.
Rog, D. J. (2012). When background becomes foreground: Toward context-sensitive evaluation practice. In D. J. Rog, J. L. Fitzpatrick, & R. F. Conner (Eds.), Context: A framework for its influence on evaluation practice. New Directions for Evaluation, 135,25–40.