Our evaluation work

Evaluation Projects

Here is a selection of published evaluation projects completed by Pragmatica, in collaboration with others.

Talanoa Ako (PowerUP)

Talanoa Ako (previously known as Pacific PowerUP) is an education programme for Pacific parents. Since 2013, it has helped Pacific parents and communities build their educational knowledge to support their children’s learning. The PowerUP model of engagement is strengths-based. It encourages parents, families and communities to take ownership of the programme.

From 2016 to 2019, PowerUP families shared their stories through the Guided Talanoa Series. 

The evaluations can be found here:

       PowerUP Evaluation 2016
       PowerUp Evaluation 2017
       PowerUP Summative Evaluation 2018
       
PowerUP Summative Evaluation 2019

The PowerUP case studies (2016-2019) can be found here:

Envirolink Evaluation

The Ministry of Business, Innovation and Employment commissioned us to review the Envirolink scheme and inform them of:

  • how well the scheme operates, and
  • whether it achieved its intended outcomes and provided value for money.

This was the first review of the scheme since it commenced in 2005.

The results of the review were positive and we rated the scheme very good overall. The review also provided evidence that the scheme makes a worthwhile and valuable contribution in supporting eligible councils to engage with and use environmental science and support uptake by users. The report’s recommendations will inform future funding decisions relating to Envirolink.

Regional Growth Programme Evaluation

The Regional Growth Programme (RGP) Evaluation was commissioned by MBIE and MPI. It assessed whether the RGP worked as intended. The evaluation reviewed some aspects of the RGP implementation, systems and processes. It assessed the value of the results so far.

This evaluation focused on ways government agencies worked with each other. It also considered ways central agencies liaised with the regional stakeholders and with Māori in the regions. It was not an evaluation of regional economic progress. Lessons learned at central, regional and project levels may inform future regional initiatives.

The evaluation team included: Judy Oakden, Kataraina Pipi, Kellie Spee, Michelle Moss, Roxanne Smith, and Julian King.

There are four documents linked to this evaluation:

The Regional Growth Programme Evaluation Report (103 pages) – full evaluation findings and methodology.

Three infographics which provide:

Sustainable Farming Fund Evaluation

The Evaluation of the Sustainable Farming Fund (SFF) provided an independent, formal assessment confirming the SFF’s value. It also provided information to help ensure the SFF is well-positioned for the future by aligning more closely with Government objectives.

Judy Oakden led the evaluation of the Sustainable Farming Fund (SFF) with team members; Julian King and Dr Will Allen. 

Three Case Studies: a companion document to the SFF Evaluation(46 pages):

  • Protecting the sustainability of New Zealand vineyards
  • Top of the South: Setting an example for sustainable water quality
  • Sustainable development and podocarp restoration on Tuawhenua lands.

Evaluation of stakeholder perceptions of the implementation of the Waste Minimisation Act

The evaluation of stakeholder perceptions of the implementation of the Waste Minimisation Act, looked into waste stakeholders’ perceptions of the early implementation phase of the Waste Minimisation Act, and the short-term outcomes (2009‒2010). This baseline evaluation of stakeholder perceptions was undertaken by the Ministry for the Environment with consultants Judy Oakden and Kate McKegg of the Kinnect Group in late 2010. The evaluation used ‘a mix of focus groups, key informant interviews and an online survey of 325 stakeholders.

Evaluation Rubrics

Pragmatica is well known for using rubrics in evaluation, having used them for over a decade. Judy regularly mentors others in their use. This section includes many of the key publications and presentations developed over time, in collaboration with others.

Use of evaluation rubrics

Understanding the components of evaluative rubrics – new thoughts

In this e-book, Judy Oakden explores the different ways evaluative rubrics can be constructed from three basic components:

  • key aspects of performance
  • levels of performance
  • importance of each aspect of performance

Here she shows some alternative ways she has combined the components in her own practice. She discusses the benefits and challenges of each approach.

Evaluation building blocks: a guide

With colleagues from Kinnect Group we published an e-book on the use of rubrics in evaluation. This free downloadable guide provides a simple nine- step process for planning and undertaking an evaluation. The ideas in this book were developed over ten years of collaboration.

Release Announcement: Evaluation Building Blocks – A Guide

This e-book was recommended reading by Better Evaluation. Better Evaluation is a website which collects best evaluation practice from around the globe.  It aims to build “evaluation practice, evaluation capacity strengthening and research and development in evaluation”.

Better Evaluation reviewer, Patricia Rogers, commented
[The guide} is particularly strong [for developing] a framework to assess performance. [It]… has detailed examples of using global scales (rubrics) to synthesise both quantitative and qualitative data... [It helps] to avoid the common problems caused by replying only on Key Performance Indicators and targets.... I’d especially recommend it in terms of developing a framework for evaluating performance.

Chapter on evaluation in "Social science research in New Zealand: An Introduction"

“Chapter 13 Evaluation” contributed to the textbook “Social science research in New Zealand: An introduction” Edited by Martin Tolich and Carl Davidson., Published by Auckland University Press

Social science research in New Zealand: An introduction (2018) is a textbook that is positioned as “the definitive introduction for students and practitioners undertaking social research in New Zealand”.  Judy Oakden and Julian King contributed a chapter on evaluation which described the evaluation approach used for the Evaluation of the Sustainable Farming Fund. An important part of this chapter was a description of the use of rubrics throughout the project.

The editors observed at the start of this chapter:

“This is an important chapter because evaluation is a discipline that is often misunderstood both among other researchers and the users or research outputs. Evaluation differs from other research approaches because it focusses on understanding the value of something. … Much of what researchers think of as ‘leading-edge’ practices have come from evaluation, and the authors capture much of that excitement and innovation here”.

For more information, please click here

Evaluation rubrics: how to ensure transparent and clear assessment that respects diverse lines of evidence

This is a practice example which is part of Judy Oakden’s early writing on how evaluators can use rubrics to make evaluative judgements. It details how the use of rubrics supports robust data collection and frames the analysis and reporting. Judy Oakden wrote this detailed account of the process as part of the first Better Evaluation writeshop process, led by Irene Guijt.

Rubrics: A Method for Surfacing Values and Improving the Credibility of Evaluation; Journal of MultiDisciplinary Evaluation

This is a practice-based article by the Kinnect Group members (Julian King, Kate McKegg, Judy Oakden, Nan Wehipeihana). It shares the Group’s learnings on the use of evaluative rubrics to deal with the challenge of surfacing values and improving the credibility of evaluation. Evaluative rubrics enable values to be dealt with in a more transparent way. In the Group’s experience, when evaluators and evaluation stakeholders get clearer about values, evaluative judgments become more credible and warrantable.

Providing mentoring support to develop rubrics

To develop and use rubrics on a large complex evaluation of the Food Systems Innovation (FSI) Initiative, team members sought mentoring from Pragmatica. This practice note, by Samantha Stone-Jovicich from CSIRO, describes the steps she went through and the challenges she overcame to develop rubrics for the FSI Initiative.

Samantha reflected afterwards:

[This was] a very large, complex project where we were trying to design a monitoring, evaluation and learning (MEL) system to capture the dynamics of innovation and its impacts. Judy helped us adapt and tailor a rubrics approach. Her abundance of knowledge and experience, coupled with her collegiality, collaborative nature, and flexibility and creativity, were instrumental to supporting us incorporating a useful and fit-for-purpose rubrics approach into our MEL system