Te arotake i te panoni pūnaha
Evaluating systems change
While having an impact on child and whānau wellbeing drives PMP, measuring, or evaluating any difference it might be making is complex. The impact of systems change projects can be challenging to detect, hard to attribute to the work being undertaken, and it can take time before the results show. Moreover, ngā Kaikōkiri (the groups PMP supports) have told us reporting requirements can be onerous, particularly for those receiving funding from different sources as they often vary between funders.
Not only do overly arduous requirements take groups away from their work, they can result in ngā Kaikōkiri feeling pressure to overstate outcomes or pick off “low hanging fruit” to demonstrate progress in a contestable funding environment.
An emphasis on producing short term outcomes can result in findings which have limited value to either ngā Kaikōkiri or the communities they are working with. Ironically, in this way evaluation can actually hinder progress towards change.
Our goal is to ensure evaluation is useful to ngā Kaikokiri and maximises systems change to achieve our vision.
So how can evaluation be helpful rather than a hindrance?
At PMP we are trying a mixture of evaluative approaches which we hope will both drive and capture change.
As systems change is challenging and there are few roadmaps on how to do the work, we asked ngā Kaikōkiri to develop a theory of change, or best (informed!) guess as to how they believe the changes they are striving towards might come about. They are then encouraged to take a developmental evaluative approach, tackling the conditions which they think will lead to change, thinking about what they are doing, what is working, not working, and what they need to adapt. The approach is designed to mitigate risk by driving learning and improvement.
Our PMP theory of change is that initiatives will be more likely to succeed as early, robust, and candid reflections will enable our innovators to change direction quickly. We think this approach might work well in emergent and fast-changing situations which our Kaikōkiri face, as it is highly adaptive.
Several Kaikōkiri have developed their own powerful developmental evaluative frameworks grounded in culture and designed to meet the needs of their organisation and the communities they work with.
Ngā Kaikōkiri with a strong culture of reflective practice and undertaking a developmental evaluative approach are making significant progress.
We are also supporting regular Mahi Tahi events, or get-togethers where our Kaikōkiri and Committee members can share what they are doing, learning and achieving with each other. Ngā Kaikōkiri and Committee members say this is a time they can reflect on their work, and share what they are learning and how they are progressing. This time together is not only growing our collective understanding of systems change, what is being tried, what is working, and whether there are emerging signs of impact; it is helping to “top up the buckets” of those doing the work. Systems change work is hard. Really hard. Ngā Kaikokiri say it is helpful to connect with and learn from others working in this space and facing similar challenges. Some waka have started identifying opportunities to collaborate and go further faster, together.
Our Kaikōkiri and Committee members are highly motivated to reduce child poverty and its impacts. Accountability is critical. Every dollar spent on systems change initiatives, is a dollar that could have been spent directly ameliorating the devastating effects of poverty by supporting programmes and service delivery. The toughest evaluators of our work are those affected by it; those living in poverty. While clearly the Committee wants to see impact, we encourage an evaluative approach in which accountability faces towards the communities ngā Kaikōkiri are working with.
Our evaluative approach, to be effective, relies on Kaikōkiri highly driven to affect change, focused on learning and adapting their practice, and open to sharing failures as well as successes more broadly. It also relies on a highly engaged Committee who are hands-on in the journey, prepared to take the risk that a developmental focus might maximise learning and outcomes thereby creating impact, and who through their engagement with ngā Kaikōkiri, can see what they are doing, trying, learning, and can take a strategic view of how to support them.
While there is early evidence this approach may be contributing to impact – watch this space….
“[PMP are] committed to the social change process much more than you would recognise in many other funders. They want to facilitate or to support your capability in unpicking the challenges and reflecting and evaluating in a way that means you are constantly growing and learning from the work that you are doing. They're less concerned about milestones than they are about your ability to reflect and understand what you're doing and get better at what you're doing, which is a bit special and unique.” (Kaikōkiri, 2021).
Article by Alex Woodley