Wouldn’t it be amazing to have all of the data and information at your fingertips to understand the change you want to see in the world? When so much data is available, and ways of collecting data have become more streamlined, it can be exasperating not to be able to pull together what is needed for the practical job of understanding whether something is making the difference intended.

We have already outlined why trying to understand complex work is difficult, and that using data is not a technical-rational process but rather a relational one – where the process of sense making together can be as important as the data, evidence or feedback you use.

In Chapter 3 we discussed challenges in the culture and in the ways of working with data for most organisations. We suggested the need to go on a data improvement journey. In this chapter we present some practical ways to approach the journey, and tools and techniques to help on the way.

This chapter covers:

• • improving approaches to data, evidence and feedback;

• • how to undertake a data audit;

• • how to fill the gaps understanding;

• • practical data collection techniques;

• • feedback in action.

We will be using examples from some of our clients, including how a research body improved feedback, ways of building on reflective practice, and getting feedback fast.

We hope you will be able to use some of the ideas presented here and apply them to any setting, share them with colleagues and start on a data improvement journey. The message is the same as in Chapter 4 – just get going with your data improvement journey!

A quick reminder of what we mean when we talk about data, evidence and feedback:

We asked some of the organisations and teams we work with to share their insights into the way of working we have set out in this book. This chapter shares their thoughts, resources they find helpful, and other resources we often recommend. We also explain the software OutNav, which we have developed to support the approach.

We would like to acknowledge Richard Darke, Edinburgh & Lothians Greenspace Trust; Julie Gardner, Thistle Foundation; Eleanor MacKillop, Wales Centre for Public Policy; Gill Toms, Bangor University; Catriona Windle, Health All Round; and Alex Perry, Helen Berry, Grace Robertson and Simon Bradstreet from Matter of Focus for their suggestions.

In this chapter we are going to explore what data, evidence and feedback practices are helpful or unhelpful for knowing whether you are making a difference. Initially, we are going to dig into some of the cultures and practices that we know get in the way of having the feedback you need to understand change.

We will talk about the data and evidence that is best suited to understanding complex change. We will draw on our experience of working in different settings to explore data cultures that exist in all organisations. We will think about how these data cultures can help or hinder initiatives to do the right thing, and know they are doing the right thing. We will give some examples and highlight ways of applying this thinking to any specific work, especially making the case for using mixed data that includes qualitative data. We will then lay out what it means to go on a data improvement journey and offer some practical ways of thinking about the data and evidence needs in any specific setting.

In this chapter we set out:

• • good data and evidence for complex change;

• • data cultures and how they affect organisations and initiatives;

• • making the case for qualitative data;

• • getting started on a data improvement journey.

If data, evidence and feedback is something that excites and interests you (and it is definitely something that excites and interests us!), then you will enjoy reading the arguments and issues we take from this chapter forward into Chapter 7, where we look in more detail at practical ways to start on a data and evidence improvement journey. We hope that this chapter will arm you with the arguments you need to make the case for the approaches to data, evidence and feedback that can help you understand if you are making a difference.

What is included when we talk about data, evidence and feedback?

We talk in this chapter, and elsewhere in the book, about data, evidence and feedback. We want to encompass all the ways that any organisation uses information about what it does to understand what difference it makes.

So far, we have shared some helpful tools for developing a clear understanding of the outcomes and impacts that matter to any initiative, expressed in an outcome map with risks and assumptions. We have considered how this lends itself to understanding data, evidence and the feedback needed to improve learning and delivery. You may even have some ideas about how to gather more helpful data in creative ways. The next step is to start thinking about how to assess whether your work is making a difference.

As already highlighted in Chapter 4, we’re not fans of simplistic approaches to assessing outcomes as met or unmet. Instead, we believe that the journey to improving outcomes or maximising impacts is never over. Effective organisations that own their outcomes are always seeking to improve, reflect and learn, to align resources and develop systems and processes to make the best contribution possible. Taking time to systematically analyse and assess progress is a linchpin of this approach. In this chapter we will outline what is required to analyse, assess and track progress towards outcomes or impacts ahead of reporting, and share some simple and accessible ways of thinking that can help even people with no experience of analysis get going.

This chapter covers:

• • getting started with data analysis;

• • a practical framework for assessing and tracking the difference you make;

• • three levels to help shape analysis.

Analysis need not be a scary or specialist process. It is something that we all do every day. The key to getting going with analysing and assessing progress is to harness these everyday sense- making skills that are so important when working with complex initiatives. We hope that this chapter will give you the confidence to do this.

Getting started with data analysis

We believe in the power of starting with the end in mind and having a clear sense of what success looks like for any process. This is the foundation for being able to break a process down and plan for that success. Many of the organisations we work with lack a clear sense of what good analysis looks like and what they need in order to do it.

Being clear about the difference an initiative wants to make and the outcomes or the impacts of the work is the first step in knowing whether it is making a difference. Taking the time to really understand how the work contributes to intended outcomes or impacts brings other benefits too. Getting a clear sense of what matters and the change the work is seeking to make in the world is vital when navigating a complex environment. Having a clear vision for where the work is heading acts as a beacon, providing a firm direction whatever changes around it.

Internationally, public services are dripping with references to outcomes, impacts, outputs, goals and results. Many of the organisations we work with around the world are in a situation where they need to report to or align with multiple outcome frameworks and indicators, sometimes for the same work. This can result in organisations feeling caught in a tangle of outcomes, expected to respond and adapt to the range of approaches taken by their funders, commissioners, policy leads and regulators. Being able to navigate around this complex outcomes and impacts landscape is becoming a key competency for anyone leading public services where they are expected to work towards, report on or be evaluated against outcomes or impacts. This is what we call ‘owning your outcomes or impacts’.

If this picture sounds familiar, then this chapter is for you. Over the coming pages we will provide some accessible ways to understand and operationalise outcomes and impacts in your work and unpack some of the contradictions surrounding the use of outcomes and impacts within wider policy and funding systems. In this chapter we will discuss:

• • navigating the landscape of outcomes and impacts;

• • an introduction to outcome thinking;

• • a new way of working with outcomes;

• • owning your outcomes and impacts: what this means in practice.

While this chapter can’t remove the complexity of your outcomes and impacts landscape, we hope it will give you a vision for what it means to own this and to make you feel more comfortable with what it means for your work. Chapters 5 and 6 provide some practical approaches to help you realise this.

Throughout this book we have set out an approach to outcome and impact evaluation that works for the complex problems and issues that public services face. We have used this approach when working with clients from settings such as health, social work and care, policy, international development, research, and children and adult services. What we have found is that there is more in common across these contrasting sectors and contexts than divides them in relation to outcome and impact evaluation. Whatever the setting, the key challenges remain: how to work well with complex contexts; how to understand and work well with outcomes and impacts; how to get the relevant data, evidence and feedback; and how to embed this in dayto-day work.

In this chapter we will explore some of the issues that often arise when working in this meaningful way with outcome and impact evaluation. We will discuss:

• • organisations and settings using this approach;

• • embedding outcome and impact tracking as a change process;

• • practical tips for your outcome- and impact- evaluation journey;

• • case studies: embedding outcome and impact tracking in organisations.

We hope you will finish this chapter with a sense of what is needed to steer the change process in any particular setting so that you can reap the rewards and benefits of really understanding the difference any work makes.

Organisations and settings using this approach

When we were planning this chapter, we liked the idea of creating a neat diagram that encapsulated all of the types of organisations and settings that find this approach useful and helpful. Once we started mapping that out, it quickly became clear that it wouldn’t be possible, or at least, wouldn’t make things any clearer. So, what is set out below is an overview of the reasons, sectors, approaches and types of work where we know outcome mapping and tracking is a great way to understand, evidence and report on change.

Whether supporting people to move on in their lives, implementing new policy initiatives, providing training or spreading good practice, the evidence shows that the most effective approaches are those that are tailored to the specific context in which the work is done (Cook, 2015). This is something we see in practice every day in our work with organisations, and something that is at the core of understanding the difference they make.

Previous chapters have already outlined how working in complex systems means there are many dynamic and moving parts, and that outcomes or impacts can be unexpected and unpredictable. We discussed in Chapter 2 how the very specific context – the time, place, people, culture, institution and setting – make a huge difference to how things are done and what difference they make. The unpredictability of the COVID-19 pandemic is one example of how context cuts across the delivery of projects, programmes and policies. It’s easy to think of other examples at different scales, such as staff recruitment, local or national policy decisions, buildings and infrastructure, and even the weather, that influence the progress of any initiatives.

In our work, we support people to deliberately embrace the complex context for their work. Rather than ignoring it, trying to work well with the challenges and opportunities of any initiative’s context can help improve delivery and really make a difference to people and communities. It’s also essential for shaping how the initiative will make a difference, and for tracking that difference.

This chapter:

• • discusses why context matters;

• • sets out a practical framework to help understand context;

• • explains why it is important to link context and outcomes with risks and assumptions.

We hope this chapter will help you realise the benefits of embracing the complex context for the work you do and give you some practical tools to help work with this complexity despite the challenges it presents. These practical tools are illustrated with a case study and with examples throughout the chapter.

Thank you for picking up this book. We want to set the scene in this preface by explaining why we wanted to write it; that is, how we met, what we both brought with us and how that was developed through working together on one of our early projects. We have used a fictionalised version of the project we introduce here throughout the rest of the book as an example. It explains why it’s often hard to demonstrate impact.

It started when we met through our connections with the University of Edinburgh. Ailsa had been working in the personal outcomes space, leading a master’s programme for health and social care managers and a programme of action research exploring meaningful and measurable outcome recording and assessment. Sarah was working in the research and evidence to action space, working with organisations to use research and evidence to drive change, and developing understanding through research and practical projects about what evidence use looks like in the real world. What we shared was a commitment to using our skills to make a difference to the problems we saw organisations facing, and to bringing robust academic thinking to very practical issues. We have come to consider ourselves practical academics – pracademics!

In particular, Ailsa was acutely aware that organisations in health and social care were troubled by the issues of meaningfully assessing outcomes, and Sarah had been experimenting with contribution analysis as a way of understanding research impact and wider change processes. Once we started talking about bringing these two issues together, and trying it out with some of our projects, we could really see the potential. We started working together in 2014 and set up our company, Matter of Focus, in 2017 as a vehicle to develop the approach, bring it to more people and create tools to make it genuinely practical for organisations to use. We want to tell you about one of our early projects together as a way of illustrating the issues that we help teams to face.

In the first year of our company, we worked with a local wellbeing service that assisted people through a personal outcomes approach – defining with people what was important to them and encouraging them towards their own goals.

Do you really know if your organisation, project, programme, service or initiative is making a difference? And are you able to confidently share the story of the difference you’re making with funders, commissioners and other stakeholders? We suspect that if you are reading this book, the answers to these questions might not be as strong as you would like. You are not alone. This is an issue that has taxed governments, philanthropists, public service organisations big and small and evaluators all over the world for many years.

We engage with teams who strongly believe in the work they do and know implicitly that it does make a difference. However, when they need to report on the difference they’re making, they struggle to bring together and evidence a story that paints a realistic picture of their contribution to change.

If you work in public services in the public or third sector and this sounds familiar, then this book is for you.

This chapter explores why it is so hard to know if projects, programmes, partnerships and organisations are making a difference, and what is needed to go on the journey to be able to make evidenced claims that really demonstrate impact.

Specifically, this chapter covers:

• • why it is so hard to know the difference you are making;

• • the benefits of knowing the difference you are making;

• • what you need to go on this journey; and

• • our approach to understanding if you are making a difference.

By reading this chapter we hope you will understand why we are writing this book, and what we are going to cover in the rest of the chapters, including an orientation to where you might focus your reading, depending on what you want to know.

Why it is so hard to know the difference you are making

Traditionally, public service systems have been focused on activity and outputs, rather than process and outcomes. This flows into the data and evidence that are captured. There aren’t many well-developed and embedded ways of understanding the difference an initiative makes. Indeed, there is often disagreement about what is the right approach. As a result, organisations face different demands for different kinds of data, which makes it hard to commit to any one approach.

Working with initiatives that are trying to tackle some of the most difficult problems society faces can be challenging. As we have already discussed, when working with people – whether empowering, educating, inspiring, supporting or collaborating with them – initiatives often have a sense of what they are trying to do, and sometimes one or more outcome they are working towards. These outcomes might come from funders, the organisational mission, the Sustainable Development Goals or national frameworks, and, as we said in Chapter 4, it is easy to get caught in a tangle of outcomes.

A meaningful connection between activities and outcomes is often vague or unexplored – but this magic ‘how’ change happens is essential for driving programmes and for understanding their impact.

Outcome or impact maps are the core of the approach in this book, as the building block for setting out, understanding, learning about and evidencing change. This is one type of ‘theory of change’ approach which is common for understanding change in complex systems. We believe that process- driven theories of change are most effective for public services, and in this chapter we set out our version of this – outcome or impact mapping – and illustrate how it can be used at different levels.

This chapter discusses:

• • thinking about whether and how something makes a difference: an introduction to theories of change and outcome mapping;

• • the key components of an outcome or impact maps;

• • what is different about this approach;

• • the role of pathways in understanding change;

• • outcome mapping at different levels: project, programme, organisation or partnership;

• • bringing an outcome or impact map to life with risks and assumptions.

This chapter builds on the concepts introduced earlier in this book – especially Chapter 4 on working with outcomes and clarifying your contribution – but also with the challenge of making services more fit for the complex world of delivery. We hope this chapter will give you a very practical way of developing a vision for the change you are working towards and opening up how you think that change will happen.

Building a strong contribution story that explains and can evidence the difference your work makes to people, policy, practice or communities is not an easy task. Teams and organisations tell us how challenging it is to report on what difference their work has made and there is often a gap between the valuable work they do and the story they’re able to tell about it. This is compounded because there are probably multiple stories they want to tell, to and with different people, for different purposes. Many organisations are great at explaining the richness of the work they do to support, empower or inspire people, but find it harder to link that to what difference it has made. Of course, our outcome mapping and tracking approach really helps to frame the process and make those links between activities and outcomes or impacts.

In this chapter we bring together our learning from working with many organisations and clients, as well as our own work in outcome and impact reporting. Thinking and using the language of ‘contribution’ – that is what contribution an initiative makes – can help make claims about this much clearer and stronger.

This chapter will consider:

• • reporting in an outcome and impact-focused way;

• • reporting for different audiences and purposes;

• • using an outcome map as a framework for reporting;

• • practical tips for great reporting;

• • taking reporting forward.

We hope by reading this chapter you will understand how outcome- focused reporting can help you highlight the difference your work makes. This is the final part of the process we have described in Chapters 5 to 8, and a crucial one in making the case for your work. This chapter also aims to help think about different kinds of reporting for different purposes: reporting as a learning and improvement tool, as well as for sharing more widely.

Reporting in an outcome- and impact- focused way

There are many benefits of reporting in an outcome- and impactfocused way. While there is a lot of interest in the things that organisations do, it’s only possible to understand their importance if the question of what difference they made is also clearly part of the story.

You are probably reading this book because you care about making things better for people or communities – through providing services, improvement projects and programmes, policy influencing, systems change or practice influencing. What is interesting and what these have in common is that they are not simple change mechanisms – they sit in complex webs of services, programmes, policies and practices, and are not under the control of any one organisation. Instead, change happens through a web of interlinked, overlapping approaches and services.

For initiatives that want to understand the change they make – and make the best change possible – they need to engage with this complexity, with thinking, approaches and tools that embrace and suit the world in which they operate. But working in a way that is compatible with this complexity can be challenging and doesn’t always sit well with current ways of doing things.

In this chapter we dig into some of the main issues we have seen people grappling with when trying to do this difficult work.

This chapter considers:

• • a shift in thinking for complexity-informed approaches;

• • working with complex systems;

• • people at the heart of complex change;

• • evaluating in complexity;

• • how our approach helps to work with complexity.

We hope in this chapter you will find familiar ideas and new ones, and it will help orientate your thinking to the challenges of delivering, developing and influencing public services in the 21st century. We want you to feel informed about the importance of complexity for delivering people-based work and have ammunition to argue for complexity-informed ways of monitoring and evaluating for your own change programmes.

A shift in thinking for complexity-informed approaches

Around the world, governments and organisations (for example, the Australian Government, 2019; The Manchester Partnership, 2021) are bringing new thinking to understand the problems that public services are tackling.

The way this is expressed across settings has differences and nuances, but this shift in thinking usually includes the following:

• • A recognition that people’s problems are complex and messy with lots of interdependencies that are unlikely to be solved with single initiatives and programmes.

• • An understanding that people are best placed to know what they need and what action to take, and that building on an asset-based model rather than a deficit one is more productive and empowering.