Increasing the impact of evidence-based policy

Oct 26, 2020

Over past decades, the term, ‘evidence-based policy’ has come to dominate policy discussions. In education, as in all policy fields, the extent to which a policy is evidence-based has become the measure of how we discuss the quality of a policy and whether it is to be supported and maintained over time.

In many respects this is a good thing – much better than a policy that contradicts the evidence! But the growth of evidence-based policy in education has not translated into impact. Virtually every system I see has evidence-based policies but only a few can show sustained improvement in schools and classrooms. System leaders I speak to regularly lament the lack of impact of their reforms, and their struggles to have meaningful discussions about evidence-based policy that doesn’t have an impact. Meanwhile, policies keep getting ticks of approval for being evidence-based, even though they don’t shift classroom outcomes. This is a problem. Actually, I think it is a huge problem.

To illustrate my point, consider these questions:

    1. Is a policy evidence-based if it cites educational research but isn’t based on evidence about what is happening in a system’s schools and classrooms?
    2. Is a policy evidence-based if its impact on schools and classrooms is not subsequently evaluated?
    3. What happens when evidence-based policy is evaluated and is shown to have minimal or no impact on student performance? Is it still evidence-based?
    4. At what point does a policy that fails to have an impact stop being referred to as evidence-based? After one year? What about after five?

These are really important policy questions that are very difficult to talk about. Questioning evidence-based policy – even when it is not working – can feel like heresy. But that means we are more likely to give a tick to policy that is failing kids.

Changing the conversation

I want to change the way we talk about evidence-based policy. I am not suggesting we get rid of it – that would be nuts – but we need to be much clearer about what we mean by it. In my mind, the whole point behind the push for evidence-based policy is to improve how policy is developed and improved over time. We therefore need to think more precisely and clearly on the links between the evidence base and policy. Effective policy development in education should not only incorporate the best research, it also needs to be built on evidence from the system in which the policy is applied. In particular, three questions need to be answered:

    1. What is the evidence from schools & classrooms of the problems we are trying to solve in our system?
    2. What does research say is the best way to solve these problems?
    3. What is the evidence we will use to monitor and evaluate whether the policy is solving these problems?

If you are a system leader, questions 1 and 3 focus on evidence from your system – what is happening in schools and classrooms. Question 1 provides evidence of the problems you are trying to solve or improve. In this sense, it is the foundational evidence on which policy is developed. Question 3 examines the evidence to know whether or not that policy is working.

Question 2 focuses on what educational research says about the problems you are trying to solve in your system. This question is how most people in education normally talk about evidence-based policy. When people ask if a policy is any good, the answer regularly focuses on research, citing the leading research on which the policy is based. The problem with this approach is that it separates the discussion of a policy’s quality from the schools and classrooms in which it is applied. It also creates difficulties for policy makers when they are confronted with what to do with evidence-based policies that don’t have an impact.

Let’s take a hypothetical example: a policy to improve school leadership has a number of training programs (an aspiring leaders program, a leading teacher mathematics program, and so on) that have shown no impact in a system for a number of years. When they were first developed, the policy and the leadership programs were considered to be evidence-based – meaning that they reflected research on the positive impact of instructional leaders and of effective training programs.

But after multiple years of no impact on schools and classrooms, system leaders have to decide what to do with the leadership policy. Should they keep it going, or cancel it, and the programs beneath it? Should they refine and make further investments?

If the system makes its decision based only on question 2 above, it will decide that the policy is still evidence-based. After all, the research still holds. By now the policy and programs will have beneficiaries and advocates, and since it is very hard these days to say that an evidence-based policy should be cancelled or even overhauled, almost invariably the case for maintaining the policy will win the day. So a leadership policy that has had no impact for several years survives, and students suffer.

This is a common scenario. I hear policy makers in many systems express frustration about it. And it won’t change, unless we change the way we talk about evidence-based policy.

A simple process

How can we have better policy discussions that reflect both the research and the policy’s impact on schools and classrooms? Below I set out a simple template that system leaders could use to help change the way we talk about evidence-based policy.

The template is in no sense a comprehensive tool for policy evaluation or development. That is not its objective. It simply identifies the evidence that will then feed into policy development and policy evaluation.

The template is built around the three questions above: what is the evidence from schools and classrooms of the problem we are trying to solve? What does research say is the best way to solve these problems? And what evidence will we use to monitor and evaluate whether we are solving it?

The template requires system leaders to identify evidence for each of these questions in the areas that most impact school performance: curriculum and instructional materials, teaching practices, and assessment practices being used in schools and classrooms.

These issues obviously intersect in schools and also in policy responses. So the evidence might cut across these categories. But I include them separately in the template as I think it’s important to get more, rather than less, specific about the evidence.

Evidence to feed into policy development and evaluation

Here is how a system leader might use the template. In column 1 we write down the evidence of what is currently happening in schools and classrooms in our system – the problems our particular policy is trying to solve. The specific questions in each row of column 1 illustrate the specific evidence to be collected.

Once we have identified evidence of the problems we are trying to solve, we can look at the research base for how best to overcome them. We record this in column 2. Critically, we need to identify the research about the specific problem identified in Column 1. We don’t want generic research about, for example, effective teachers and school leaders. If there is a specific problem in teaching fractions, the evidence has to be about how to teach fractions.

The last column of the template requires identifying the evidence to be collected from schools and classrooms in order to monitor and evaluate the impact of a policy. It is important that column 3 links to the problems identified in column 1; in other words, the system needs to evaluate and monitor changes in the specific problems we are trying to solve.

To repeat, this is a not a comprehensive policy development template. For example, there is no space to identify initiatives or implementation planning. This is all about the evidence to feed into policy development and evaluation to improve how we discuss – and formulate – evidence-based policy.

To illustrate what this approach can look like, I have completed below the template for a K-6 mathematics policy. Although I have included common problems I have seen in K-6 mathematics, it is only an illustration – the whole point of this exercise is that it needs to be done in your system.

Putting K-6 maths in the top left-hand corner of the template, I start with column 1: What is the evidence from schools and classrooms of the problems we are trying to solve in our system? This requires extensive data collection: school visits, surveys, case studies and so on. Importantly, it should also involve tangible evidence of what students are being taught, the tasks they complete, and tangible evidence of their work and learning. In our experience working with systems, this evidence is fundamental to effective policy development.

The template, by identifying specific evidence of different parts of K-6 mathematics teaching and learning, will enable you to target the actual problems faced by teachers and school leaders. I won’t discuss each box as the template is only meant to be illustrative (I will talk about K-6 mathematics in another blog post). But it’s worth emphasising a few things.

In row 1, I have put evidence about the K-6 mathematics curriculum and instructional materials currently being used in schools. It includes problems I have seen in K-6 mathematics: wide variation in the quality of curricular and instructional resources and teacher uncertainty in their curricular choices. I have also highlighted a problem of low-quality measurement tasks given to students. I have seen evidence of this problem in systems we have worked in but you will need to identify the evidence of specific issues in your system.

In the (horizontal) row on assessment practices, I have included evidence about system-wide or standardised assessment results in columns 1 and 3. Often a fall in standardised assessments is the impetus for reform, so it is often the first work done in policy development. Including it in row 3 also highlights evidence on the links between assessment that is standardised, system-wide or school-based. Often the three don’t align or effectively feed into decisions of curricular and pedagogical practice in schools.

Column 2 is about what research says is the best way to solve the problems identified in your schools and classrooms. The research needs to directly address these problems (column 1). Column 1 identifies problems with teaching measurement and geometry. For example, the measurement tasks students undertake in class can sometimes have them using a measuring device but not understanding how it works; students can count the numbers on a ruler but not properly understand what this means for measuring the length of an object. So the research cited in column 2 needs to be about addressing this problem and improving teaching measurement and geometry. For example, a range of specific tasks are needed for students to properly understand measurement that then feeds into understanding different shapes in geometry. This process pushes policy away from generic statements about developing effective teachers and great leaders, and instead links specific research to the specific curricular, pedagogical, and assessment problems in classrooms in your system.

Column 3 identifies the evidence used to monitor and evaluate the quality of curriculum and instructional materials used in schools. The evidence I have included focuses on the Use of high-quality curriculum & instructional materials. While more detail is obviously needed to properly evaluate if this is happening in your system, it identifies monitoring and evaluation evidence that corresponds both to the problem identified in column 1 and what the research says about this problem in column 2. In other words, we have clear line of sight between identifying a problem in classrooms, what the research says about solving the problem, and monitoring and evaluating its impact. In column 3, I have also included improvement in student assessments to complement other monitoring and evaluation evidence.

I don’t pretend that a short template like this will suddenly improve policy. I am trying to change the way we talk about evidence-based policy to make it more effective – because it focuses both on the research behind the policy and the problems in schools and classrooms the policy is trying to solve.

In subsequent blogposts I will work through some more of the detail of this process and how we can make evidence-based policy more impactful. My hope is that we develop a new narrative for evidence-based policy, one that recognises current problems and leads to better (and easier) policy decision-making that produces better outcomes for students.

waves

Subscribe to Ben Jensen's Blog