comment 0

A conversation about policy experiments


The search for ways to design policies that help to solve societal problems is continuously evolving. The traditional evidence-based approach to policy making involves generating research results and using these to inform policy decisions.

This blog post is based on an interview with Mikko Annala, Head of Governance Innovation at Demos Helsinki. It was originally posted on the Think Tank Initiative Blog.

Road smaller

Think tanks around the world play an essential role in the knowledge ecosystem: they generate creative approaches grounded in local realities and use this evidence to inform policy-making. Through my experience working at the Knowledge Sector Initiative in Indonesia I have seen how research and advocacy by think tanks can inform and change policy. For example, we documented how the Institute for Research and Empowerment made a crucial contribution to the passing in 2013 of the new Village Law. In addition, SurveyMETER, a think tank based in Yogyakarta, conducted research which, in the Municipality of Balikpapan, informed plans to invest in better urban infrastructures for the elderly. As a result, I’ve seen the impact that think tanks in the global South are making.

I’m now living in Finland and am interested in the innovative approach to policy experiments they’re adopting here, as this may be of interest to others engaged in similar work. Policy experiments take a different approach: they test solutions and then generate evidence about what works and what does not work, which can inform policies and government programmes. But, what are the key factors that characterise an experimental culture in government? Are policy experiments something new in policymaking? To learn more, I got in touch with Mikko Annala, Head of Governance Innovation at Demos Helsinki, a Finnish independent think tank which is involved, among other things, in policy experiments with the Finnish government as well as overseas.

Arnaldo Pellini: what are policy experiments?

Mikko Annala: “Policy experiments are usually defined as initiatives that help ministries and government departments test new ways to solve policy problems within a limited scale, and within a set timeframe.

I find this definition limiting. It presents experimentation merely as a tool for discovering and testing new solutions. I think that there is more to it.

In our policy experiment work with the Prime Minister’s Office, we have realised that experimentation can also contribute to building trust between citizens and policy-makers. The dialogue and debate around the design of the experiments, the alignment with the government policy goals, the trade-offs between different solutions, and so on, helps to share different perspectives and points of views. In turn, this helps to strengthen trust towards policy institutions.  One example is TAIKA – Taidetta kansalle (MAGIC – Art for the People), a policy experiment that seeks to develop new human-centric ways for connecting arts to social and welfare services in Finland and see this can promote better health and well-being.

My definition of policy experiments includes:

  • Uncertainty of outcomes.
  • The courage to find new, and sometimes even radical, solutions to social problems.
  • Building learning within experiments by setting clear objectives and measurements.
  • A limited timeframe and scale.
  • Open and active inclusion of different stakeholders from the design stage.”

Are policy experiments new?

“Policy experiments are not new, but a lot has been happening in this field lately.

During the 1930s, President Roosevelt led a policy experimentation boom in the United States. But in my opinion, it is only in the last 10 years or so that we have seen a new wave of policy experimentation emerging across the world.

Today the term ‘experimentation’ is not only used by experts, but also by politicians, policy-makers and civil servants.

This is not a coincidence. In my opinion, there has never been a greater need for experimentation in public policy as today. Change in our societies and economies is happening at a fast pace. The complexity of the world is rapidly increasing and will continue to do so.

There is an increasing need to develop faster and more reliable learning processes to solve problems and, at the same time, strengthen the trust between citizen and policy institutions. Policy experiments are a great way to do this.”

Experiments will sometimes fail, failure is not always welcome

“Policy experiments require certain capabilities within government departments and, importantly, a culture that supports policy experimentation. When the culture is not there, the shift can be difficult.

This is why it is important to document and communicate good experimental practices. There are three elements which are a key to experimental culture:

1) Incentives. For example, requiring ministries to commit and use a certain part of their budgets to conduct evaluations of policies and programmes enables, in my experience, a greater openness towards experimental methods within a bureaucracy.

2) Leadership. An experimental culture, requires a high level of leadership and opening up of public discussions about the benefits of experimentation, as well as the value of failing and learning.

3) Results. Policy experiments have to try to show success pretty fast, which is easier if flexibility, adaptation and a limited scale is part of the design.”

You seem to take a problem driven approach in your report to the Prime Minister’s Office, Design for Government: Human-centric governance through experiments. Is that true?

“The approach we suggest in the paper starts from understanding the problem. But let me explain how we suggest going about it.

First, review the research and literature around the problem and map how much is already known.

Second, we research interesting practices that seems to address, and potentially solve, the same challenge or parts of it.

Third, and perhaps most importantly, we build a dialogue with individuals and stakeholders related to the problem.

In our experience, (co)defining the problem among stakeholders increases trust towards the experimentation process and the risks it entails. It also reduces the complexity of problems, because the experts planning the experiments don’t have to do so much guesswork and can actually learn a lot about the problem through interaction with the stakeholders.

Having said this, I also think that one should not fall in love with the problem. After a certain level of elaboration, a team has to come up with potential solutions and start testing whether they work or not!”

What types of evidence do experiments produce?

“Without measurements, we can’t tell what works and what doesn’t. In other words, we can’t learn from what we are doing.

Policy experiments should always try to generate different types of evidence and in my experience, it pays off to remain patient. With sufficient sample sizes, for example, randomised control trials can provide useful evidence through pre- and post-experiment analysis.

At the same time, some policy experiments may have to deliver results and evidence quickly. In those cases, policy experiments need to develop ‘quick and dirty’ learning methods that enable them to learn without major time and resource investment.”

Policy experiments are an approach to test policy solutions and, at the same time, generate evidence that can inform policy decisions around solutions to very policy problems. Being an approach, rather than a model, they can be applied to different contexts and policy issues in a very flexible way. The experiences in Finland, which I discussed with Mikko Annala, can provide ideas and suggestions for other countries, though, as always, the answers and solutions have to be found locally.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s