Evaluation of climate policy is rocketing but in an ‘undeveloped and unsystematic’ way, according to new study

Published in the international journal Policy Sciences, a meta-analysis by a team of researchers from across Europe offers the very first systematic cataloging of the emerging patterns of policy evaluation undertaken in different parts of the European Union.

In the last decade or so the politics surrounding the development of new policies has attracted unprecedented attention. Many new targets and policies have been adopted. But a lot less is known about what is being done to check that the resulting policies are actually delivering on their promises.

The findings reveal that a culture of evaluation is emerging: the number of evaluations produced has grown spectacularly in recent years. Data collected for six EU states and for the EU as a whole reveal an eightfold increase in the number of reports produced between 2000 and 2005. This growth, however, is more evident in some states than others. Policy effects in the UK, for example, are much more commonly evaluated than those in Portugal and Poland.

However, the culture of evaluation varies in other important ways. The majority of the 259 evaluations identified and studied also adopt a relatively narrow selection of evaluation tools and lack intensive stakeholder involvement. Crucially, over 80 per cent are uncritical i.e. they take existing policy goals as given. Finally, the majority are also quite narrowly framed, in focusing mainly on the environmental effectiveness and/or cost effectiveness of existing policies.

"Whether climate governance is undertaken through the United Nations or -- as now seems more likely -- via more informal 'pledge and review' type processes, evaluation practices are absolutely crucial for fine-tuning policy interventions and building and sustaining public trust," said joint lead author Prof Andrew Jordan of the Tyndall Centre for Climate Change Research at the University of East Anglia.

"The most striking finding of our analysis is just how undeveloped and unsystematic are most current evaluation practices. Great efforts have been made to inform and understand policy making procedures in Europe, but most policy evaluation remains piecemeal and non-consultative."

As the political pressure on policy makers to describe and explain what is being done to tackle climate change increases, calls will grow for evaluation to be undertaken in a more open and transparent fashion.

"At present, policy systems in Europe seem ill-prepared to rise to that challenge," said Prof Jordan.

The other lead author, Dr Dave Huitema of the Institute for Environmental Studies at the VU University Amsterdam (IVM), said there was a "wide gap between evaluation theory and practice, which suggests that current evaluations underestimate the complexity of climate change issues."

University researchers emerged as the most active policy evaluators in Europe. At present the majority of evaluations (58 per cent) are not commissioned. Policy makers could increase the total evaluation effort by commissioning more evaluations from a wider array of organisations. However, this may not necessarily produce a more active and questioning culture of evaluation. At present, non-commissioned evaluations are twice as likely to question policy goals as commissioned ones. By contrast, parliamentary bodies have produced a relatively large number of critical evaluations. The responsibility for enhancing the quality and the quantity of evaluations is therefore shared.

The research was funded by the EU FP6 ADAM project, which UEA coordinated between 2006 and 2009.