Sense and non-sense in evaluating agri-environmental schemes

Biodiversity is declining, not least in agricultural landscapes. Large parts of the world therefore use agri-environmental schemes (AES) to promote practices that benefit biodiversity and ecosystem services. This includes support for practices such as organic farming, management of semi-natural pastures and flower-rich fields and field edges. In many cases these agri-environment measures possibly have positive effects on biological diversity of agricultural landscapes.

Still, this belief is based mainly on correlative evidence open to alternative explanations. For example, as the choice of sites where agri-environment measures are implemented is based on the free will of farmers, there is always a risk that land entered in agri-environmental agreements will have higher biodiversity from the very beginning. Two such examples could be support for semi-natural pastures and organic farming. Because data on biodiversity are missing from the time before the measures were implemented, we do not know if the difference we see already existed from the beginning.

The entire unit participated in the study

This caveat regarding the evaluation agri-environmental schemes was highlighted in Kleijn and Sutherlands review on the effects of agri-environment schemes on biodiversity already in 2003.

“We were simply curious whether evaluations have improved since then and we gathered the entire Landscape ecology unit at the Department of Ecology to review all studies evaluating AES between 2008 and 2017”, says Tomas Pärt – one of the authors of the paper in Conservation Letters.

Language and methodological problems

The researchers also took the opportunity to examine the language used to communicate research findings – whether results were described in terms of cause and effect or in correlative terms. Last they investigated whether methodological problems and alternative explanations were discussed.

“That is, we examined some of the classic cornerstones of the scientific method and the transparency in scientific papers evaluating the effects of AES on biodiversity and ecosystem services”, says Jonas Josefsson, first author of the paper.

The majority (75%) of the studies had a correlative approach with a comparison of AES vs. control sites only after measures were implemented. Still, the results of these correlative studies were often expressed in terms of causation. To a large extent, the discussion about possible methodological problems and alternative explanations were also avoided. About half of all studies compared organic and conventional agriculture.

“Despite this dismal result, our review still showed some improvement in study designs since 2003”, says Jonas.

Possible improvements

In the paper, the researchers suggest several possibilities for improvement for future evaluations. Self-critically, they also examined themselves concerning language use when data has been correlative. –

“We are responsible for some of the most blatant examples, and it was precisely in the pale cast of thought that the seed for this study was sown Because, why use causal language when we know we cannot do so”, says Tomas.

In the article Jonas, Tomas and their colleagues give some explanations.

Read it, it is relatively easy to read, it gives tips on how to better design AES to include evaluation, as well as provide guidelines on how results and method problems can be reported.

Contact

Jonas Josefsson 
Researcher at the department of ecology, Landscape ecology unit, SLU

Tomas Pärt
Professor at the department of ecology, Landscape ecology unit, SLU

Scientific article

Improving scientific rigour in conservation evaluations and a plea deal for transparency on potential biases, Jonas Josefsson et al, Conservation Letters

All authors work or have worked at the Department of Ecology. Also Åke Berg at