Sense and non-sense in evaluating agri-environmental schemes
What is known about the effects of agri-environmental schemes on farmland biodiversity? Quite a lot, but at the same time not as much as we would like, concludes a new study from the Swedish University of Agricultural Sciences, SLU, Sweden. There is a general lack of robust, well-designed studies and a discussion about method problems and alternative explanations are often missing. In addition, results are regularly exaggerated through incorrect use of causal language, as researchers express causal relationships when the study design does not support it.
Biodiversity is declining, not least in agricultural landscapes. Large parts of the world therefore use agri-environmental schemes (AES) to promote practices that benefit biodiversity and ecosystem services. This includes support for practices such as organic farming, management of semi-natural pastures and flower-rich fields and field edges. In many cases these agri-environment measures possibly have positive effects on biological diversity of agricultural landscapes.
Still, this belief is based mainly on correlative evidence open to alternative explanations. For example, as the choice of sites where agri-environment measures are implemented is based on the free will of farmers, there is always a risk that land entered in agri-environmental agreements will have higher biodiversity from the very beginning. Two such examples could be support for semi-natural pastures and organic farming. Because data on biodiversity are missing from the time before the measures were implemented, we do not know if the difference we see already existed from the beginning.
The entire unit participated in the study
This caveat regarding the evaluation agri-environmental schemes was highlighted in Kleijn and Sutherlands review on the effects of agri-environment schemes on biodiversity already in 2003.
“We were simply curious whether evaluations have improved since then and we gathered the entire Landscape ecology unit at the Department of Ecology to review all studies evaluating AES between 2008 and 2017”, says Tomas Pärt – one of the authors of the paper in Conservation Letters.
Language and methodological problems
The researchers also took the opportunity to examine the language used to communicate research findings – whether results were described in terms of cause and effect or in correlative terms. Last they investigated whether methodological problems and alternative explanations were discussed.
“That is, we examined some of the classic cornerstones of the scientific method and the transparency in scientific papers evaluating the effects of AES on biodiversity and ecosystem services”, says Jonas Josefsson, first author of the paper.
The majority (75%) of the studies had a correlative approach with a comparison of AES vs. control sites only after measures were implemented. Still, the results of these correlative studies were often expressed in terms of causation. To a large extent, the discussion about possible methodological problems and alternative explanations were also avoided. About half of all studies compared organic and conventional agriculture.
“Despite this dismal result, our review still showed some improvement in study designs since 2003”, says Jonas.
In the paper, the researchers suggest several possibilities for improvement for future evaluations. Self-critically, they also examined themselves concerning language use when data has been correlative. –
“We are responsible for some of the most blatant examples, and it was precisely in the pale cast of thought that the seed for this study was sown Because, why use causal language when we know we cannot do so”, says Tomas.
In the article Jonas, Tomas and their colleagues give some explanations.
Read it, it is relatively easy to read, it gives tips on how to better design AES to include evaluation, as well as provide guidelines on how results and method problems can be reported.
Researcher at the department of ecology, Landscape ecology unit, SLU
Professor at the department of ecology, Landscape ecology unit, SLU
Improving scientific rigour in conservation evaluations and a plea deal for transparency on potential biases, Jonas Josefsson et al, Conservation Letters
All authors work or have worked at the Department of Ecology. Also Åke Berg at