New approach to evaluating agricultural development programs

Involving local people in figuring out how to improve their farming and fishing methods provides more lasting and widespread benefits than just introducing new technologies or methods, the researchers showed. The findings are described in the journal Agricultural Systems, in a paper by Boru Douthwaite of the research funding agency WorldFish, based in Malaysia, and Elizabeth Hoffecker, lead researcher at the International Development Innovation Network (IDIN), based at the MIT D-Lab.

Considerable research over the last few decades has shown that bringing about improvements in agricultural systems is a highly complex challenge, with many interrelationships and feedbacks determining how well new methods and devices take hold or provide a real improvement. Yet government agencies as well as research and nonprofit organizations still mostly evaluate the success of their programs using simple metrics that overlook much of this complexity, Hoffecker says.

For three decades, Douthwaite has been studying how these programs work in practice. He says he has often observed a disconnect between the measures agencies use to decide whether a program is working, versus the real effects he saw in some of the communities involved.

For this study, the researchers focused on two quite different examples that help to illustrate these disparities: fishing on lakes and rivers in Zambia, and growing a fiber crop called abaca in the Philippines.

In Zambia, a landlocked country in southern Africa, refrigeration facilities and ice are scarce, and nearly one-third of the fish caught there is lost to spoilage before it ever reaches the market. Many of the fish are currently dried or smoked, but they are vulnerable to insects and rodents during the drying process, and they become brittle and subject to damage during transportation. Reducing such spoilage could both provide financial benefits for the economically struggling fishing community and help alleviate food shortages for consumers.

The Zambian fisheries were facing two related issues, Hoffecker says: "The narrow challenge was to come up with a way to prevent fish from spoiling." But in addressing that challenge, it became apparent that "there was a much bigger challenge, which was overfishing." Though many communities in the region were facing these same challenges, "some of the stakeholders were not working together" to address them, she says. If people had tried to get these groups to work together on the bigger challenge right at the start, she says, "it probably would have failed," because there was so much mistrust between the different communities.

But instead, she said, "they started out working on this technical challenge," of reducing spoilage, "which built relationships that allowed them to tackle the bigger challenge." The participatory research process included meetings of different stakeholders including government officials, non-governmental organizations, researchers, and residents, which were followed by village-level workshops in 10 communities. This resulted in establishing three ongoing working groups to tackle different aspects of the issue: fisheries co-management, establishing cooperatives and other economic associations, and postharvest processing. Among other solutions, the group decided to introduce salting of fish as an improved preservation method.

The overall process led to four significant outcomes, Douthwaite says -- none of which had been planned or anticipated initially and thus might have been missed in an evaluation based just on meeting initial, stated goals. The four outcomes consisted of developing a locally sourced fish-processing method (the salting), developing a value chain for the salted fish from harvest to market, creating working groups that could continue to evaluate and improve innovations in the fishery, and improving relationships among the different groups involved, from the fishermen to the government agencies to the traders and buyers. In the end, this led to a growing consensus about the need for measures to prevent overfishing.

In the other case studied, the Philippine abaca farmers had been facing a virus that threatened to greatly diminish their harvests of the widely used fiber plant, which is the nation's primary source of cordage and paper. With some regions experiencing a 90 percent decline in harvests, the government's initial strategy was to eradicate all the infected plants to curtail the virus' spread. But farmers were wary of efforts to destroy the plants they relied on, especially when there was miscommunication about what exactly was being done.

So when a new, virus-resistant variety of the plant was developed, the local farmers' weren't willing to make the switch, as they considered the new varieties inferior for fiber-making.

Instead of just pressing the farmers to change, the team used a different approach, "enlisting the farmers in a process of experimentation," as Hoffecker describes it. Several hybrid varieties were developed, and the local farmers tested them in their fields. "Because they were involved in the process, they were much more receptive to the results," she says. In fact, many of them came up with their own suggestions for furthering the research, including testing local varieties that seemed to be naturally resistant and trying plantings on different kinds of soils and slopes.

They not only came up with an acceptable resistant variety, but when it turned out there were not enough seeds available, the farmers developed their own strategy for sharing the seeds, requiring those who got the initial seedling allotments to pay back new seedlings into the system for others to plant.

Many development organizations are well aware of these kinds of complexities and of the need for more community involvement and fewer "top-down" aid solutions, Hoffecker says. A problem, though, is that the metrics and results-assessment frameworks used to measure success often leave no room for complex, emergent outcomes. Instead, they typically focus on measuring the extent to which various solutions -- such as a new crop variety, tool, or farming method -- are adopted, equating scale of use with success.

Outcomes associated with how the solutions were developed, such as the creation of greater cooperation among communities or stakeholder groups, and the instilling of local empowerment and problem-solving abilities and motivation, are much harder to measure and typically left to anecdotes rather than rigorously assessed, she says.

This research was designed to help provide a basis for new ways to assess the success of programs that are working towards these types of outcomes. "It's a first step in developing such a model and encouraging others to develop such models," Douthwaite says.

And such assessments are essential, Hoffecker says, for making development interventions more effective and lasting. Some worthy projects, she says, are "not getting funded, because the results are not understood by the donors. Some projects are producing important outcomes, but they're not being seen and appreciated." Hopefully, she says, this new study can begin to address that need.