Although we may not always agree on the specifics – or the application of the concept given its political sensitivities – there is a degree of consensus on the general traits of state fragility. These include, for example, weak capacity to provide basic services, public security and rule of law; inability to manage political conflict; and delegitimization of the state. But this year’s Global Monitoring Report, produced by the World Bank and the International Monetary Fund, threw a new trait into the mix. “Fragile states,” it argued, are characterized by, among other things, a “lack of timely and reliable statistics on the basis of which policies can be formulated.”
It is true that data deficiencies are often associated with conflict, and it is also true that violent settings pose enormous challenges to knowledge generation. Yet, it is in these contexts that the need for reliable evidence is perhaps greater than anywhere else. Fragility and conflict occupy spaces that are highly dynamic and uncertain, and bring with them problems that are irreducibly complex. Finding ways to get at the solutions to some of these problems is a pressing task for those working on security and development in the world’s epicenters of fragility, from the favelas of Rio de Janeiro to the valleys of Helmand. In many cases, understanding the realities of such places and identifying priorities requires a geographical focus on local areas and an analytical focus on the views of local people. So too do assessments of program impact: as noted recently by the UK’s Department for International Development (DFID), people’s perceptions of change often matter as much as the actual change itself.
It is against this backdrop that perception surveys have come to constitute an important element of research methodologies in conflict-affected situations. But just how far do they get us in terms of enriching understanding and influencing policy? A recent Overseas Development Institute (ODI) public event, organized by the Secure Livelihoods Research Consortium (SLRC), set out to try and shed some light on this question, bringing together a panel of experts with experience of conducting perception surveys in hostile environments. What emerged was not a narrow debate on the technicalities of doing perception surveys, but a far broader set of reflections on the nature of evidence-making in conflict-affected situations.
The use of perception surveys in fragile and conflict-affected situations
At their core, perception surveys exist to gather information on people’s views. Typically administered in the Organisation for Economic Co-operation and Development (OECD) countries, but increasingly applied in developing and conflict-affected states, perception surveys are considered useful for exploring difficult-to-measure, subjective issues such as trust, satisfaction and expectations, sometimes at representative levels. In recent years they have been used to investigate service delivery in Bosnia and Herzegovina, policing in (then) southern Sudan, security in urban Haiti, and governance in Afghanistan. Perception surveys are also becoming more and more relevant to the evolving fragile states agenda, the optics of which are shifting from questions of administrative capacity to those around legitimacy, social contracts and state-society relations. Such methods may have a central role to play in monitoring countries’ progress towards achieving the five Peacebuilding and Statebuilding Goals as set out in the recently engineered “New Deal for Engagement in Fragile States.”
How far do perception surveys take us?
It is quite clear that perception surveys have the potential to add value to research and policy-making in conflict-affected situations. At the very least, they can play an important role in improving the size and rigor of the existing evidence base – something which is desperately needed. Indeed, a key message emerging from the ODI public event was that the question is no longer whether to do perception surveys, but how we can do them better.
But there are limits. Perception surveys can only fill in part of the picture, meaning that they must be supplemented by other methods or embedded within triangulation approaches. More fundamentally, doing surveys in violent and volatile contexts generates massive costs and splitting logistical headaches, particularly if they are to be done properly: as George Varughese of The Asia Foundation pointed out, “if you put rubbish in, then you’ll get rubbish out.” In other words, questions that aim to better understand public anxiety surrounding governance are of greater utility than questions that reflect donor curiosity about aid and regime performance and which fail to produce actionable findings. And, when combined with a host of generic survey problems that become more pronounced in settings of war and violence – from high rates of attrition in follow-up panel surveys to the “shades of truth and lies” that accompany respondents’ testimonies – the bounds of how far perception surveys are able to take us become increasingly visible.
So, as The Asia Foundation’s Tom Parks argued, although the need for more and better evidence in fragile and conflict-affected situations is clear, the required “burden of proof” – or standard of evidence – simply cannot be the same as that demanded of their more stable counterparts. In an era when aid programs have to be justified – usually in terms of their value for money– it is becoming increasingly difficult to produce the level of evidence that is expected. Taking this into account, the real question then becomes: does the results-based agenda for fragile states need a rethink?
Richard Mallett is a research officer on the Secure Livelihoods Research Consortium (SLRC), based at the Overseas Development Institute (ODI) – Britain’s leading independent think tank on international development and humanitarian issues. SLRC is a six year global research program exploring livelihoods, basic services and social protection in conflict-affected situations. Richard can be reached at email@example.com.
Cross-posted from the The Asia Foundation.