Person:

Woolcock, Michael

Rising standards for accurately inferring the impact of development projects has not been matched by equivalently rigorous procedures for guiding decisions about whether and how similar results might be expected elsewhere. These 'external validity' concerns are especially pressing for 'complex' development interventions, in which the explicit purpose is often to adapt projects to local contextual realities and where high quality implementation is paramount to success. A basic analytical framework is provided for assessing the external validity of complex development interventions. It argues for deploying case studies to better identify the conditions under which diverse outcomes are observed, focusing in particular on the salience of contextual idiosyncrasies, implementation capabilities and trajectories of change. Upholding the canonical methodological principle that questions should guide methods, not vice versa, is required if a truly rigorous basis for generalizing claims about likely impact across time, groups, contexts and scales of operation is to be discerned for different kinds of development interventions.

The coherence and effectiveness of engagement with the world's 'fragile and conflict-affected states'—beyond ethical imperatives and geo-strategic considerations—turns on answers to two vexing questions. First, on what defensible basis is any given country, at any given historical moment, deemed to be (or not to be) 'fragile'? Second, if a defining characteristic of state fragility is low levels of capability to implement core responsibilities, how can international agencies best support domestic public organizations to acquire capability? The first issue may appear to be a methodological one (wherein more and better data would provide a firmer empirical foundation on which to base key decisions) but any determination, especially of marginal cases, must also be grounded in a correspondingly comprehensive theory of change. Similarly, the optimal response to the second issue may appear to be importing technical and rigorously verified ('best practice') solutions, but in fact it is more likely to require a qualitatively different strategy, one able to experiment with alternative design specifications and adapt in real time to changing contextual realities (thereby iterating towards customized 'best fit' solutions). To this end, an alternative approach to the theory, measurement and practice of engaging with fragile states is outlined, in the spirit of rising concerns across the development community that prevailing strategies have demonstrably reached the limits of their effectiveness.

A defining task of development is enhancing a state’s capability for policy implementation. In most low- income countries, alas, such capabilities seem to be stagnant or declining, in no small part because dominant reform strategies are ill-suited to addressing complex non-technical aspects. This has been recognized for at least six decades – indeed, it was a centerpiece of Albert Hirschman’s understanding of the development process – yet this critique, and the significance of its implications, remain on the margins of scholarship and policy. Why? I consider three options, concluding that, paradoxically, followers of Hirschman’s approach inadequately appreciated that gaining more operational traction for their approach was itself a type of problem requiring their ideas to embark on ‘a long voyage of discovery’, a task best accomplished, in this instance, by building – and tapping into the distinctive insights of – a diverse community of development practitioners.

Evaluations of development projects are conducted to assess their net effectiveness and, by extension, to guide decisions regarding the merits of scaling-up successful projects and/or replicating them elsewhere. The key characteristics of ‘complex’ interventions – numerous face- to-face interactions, high discretion, imposed obligations, pervasive unknowns – rarely fit neatly into standard evaluation protocols, requiring the deployment of a wider array of research methods, tools and theory. The careful use of such ‘mixed methods’ approaches is especially important for discerning the conditions under which ‘successful’ projects of all kinds might be expanded or adopted elsewhere. These claims, and the practical implications to which they give rise, draw on an array of recent evaluations in different sectors in development.

Many development agencies and governments now seek to engage directly with local communities, whether as a means to the realization of more familiar goals (infrastructure, healthcare, education) or as an end in itself (promoting greater inclusion, participation, well-being). These same agencies and governments, however, are also under increasing pressure to formally demonstrate that their actions ‘work’ and achieve their goals within relatively short timeframes – expectations which are, for the most part, necessary and desirable. But adequately assessing ‘community-driven’ approaches to development requires the deployment of theory and methods that accommodate their distinctive characteristics: building bridges is a qualitatively different task to building the rule of law and empowering minorities. Moreover, the ‘lessons’ inferred from average treatment effects derived from even the most rigorous assessments of community-driven interventions are likely to translate poorly to different contexts and scales of operation. Some guidance for anticipating and managing these conundrums are provided.

In rich and poor countries alike, a core challenge is building the state’s capability for policy implementation. Delivering high-quality public health and health care – affordably, reliably, at scale, for all – exemplifies this challenge, since doing so requires deftly integrating refined technical skills (surgery), broad logistics management (supply chains, facilities maintenance), adaptive problem solving (curative care) and resolving ideological differences (who pays? who provides?), even as the prevailing health problems themselves only become more diverse, complex and expensive as countries become more prosperous. The current state of state capability in developing countries, however, is demonstrably alarming, with the strains and demands only likely to intensify in the coming decades. Prevailing ‘best practice’ strategies for building implementation capability – copying and scaling putative successes from abroad – are too often part of the problem, while individual training (‘capacity building’) and technological upgrades (e.g., new management information systems) remain necessary but deeply insufficient. An alternative approach is outlined, one centered on building implementation capability by working iteratively to solve problems nominated and prioritized by local actors.

The Middle East and North Africa (MENA) is a rising middle-income region, and its citizens rightly expect quality public services. Yet too often they experience disappointment: students attending local schools are insufficiently prepared for the 21st century economy, and those needing health care too often find that public clinics have no doctors or medicines. Few in positions of authority are held accountable for such shortcomings. This situation both undermines the potential for improvement and heightens people’s unhappiness with the delivery system.

Although dissatisfaction with education and health services is widespread in the MENA region, local successes do exist and offer inspiration. At the Kufor Quod Girls’ Secondary School in the rural West Bank, for example, Ms. Abla Habayeb, the school’s principal, provides her teachers with daily encouragement and support, and she involves community members, parents, and teachers in decisions about improving the school. Teachers, students, and the community then reciprocate that commitment. Thus, amid the surrounding poverty and instability, Kufor Quod girls excel in national tests. Similarly, in some poor villages in Jordan and Morocco, the leaders of schools and clinics are reaching out to the community, inspiring citizens’ trust and engagement through transparent and inclusive decision making and the delivery of excellent services.

Learning from such local successes is vital because there are no blueprints for solving service quality problems. Countries around the world are striving to improve education and health care quality. But simply modernizing school and hospital facilities and training staff are no longer sufficient. Delivering quality services requires motivated staff. And staff motivation arises in turn from values and accountability, which are grounded in the wider political, administrative, and social rules, practices, and relationships. Providing high-quality services is hard; the World Bank itself has struggled to ensure that its projects enhance incentives in country systems to achieve better learning and health outcomes.

We argue that because of the complex circumstances found in MENA countries, it is necessary to build on evidence of local successes and positive trends at the level of institutions, performance, and citizens’ trust and engagement. We hope that this report and its recommendations will help citizens, civil servants, policy makers, and donors alike jointly identify and build on the present foundation to improve the delivery of social services, shifting the cycle of performance into a virtuous gear. An improved cycle of performance is what those living in the MENA countries deserve and what would enable them to fulfill their hopes and dreams for the future.

Many reform initiatives in developing countries fail to achieve sustained improvements in performance because they are merely isomorphic mimicry—that is, governments and organizations pretend to reform by changing what policies or organizations look like rather than what they actually do. In addition, the flow of development resources and legitimacy without demonstrated improvements in performance undermines the impetus for effective action to build state capability or improve performance. This dynamic facilitates “capability traps” in which state capability stagnates, or even deteriorates, over long periods of time even though governments remain engaged in developmental rhetoric and continue to receive development resources. How can countries escape capability traps? We propose an approach, Problem-Driven Iterative Adaptation (PDIA), based on four core principles, each of which stands in sharp contrast with the standard approaches. First, PDIA focuses on solving locally nominated and defined problems in performance (as opposed to transplanting preconceived and packaged “best practice” solutions). Second, it seeks to create an authorizing environment for decision-making that encourages positive deviance and experimentation (as opposed to designing projects and programs and then requiring agents to implement them exactly as designed). Third, it embeds this experimentation in tight feedback loops that facilitate rapid experiential learning (as opposed to enduring long lag times in learning from ex post “evaluation”). Fourth, it actively engages broad sets of agents to ensure that reforms are viable, legitimate, relevant, and supportable (as opposed to a narrow set of external experts promoting the top-down diffusion of innovation).

Efforts to build state capability often take the form of commonly used, highly designed and engineered best practice solutions that have worked in many other places and that we suspect (and hope) will work again in many contexts. Such interventions do sometimes work, especially when the treatment actually addresses problems that fester in the context. Where the contextual problems are different, however, the treatment is just isomorphic mimicry—it looks good but will not be a solution to problems that actually matter. Development organizations often cannot see this, however, and offer the same solution again and again—hoping for a different outcome but imposing a capability trap on the policy context, where a new diagnosis and prescription is actually needed. In some countries the treatment has an even worse impact, fostering premature load bearing—where the context cannot actually handle what is prescribed. How can development experts identify in advance where they will have such negative impacts, and how can they identify in advance where they need to do development differently? This paper addresses such questions, and introduces an approach to building state capability in the latter contexts (called 1804 contexts), called problem driven iterative adaptation.

Many development challenges are complex, involving a lot of different agents and with unknown dimensions. Solutions to these challenges are often unknown, and contextually dependent. At the same time, there are political imperatives at play in many contexts which create pressure to 'find the solution now…and then scale it up.' Such pressure raises a question: how does a policy entrepreneur or reformer find a new solution and scale it up when dealing with complexity? This is the subject we address in the current paper, which is the fifth in a series on 'how to' do problem driven iterative adaptation (PDIA) (Andrews et al. 2015, 2016a, 2016b, 2016c).

The paper focuses on building broad agency solutions in the process of identifying problems and finding and fitting contextually appropriate solutions. The broad agency is, in our opinion, a most effective mechanism to ensure scaling and dynamic sustainability in the change process. As with other working papers on this topic, the contents here do not offer all answers to those asking questions about how to do development effectively. It closes by reflecting on the importance of 'you' (the reader, and ostensibly part of a policy change or reform team somewhere) using this and the other ideas as heuristics to rethink and reorient how you work—but with your own signature on each idea.