Person:

Woolcock, Michael

Rising standards for accurately inferring the impact of development projects has not been matched by equivalently rigorous procedures for guiding decisions about whether and how similar results might be expected elsewhere. These 'external validity' concerns are especially pressing for 'complex' development interventions, in which the explicit purpose is often to adapt projects to local contextual realities and where high quality implementation is paramount to success. A basic analytical framework is provided for assessing the external validity of complex development interventions. It argues for deploying case studies to better identify the conditions under which diverse outcomes are observed, focusing in particular on the salience of contextual idiosyncrasies, implementation capabilities and trajectories of change. Upholding the canonical methodological principle that questions should guide methods, not vice versa, is required if a truly rigorous basis for generalizing claims about likely impact across time, groups, contexts and scales of operation is to be discerned for different kinds of development interventions.

The coherence and effectiveness of engagement with the world's 'fragile and conflict-affected states'—beyond ethical imperatives and geo-strategic considerations—turns on answers to two vexing questions. First, on what defensible basis is any given country, at any given historical moment, deemed to be (or not to be) 'fragile'? Second, if a defining characteristic of state fragility is low levels of capability to implement core responsibilities, how can international agencies best support domestic public organizations to acquire capability? The first issue may appear to be a methodological one (wherein more and better data would provide a firmer empirical foundation on which to base key decisions) but any determination, especially of marginal cases, must also be grounded in a correspondingly comprehensive theory of change. Similarly, the optimal response to the second issue may appear to be importing technical and rigorously verified ('best practice') solutions, but in fact it is more likely to require a qualitatively different strategy, one able to experiment with alternative design specifications and adapt in real time to changing contextual realities (thereby iterating towards customized 'best fit' solutions). To this end, an alternative approach to the theory, measurement and practice of engaging with fragile states is outlined, in the spirit of rising concerns across the development community that prevailing strategies have demonstrably reached the limits of their effectiveness.

Beyond assessing whether or not interventions have achieved their stated goals, evaluations can also provide greater understanding—in real-time and at completion—about how, where, for whom, over what time frame, and which aspects of an intervention may (or may not) have succeeded, and thus where improvements might be sought. Moreover, to the extent any intervention is only as good as its implementation, evaluations can also help identify where breakdowns in the delivery system may have occurred and spaces wherein frontline implementers were able to find innovative solutions to local (“binding constraint”) problems. Process and implementation evaluations thus serve the vital purpose of jointly promoting accountability and learning, thereby expanding the common perception of evaluations as external instruments of compliance and discipline to internal procedures for promoting partnerships, innovation, and improvement (organizationally or technically). In this chapter, we describe six different types of process and implementation evaluations and describe their respective strengths and weaknesses in various contexts, for various purposes. As part of collective efforts to enhance the effectiveness of all classes of interventions, impact and process evaluations should be regarded as necessary complements.

A defining task of development is enhancing a state’s capability for policy implementation. In most low- income countries, alas, such capabilities seem to be stagnant or declining, in no small part because dominant reform strategies are ill-suited to addressing complex non-technical aspects. This has been recognized for at least six decades – indeed, it was a centerpiece of Albert Hirschman’s understanding of the development process – yet this critique, and the significance of its implications, remain on the margins of scholarship and policy. Why? I consider three options, concluding that, paradoxically, followers of Hirschman’s approach inadequately appreciated that gaining more operational traction for their approach was itself a type of problem requiring their ideas to embark on ‘a long voyage of discovery’, a task best accomplished, in this instance, by building – and tapping into the distinctive insights of – a diverse community of development practitioners.

Evaluations of development projects are conducted to assess their net effectiveness and, by extension, to guide decisions regarding the merits of scaling-up successful projects and/or replicating them elsewhere. The key characteristics of ‘complex’ interventions – numerous face- to-face interactions, high discretion, imposed obligations, pervasive unknowns – rarely fit neatly into standard evaluation protocols, requiring the deployment of a wider array of research methods, tools and theory. The careful use of such ‘mixed methods’ approaches is especially important for discerning the conditions under which ‘successful’ projects of all kinds might be expanded or adopted elsewhere. These claims, and the practical implications to which they give rise, draw on an array of recent evaluations in different sectors in development.

Many development agencies and governments now seek to engage directly with local communities, whether as a means to the realization of more familiar goals (infrastructure, healthcare, education) or as an end in itself (promoting greater inclusion, participation, well-being). These same agencies and governments, however, are also under increasing pressure to formally demonstrate that their actions ‘work’ and achieve their goals within relatively short timeframes – expectations which are, for the most part, necessary and desirable. But adequately assessing ‘community-driven’ approaches to development requires the deployment of theory and methods that accommodate their distinctive characteristics: building bridges is a qualitatively different task to building the rule of law and empowering minorities. Moreover, the ‘lessons’ inferred from average treatment effects derived from even the most rigorous assessments of community-driven interventions are likely to translate poorly to different contexts and scales of operation. Some guidance for anticipating and managing these conundrums are provided.

Since March 2020, the novelty, intensity, and scale of the COVID-19 pandemic has placed enormous stress on governments, delivery systems, and social order around the world, especially so in countries with modest public health resources, where targeting is especially difficult, and among occupational groups working in close proximity to others. Those overseeing Indonesia’s health care system from mid-2020 onwards faced precisely this vortex of existential challenges, but certain organizations within this system were well placed to deploy an adaptive implementation strategy. KOMPAK was one such organization2; its efforts were not universally successful, but their achievements (e.g., coherently coordinating governance efforts between national, regional, and local levels of government; building effective village information systems) were nonetheless distinctive, consequential and enduring. In this sense, COVID-19 can be understood as a perverse but instructive “natural experiment” in how well public sector organizations respond in the face of unexpected high-stakes, high-uncertainty, low-resource, low-prior-experience crises. We document key insights and implications for public sector administration from KOMPAK’s efforts, not just for Indonesia and other developing countries but for public service delivery systems more generally.

In rich and poor countries alike, a core challenge is building the state’s capability for policy implementation. Delivering high-quality public health and health care – affordably, reliably, at scale, for all – exemplifies this challenge, since doing so requires deftly integrating refined technical skills (surgery), broad logistics management (supply chains, facilities maintenance), adaptive problem solving (curative care) and resolving ideological differences (who pays? who provides?), even as the prevailing health problems themselves only become more diverse, complex and expensive as countries become more prosperous. The current state of state capability in developing countries, however, is demonstrably alarming, with the strains and demands only likely to intensify in the coming decades. Prevailing ‘best practice’ strategies for building implementation capability – copying and scaling putative successes from abroad – are too often part of the problem, while individual training (‘capacity building’) and technological upgrades (e.g., new management information systems) remain necessary but deeply insufficient. An alternative approach is outlined, one centered on building implementation capability by working iteratively to solve problems nominated and prioritized by local actors.

The Middle East and North Africa (MENA) is a rising middle-income region, and its citizens rightly expect quality public services. Yet too often they experience disappointment: students attending local schools are insufficiently prepared for the 21st century economy, and those needing health care too often find that public clinics have no doctors or medicines. Few in positions of authority are held accountable for such shortcomings. This situation both undermines the potential for improvement and heightens people’s unhappiness with the delivery system.

Although dissatisfaction with education and health services is widespread in the MENA region, local successes do exist and offer inspiration. At the Kufor Quod Girls’ Secondary School in the rural West Bank, for example, Ms. Abla Habayeb, the school’s principal, provides her teachers with daily encouragement and support, and she involves community members, parents, and teachers in decisions about improving the school. Teachers, students, and the community then reciprocate that commitment. Thus, amid the surrounding poverty and instability, Kufor Quod girls excel in national tests. Similarly, in some poor villages in Jordan and Morocco, the leaders of schools and clinics are reaching out to the community, inspiring citizens’ trust and engagement through transparent and inclusive decision making and the delivery of excellent services.

Learning from such local successes is vital because there are no blueprints for solving service quality problems. Countries around the world are striving to improve education and health care quality. But simply modernizing school and hospital facilities and training staff are no longer sufficient. Delivering quality services requires motivated staff. And staff motivation arises in turn from values and accountability, which are grounded in the wider political, administrative, and social rules, practices, and relationships. Providing high-quality services is hard; the World Bank itself has struggled to ensure that its projects enhance incentives in country systems to achieve better learning and health outcomes.

We argue that because of the complex circumstances found in MENA countries, it is necessary to build on evidence of local successes and positive trends at the level of institutions, performance, and citizens’ trust and engagement. We hope that this report and its recommendations will help citizens, civil servants, policy makers, and donors alike jointly identify and build on the present foundation to improve the delivery of social services, shifting the cycle of performance into a virtuous gear. An improved cycle of performance is what those living in the MENA countries deserve and what would enable them to fulfill their hopes and dreams for the future.

Many reform initiatives in developing countries fail to achieve sustained improvements in performance because they are merely isomorphic mimicry—that is, governments and organizations pretend to reform by changing what policies or organizations look like rather than what they actually do. In addition, the flow of development resources and legitimacy without demonstrated improvements in performance undermines the impetus for effective action to build state capability or improve performance. This dynamic facilitates “capability traps” in which state capability stagnates, or even deteriorates, over long periods of time even though governments remain engaged in developmental rhetoric and continue to receive development resources. How can countries escape capability traps? We propose an approach, Problem-Driven Iterative Adaptation (PDIA), based on four core principles, each of which stands in sharp contrast with the standard approaches. First, PDIA focuses on solving locally nominated and defined problems in performance (as opposed to transplanting preconceived and packaged “best practice” solutions). Second, it seeks to create an authorizing environment for decision-making that encourages positive deviance and experimentation (as opposed to designing projects and programs and then requiring agents to implement them exactly as designed). Third, it embeds this experimentation in tight feedback loops that facilitate rapid experiential learning (as opposed to enduring long lag times in learning from ex post “evaluation”). Fourth, it actively engages broad sets of agents to ensure that reforms are viable, legitimate, relevant, and supportable (as opposed to a narrow set of external experts promoting the top-down diffusion of innovation).