Person:

Levy, Dan

This paper summarizes the findings of an evaluation of the Programme of Advancement through Health and Education (PATH), a conditional cash transfer programme implemented by the Government of Jamaica. We find that PATH was generally implemented as intended; exhibited better targeting to the poor than other similar social assistance programmes in Jamaica; and had positive and statistically significant impacts on school attendance and number of preventive health care visits for children. We find no evidence, however, that PATH was able to affect longer-term outcomes such as marks, grade progression, or health care status.

Cold calling and pre-class web postings are frequently used to encourage students to read and come better prepared to class. Randomized experiments were conducted in two sections of an Empirical Methods course (API- 202) to assess to what extent these techniques increased student preparation and learning. Main conclusion is that in the context of this course, these two techniques led to increases in the amount of time that students spend reading before class, but not to learning gains. The study led the instructor to reflect on several aspects of his teaching practice, including the use of these two techniques for his courses, the interaction between the reading material before class and the material used during class, and the crowding out of student time from one course-related activity to another.

Some preliminary experiments the authors conducted suggested that when instructors asked students to raise their hands to indicate support for a certain answer or position, the results they got were very different than those that would be obtained through the use of polling devices (i.e. clickers). The authors hypothesized that raising hands is an act subject to peer influence, and hence serves as a poor indicator of what students are really thinking. The authors therefore conducted experiments more broadly and systematically across several classrooms, and assessed the types of questions in which the raising of hands were more (or less) likely to be an accurate indicator of the distribution of answers in the classroom.

No matter which way you look at it, Chiapas is the most backward of any state in Mexico. Its per capita income is the lowest of the 32 federal entities, at barely 40% of the national median (Figure 1). Its growth rate for the decade 2003-2013 was also the lowest (0.2%),1 causing the income gap separating Chiapas from the national average to increase from 53% to 60%. That is to say that today the average income for a worker in Mexico is two and a half times greater than the average in Chiapas. The two next poorest states, Oaxaca and Guerrero, are 25% and 30% above Chiapas.2 According to the Instituto Nacional de Estadística y Geografía de México (INEGI, National Institute of Statistics and Geography), Chiapas is also the state with the highest poverty rate (74.7%) as well as extreme poverty (46.7%).3

These major differences in income levels among Mexican federal entities are reproduced as in a fractal within Chiapas. In fact, while the wealthiest entity (Mexico City) is wealthier than the poorest (Chiapas) by a factor of six, the difference within Chiapas between the wealthiest municipality (Tuxtla Gutiérrez) and the poorest (Aldama and Mitontic) is by a factor greater than eight.4

As there are different "Mexicos" within Mexico,5 in Chiapas there are also different sorts of Chiapas (Figure 2). Income per capita in Tuxtla Gutiérrez, to the right of the distribution, is five standard deviations above the state average. Next comes a series of intermediate cities, San Cristóbal de las Casas, Comitán de Domínguez, Tapachula, and Reforma, between two and a half to four standard deviations above the average. The remaining municipalities of Chiapas follow (122 in all), clustered to the far left of the distribution. In addition, both the statistics available at the town level and our visits to various municipalities in Chiapas seem to indicate that significant differences also exist within these municipalities.

From this vantage point, questions as to why Chiapas is poor, or what explains its significant backwardness compared to other areas of Mexico, become much more complex. Why do some regions in Chiapas have high income levels, while other regions remain stagnant, fully dependent on federal transfers and deprived from the benefits of modern life?

1 This is the non-oil gross domestic product growth rate reported by INEGI, considered to be more representative of the productive spectrum. In any case, the overall rate of growth in Chiapas (-0.2%) was also the lowest amongst all Mexican entities for the decade.

2 Refers to non-oil GDP; in general terms, Guerrero and Oaxaca are 19% and 16% above Chiapas.

3 Growth figures refer to the decade 2003-2013, poverty figures are those published by INEGI for 2012.

4 Comparisons of Chiapas municipalities are made based on the data from the 10% sample of the 2010 Population Census, which is representative at the state level.

5 This is a reference to the report, A tale of two Mexicos: Growth and prosperity in a two-speed economy, McKinsey Global Institute (2014).

The Transparency for Development (T4D) project was launched in 2013 to try to answer these challenging questions and to make sense of a highly mixed evidence base. Ultimately, we sought to design research that could provide useful and actionable guidance to policymakers, donors, and practitioners alike seeking to improve the effectiveness of their work.

Two-stage examinations consist of a first stage in which students work individually as they typically do in examinations (stage 1), followed by a second stage in which they work in groups to complete another examination (stage 2), which typically consists of a subset of the questions from the first examination. Data from two-stage midterm and final examinations are used to assess the extent to which individuals improve their performance when collaborating with other students. On average, the group (stage 2) score was about one standard deviation above the individual (stage 1) score. While this difference cannot be interpreted as the causal effect of two-stage examinations on learning, it suggests that individuals experienced substantial performance gains when working in groups in an examination. This average performance gain was comparable with the average difference between the top performer of the group in stage 1 and the group’s stage 1 average, and was equivalent to about two-thirds of the difference between the “super student” score (i.e. the sum of the maximum score for each question in stage 1) and the group’s stage 1 average. This last result suggests that group collaboration takes substantial (albeit partial) advantage of the aggregate knowledge and skills of the group’s individual members. Student feedback about their experience with two-stage examinations reveal that that these types of examinations are generally perceived to be more helpful for learning and are less stressful than traditional examinations. Finally, using data on group gender compositions, we investigate the potential role of gender dynamics on group efficiency.

This baseline report describes the baseline data collection activities, baseline findings and revised primary outcomes for the impact evaluation of phase one of the Transparency for Development (T4D) project.

The report is broken into five chapters. Chapter I describes the T4D intervention and evaluation, and situates the project within the broader context of the fields of transparency and accountability (T/A) and maternal and neonatal health (MNH). Chapter II describes the T4D data collection process, including a description of the survey tools, sampling strategy, and how data were cleaned and prepared. Chapter III presents the baseline findings for the T4D communities in Indonesia and Tanzania. It also describes how these findings were used to revise the evaluation design. Chapter IV details the T4D impact evaluation design. This chapter includes details on random assignment and verification of balance on observable variables between treatment and control villages, a listing of primary outcomes by research question, and the T4D impact estimation strategy. Finally, chapter V presents conclusions and next steps.

This chapter is organized as follows: Section 1 describes the current state of MNH worldwide, and in Tanzania and Indonesia specifically. It explores the use of transparency and accountability (T/A) interventions to improve MNH and positions the T4D project within this debate. Section 2 provides a detailed description of the T4D intervention, including its underlying logic model. Section 3 describes the T4D evaluation, in which the impact evaluation (the primary topic of this report) plays a major role.

The Transparency for Development Evaluation Design Report describes the mixed methods evaluation design for the Transparency for Development project as of February 2015 – just before the start of the first data collection activities associated with the project (baseline data collection in Indonesia).

Since the writing of this report, a number of changes have been made to the design. For the most recent details, visit t4dproject.org.

This paper provides insight into community designed and led actions in Indonesia and Tanzania that were prompted by Transparency for Development (T4D), a six-year research project that explores whether, how, and in what conditions “transparency and accountability” or “social accountability” programs improve maternal and newborn health care. We find that all communities participating in the T4D program planned social actions, with the vast majority completing at least one action. We also find that the focus of the actions was diverse in nature, though participants in nearly every community planned at least one action aimed at educating members of the community. We compare actions designed in Indonesia to those in Tanzania and find a number of similarities and differences in the types of actions designed and whether the actions were completed. When analyzed from a social accountability lens, we find three trends. Firstly, the actions were overwhelmingly collaborative in nature. Secondly, the majority of the actions were short route, meaning they targeted the health facility or provider directly, rather than government officials higher up the accountability chain. Finally, when classified by accountability “type” we find that more than half of communities took a self-help approach, with only about a quarter pursuing solutions through social accountability channels.

The goal of the analysis described here is to identify the effects of the Transparency for Development program intervention on a range of maternal health and community participation outcomes as well as intermediate or process outcomes.

The plan pre-specifies the analysis that will be conducted, before comparing outcomes between treatment and control groups. It outlines the intervention, evaluation design, data sources, hypotheses and outcomes of interest, and the impact estimation strategy.

By committing to pre-specified analysis plans we hope to minimize issues of data mining and specification searching. The pre-analysis plan serves the dual purpose of ensuring the endline data collection tools are sufficient for the planned analysis. This plan was written and submitted after baseline data collection and the implementation of the intervention, but prior to the start of endline data collection.