Activity Allocation in an Under-Resourced World: Toward Improving Engagement with Public Health Programs via Restless Bandits

Abstract

Artificial intelligence (AI) tools are being developed widely within society to improve decision-making, especially in resource-constrained settings like public health. However, developing effective AI tools for public health is complicated by scale, the time-varying and intervention-dependent nature of individuals' behavior, and scarcity --- of intervention resources, historical data, and real-time observations. Many of these challenges are unaddressed in literature and can lead to poor outcomes if ignored in real-world AI support systems for public health. For example, previous works have modeled the problem of delivering interventions to improve engagement with health programs as a restless bandit, a widely-studied framework in which a set of stochastic arms are controlled by a planner with a limited intervention budget. However, these works do not account for the uncertainty that results from estimating the dynamics of stochastic arms from noisy observations and historical data. To address this, I introduce the first methods for computing uncertainty-robust restless bandit policies, across a range of assumptions on prior information and observability. I achieve this by designing novel approaches to efficiently search large policy and uncertainty spaces, e.g., a new multi-agent deep reinforcement learning paradigm in which a centralized budget-network communicates with per-arm policy-networks to learn globally optimal policies in an environment controlled by a regret-maximizing adversary-network. In union with this work, I advance art within the more computationally intensive generalization of restless bandits that finds policies which balance many types of interventions with unique costs and effects, e.g., a phone call vs.~in-person visit; my works identify and exploit functional structures to design new algorithms that scale. This dissertation tackles several such challenges driven by identifying the key missing capabilities of interdisciplinary collaborators, especially tuberculosis healthcare workers and workers within a maternal health nonprofit in India.