When Does Model-Based Control Pay Off?

DSpace/Manakin Repository

When Does Model-Based Control Pay Off?

Citable link to this page


Title: When Does Model-Based Control Pay Off?
Author: Kool, Wouter; Cushman, Fiery A.; Gershman, Samuel J.

Note: Order does not necessarily reflect citation order of authors.

Citation: Kool, Wouter, Fiery A. Cushman, and Samuel J. Gershman. 2016. “When Does Model-Based Control Pay Off?” PLoS Computational Biology 12 (8): e1005090. doi:10.1371/journal.pcbi.1005090. http://dx.doi.org/10.1371/journal.pcbi.1005090.
Full Text & Related Files:
Abstract: Many accounts of decision making and reinforcement learning posit the existence of two distinct systems that control choice: a fast, automatic system and a slow, deliberative system. Recent research formalizes this distinction by mapping these systems to “model-free” and “model-based” strategies in reinforcement learning. Model-free strategies are computationally cheap, but sometimes inaccurate, because action values can be accessed by inspecting a look-up table constructed through trial-and-error. In contrast, model-based strategies compute action values through planning in a causal model of the environment, which is more accurate but also more cognitively demanding. It is assumed that this trade-off between accuracy and computational demand plays an important role in the arbitration between the two strategies, but we show that the hallmark task for dissociating model-free and model-based strategies, as well as several related variants, do not embody such a trade-off. We describe five factors that reduce the effectiveness of the model-based strategy on these tasks by reducing its accuracy in estimating reward outcomes and decreasing the importance of its choices. Based on these observations, we describe a version of the task that formally and empirically obtains an accuracy-demand trade-off between model-free and model-based strategies. Moreover, we show that human participants spontaneously increase their reliance on model-based control on this task, compared to the original paradigm. Our novel task and our computational analyses may prove important in subsequent empirical investigations of how humans balance accuracy and demand.
Published Version: doi:10.1371/journal.pcbi.1005090
Other Sources: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC5001643/pdf/
Terms of Use: This article is made available under the terms and conditions applicable to Other Posted Material, as set forth at http://nrs.harvard.edu/urn-3:HUL.InstRepos:dash.current.terms-of-use#LAA
Citable link to this page: http://nrs.harvard.edu/urn-3:HUL.InstRepos:29407856
Downloads of this work:

Show full Dublin Core record

This item appears in the following Collection(s)


Search DASH

Advanced Search