Cardinality Restricted Boltzmann Machines

View/ Open
Author
Swersky, Kevin
Tarlow, Daniel
Sutskever, Ilya
Salakhutdinov, Ruslan
Zemel, Richard
Metadata
Show full item recordCitation
Swersky, Kevin, Daniel Tarlow, Ilya Sutskever, Ruslan Salakhutdinov, Richard S. Zemel, and Ryan P. Adams. 2012. Cardinality restricted Boltzmann machines. Advances in Neural Information Processing Systems 25: 3293-3301.Abstract
The Restricted Boltzmann Machine (RBM) is a popular density model that is also good for extracting features. A main source of tractability in RBM models is that, given an input, the posterior distribution over hidden variables is factorizable and can be easily computed and sampled from. Sparsity and competition in the hidden representation is beneficial, and while an RBM with competition among its hidden units would acquire some of the attractive properties of sparse coding, such constraints are typically not added, as the resulting posterior over the hidden units seemingly becomes intractable. In this paper we show that a dynamic programming algorithm can be used to implement exact sparsity in the RBM’s hidden units. We also show how to pass derivatives through the resulting posterior marginals, which makes it possible to fine-tune a pre-trained neural network with sparse hidden layers.Terms of Use
This article is made available under the terms and conditions applicable to Open Access Policy Articles, as set forth at http://nrs.harvard.edu/urn-3:HUL.InstRepos:dash.current.terms-of-use#OAPCitable link to this page
http://nrs.harvard.edu/urn-3:HUL.InstRepos:11315421
Collections
- FAS Scholarly Articles [18153]
Contact administrator regarding this item (to report mistakes or request changes)