This approach combines the strengths of two algorithms that are already well known in reinforcement learning and are also believed to exist in humans and rodents. “Model-based” algorithms learn models of the environment that can then be simulated to produce estimates of future reward, while “model-free” algorithms learn future reward estimates directly from experience in the environment. Model-based algorithms are flexible but computationally expensive, while model-free algorithms are computationally cheap but inflexible.
The algorithm that inspired our theory combines some of the flexibility of model-based algorithms with the efficiency of model-free algorithms. Because the calculation is a simple weighted sum, it is computationally efficient, much like a model-free algorithm. At the same time, by separating reward expectations and state expectations (the predictive map), it can rapidly adapt to changes in reward by simply updating the reward expectations while leaving the state expectations intact (see our recent paper for further detail).
In future work, we plan to test the theory further. Since the predictive map theory can be translated into a neural network architecture, we want to explore the extent to which this learning strategy can promote flexible, rapid planning in silico.
More generally, a major future task will be to look at how the brain integrates different types of learning. While we posed this model as an alternative to model-based and model-free learning in the brain, a more realistic view is that many types of learning are simultaneously coordinated by the brain during learning and planning. Understanding how these learning algorithms are combined is an important step towards understanding human and animal brains, and could provide key insights for designing equally complex, multifaceted AI.
Read the full paper
Read AI and Neuroscience: A virtuous circle
Source: https://deepmind.com/blog/article/hippocampus-predictive-map
