A new study on meta reinforcement learning algorithms helps us understand how the human brain learns to adapt to complexity and uncertainty when learning and making decisions. A research team, led by Professor Sang Wan Lee at KAIST jointly with John O’Doherty at Caltech, succeeded in discovering both a computational and neural mechanism for human meta reinforcement learning, opening up the possibility of porting key elements of human intelligence into artificial intelligence algorithms. This study provides a glimpse into how it might ultimately use computational models to reverse engineer human reinforcement learning.
This work was published on Dec 16, 2019 in the journal Nature Communications. The title of the paper is “Task complexity interacts with state-space uncertainty in the arbitration between model-based and model-free learning.”
Human reinforcement learning is an inherently complex and dynamic process, involving goal setting, strategy choice, action selection, strategy modification, cognitive resource allocation etc. This a very challenging problem for humans to solve owing to the rapidly changing and multifaced environment in which humans have to operate. To make matters worse, humans often need to often rapidly make important decisions even before getting the opportunity to collect a lot of information, unlike the case when using deep learning methods to model learning and decision-making in artificial intelligence applications.
In order to solve this problem, the research team used a technique called 'reinforcement learning theory-based experiment design' to optimize the three variables of the two-stage Markov decision task - goal, task complexity, and task uncertainty. This experimental design technique allowed the team not only to control confounding factors, but also to create a situation similar to that which occurs in actual human problem solving.
Secondly, the team used a technique called ‘model-based neuroimaging analysis.’ Based on the acquired behavior and fMRI data, more than 100 different types of meta reinforcement learning algorithms were pitted against each other to find a computational model that can explain both behavioral and neural data. Thirdly, for the sake of a more rigorous verification, the team applied an analytical method called ‘parameter recovery analysis,’ which involves high-precision behavioral profiling of both human subjects and computational models.
In this way, the team was able to accurately identify a computational model of meta reinforcement learning, ensuring not only that the model’s apparent behavior is similar to that of humans, but also that the model solves the problem in the same way as humans do.
The team found that people tended to increase planning-based reinforcement learning (called model-based control), in response to increasing task complexity. However, they resorted to a simpler, more resource efficient strategy called model-free control, when both uncertainty and task complexity were high. This suggests that both the task uncertainty and the task complexity interact during the meta control of reinforcement learning. Computational fMRI analyses revealed that task complexity interacts with neural representations of the reliability of the learning strategies in the inferior prefrontal cortex.
These findings significantly advance understanding of the nature of the computations being implemented in the inferior prefrontal cortex during meta reinforcement learning as well as providing insight into the more general question of how the brain resolves uncertainty and complexity in a dynamically changing environment. Identifying the key computational variables that drive prefrontal meta reinforcement learning, can also inform understanding of how this process might be vulnerable to break down in certain psychiatric disorders such as depression and OCD. Furthermore, gaining a computational understanding of how this process can sometimes lead to increased model-free control, can provide insights into how under some situations task performance might break down under conditions of high cognitive load.
Professor Lee said, “This study will be of enormous interest to researchers in both the artificial intelligence and human/computer interaction fields since this holds significant potential for applying core insights gleaned into how human intelligence works with AI algorithms.”
This work was funded by the National Institute on Drug Abuse, the National Research Foundation of Korea, the Ministry of Science and ICT, Samsung Research Funding Center of Samsung Electronics.
< external_image >
Figure 1 (modified from the figures of the original paper doi:10.1038/s41467-019-13632-1). Computations implemented in the inferior prefrontal cortex during meta reinforcement learning. (A) Computational model of human prefrontal meta reinforcement learning (left) and the brain areas whose neural activity patterns are explained by the latent variables of the model. (B) Examples of behavioral profiles. Shown on the left is choice bias for different goal types and on the right is choice optimality for task complexity and uncertainty. (C) Parameter recoverability analysis. Compared are the effect of task uncertainty (left) and task complexity (right) on choice optimality.
Risk Quotient 2020, an international conference co-hosted by KAIST and the National University of Singapore (NUS), will bring together world-leading experts from academia and industry to help Asia navigate the post-COVID-19 and Fourth Industrial Revolution (4IR) eras. The online conference will be held on October 29 from 10 a.m. Korean time under the theme “COVID-19 Pandemic and A Brave New World”. It will be streamed live on YouTube at https://www.youtube.com/c/KAISTofficial and ht2020-10-22
PhD candidate Taesik Gong from the School of Computing was named a 2020 Google PhD Fellow in the field of machine learning. The Google PhD Fellowship Program has recognized and supported outstanding graduate students in computer science and related fields since 2009. Gong is one of two Korean students chosen as the recipients of Google Fellowships this year. A total of 53 students across the world in 12 fields were awarded this fellowship. Gong’s research on condition-independent mobil2020-10-15
Professor Jungwon Kim from the Department of Mechanical Engineering was selected as the ‘Scientist of the Month’ for October 2020 by the Ministry of Science and ICT and the National Research Foundation of Korea. Professor Kim was recognized for his contributions to expanding the horizons of the basics of precision engineering through his research on multifunctional ultrahigh-speed, high-resolution sensors. He received 10 million KRW in prize money. Professor Kim was selected as th2020-10-15
Professor Won-Ki Cho from the Department of Biological Sciences was named one of three recipients of the 2020 Suh Kyung-Bae Science Foundation (SUHF) Young Investigator Award. The SUHF is a non-profit organization established in 2016 and funded by a personal donation of 300 billion KRW in shares from Chairman and CEO Kyung-Bae Suh of the Amorepacific Group. The primary purpose of the foundation is to serve as a platform to nurture and provide comprehensive long-term support for creative and p2020-10-15
Faster ships could be on the horizon after KAIST scientists develop a slippery surface inspired by fish and seaweed to reduce the hull's drag through the water. Long-distance cargo ships lose a significant amount of energy due to fluid friction. Looking to the drag reduction mechanisms employed by aquatic life can provide inspiration on how to improve efficiency. Fish and seaweed secrete a layer of mucus to create a slippery surface, reducing their friction as they travel through water. A pote2020-10-12