Cognitive Control Models Based on Reinforcement Learning Theory
Shi Kangyin1, Fei Boyao2, Guo Mingqian3
Author information+
1School of Education and Psychology, Chengdu Normal University, Chengdu, 611130; 2School of Psychology, South China Normal University, Guangzhou, 510631; 3Behavioral Science Institute, Radboud University, Nijmegen, 6525GD
Cognitive control refers to the ability to regulate and coordinate cognitive processes, enabling individuals to flexibly adjust their behavior in alignment with current goals. Understanding the underlying mechanisms of cognitive control has become a focal point in cognitive psychology research. A class of computational models based on reinforcement learning has been developed to explain cognitive control, including the Conflict Monitoring (CM) model, the Expected Value of Control (EVC) theory, the Learned Value of Control (LVOC) theory, and the Predicted Response-Outcome (PRO) model. These four models aim to account for the implementation of cognitive control from different perspectives, and there are ongoing debates among them. The CM model primarily addresses when control should be implemented, suggesting that conflict serves as a signal for the adjustment of control. In contrast, the EVC theory emphasizes the comparison and selection process of control signals within the cognitive control system prior to implementation. It conceptualizes the selection of control signals as a decision-making problem, involving a trade-off between reward and cost, with the expected value of control acting as the criterion for selecting signals. The LVOC theory seeks to investigate how individuals determine the value of control at a more fundamental level, suggesting that environmental and stimulus characteristics are critical factors in learning and controlling the value of cognitive control. The PRO model, meanwhile, aims to explain a range of phenomena related to cognitive control through the framework of prediction error. Research suggests that brain regions associated with cognitive control, such as the fronto-parietal network, the anterior insula, and subcortical structures like the basal ganglia, provide the biological basis for the implementation of reinforcement learning in cognitive control. These regions play a critical role in the regulation and management of control processes. Specifically, the anterior cingulate cortex is responsible for monitoring control demand signals in the environment and then transmitting these signals to the dorsolateral prefrontal cortex. The latter is responsible for implementing control. Additionally, studies have indicated that the insula and the caudate nucleus also play a role in the modulation of control processes. In recent years, research in the field of cognitive control has shifted its focus from control regulation to control management processes, with an emphasis on the learning processes related to control value. Research has found that the computation of control rewards and control costs is associated with the activity of the medial prefrontal cortex, anterior cingulate cortex, and striatum. A key aspect of the brain's learning of control value lies in reward prediction error, whereby the brain updates its estimation of control value in future situations by comparing the difference between expected and actual control rewards. This process is related to brain regions such as the anterior cingulate cortex, orbitofrontal cortex, anterior insula, and striatum. Future research is encouraged to explore the following questions. First, it is recommended that future studies recruit special populations to address the ongoing debates between the EVC model and the PRO model. For instance, individuals with schizophrenia may exhibit deficits in generating prediction errors, while their ability to compute control value might remain unaffected or even be enhanced. Studying this population could provide valuable insights to resolve these theoretical controversies. Second, future research should reassess the impact of cognitive effort on the expected value of control. Both the EVC and LVOC models should integrate the influence of cognitive effort on control value, as cognitive effort can yield rewards, thereby mitigating aversion to effort. Investigating how to quantify this effect within these models will be an important avenue for future work. Finally, further development of the clinical application of reinforcement learning-based cognitive control models is essential. Applying these computational models to individuals with mental disorders and comparing their data to that of healthy controls may offer insights into the mechanisms underlying cognitive impairments across various mental disorders and subtypes. Coupled with functional magnetic resonance imaging (fMRI), this approach could link impaired cognitive processes to abnormal activation patterns in specific brain regions or dysfunctional brain network connectivity. The goal is to identify stable, generalizable biomarkers associated with different mental disorders and their subtypes, providing a foundation for more accurate diagnoses.
Shi Kangyin, Fei Boyao, Guo Mingqian.
Cognitive Control Models Based on Reinforcement Learning Theory[J]. Journal of Psychological Science. 2025, 48(5): 1076-1088 https://doi.org/10.16719/j.cnki.1671-6981.20250505
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}
参考文献
[1] 曹思琪, 汤晨晨, 伍海燕, 刘勋. (2022). 价值计算决定何时与如何努力. 心理科学进展, 30(4), 877-887. [2] 司双庆, 周思宏, 袁加锦, 杨倩. (2024). 认知控制过程的影响因素: 回报与代价. 心理科学, 47(2), 258-266. [3] Abler B., Walter H., Erk S., Kammerer H., & Spitzer M. (2006). Prediction error as a linear function of reward probability is coded in human nucleus accumbens. NeuroImage, 31(2), 790-795. [4] Alexander, W. H., & Brown, J. W. (2010). Computational models of performance monitoring and cognitive control. Topics in Cognitive Science, 2(4), 658-677. [5] Alexander, W. H., & Brown, J. W. (2011). Medial prefrontal cortex as an action-outcome predictor. Nature Neuroscience, 14(10), 1338-1344. [6] Alexander, W. H., & Brown, J. W. (2014). A general role for medial prefrontal cortex in event prediction. Frontiers in Computational Neuroscience, 8, Article 69. [7] Alexander, W. H., & Brown, J. W. (2015). Hierarchical error representation: A computational model of anterior cingulate and dorsolateral prefrontal cortex. Neural Computation, 27(11), 2354-2410. [8] Alexander, W. H., & Brown, J. W. (2019). The role of the anterior cingulate cortex in prediction error and signaling surprise. Topics in Cognitive Science, 11(1), 119-135. [9] Alexander W. H., Vassena E., Deraeve J., & Langford Z. D. (2017). Integrative modeling of prefrontal cortex. Journal of Cognitive Neuroscience, 29(10), 1674-1683. [10] Amiez C., Joseph J. P., & Procyk E. (2006). Reward encoding in the monkey anterior cingulate cortex. Cerebral Cortex, 16(7), 1040-1055. [11] Bailey, C. E. (2007). Cognitive accuracy and intelligent executive function in the brain and in business. Annals of the New York Academy of Sciences, 1118(1), 122-141. [12] Bartra O., McGuire J. T., & Kable J. W. (2013). The valuation system: A coordinate-based meta-analysis of BOLD fMRI experiments examining neural correlates of subjective value. NeuroImage, 76, 412-427. [13] Bausenhart K. M., Ulrich R., & Miller J. (2021). Effects of conflict trial proportion: A comparison of the Eriksen and Simon tasks. Attention, Perception, and Psychophysics, 83(2), 810-836. [14] Behrens T. E. J., Woolrich M. W., Walton M. E., & Rushworth, M. F. S. (2007). Learning the value of information in an uncertain world. Nature Neuroscience, 10(9), 1214-1221. [15] Blais, C., & Bunge, S. (2010). Behavioral and neural evidence for item-specific performance monitoring. Journal of Cognitive Neuroscience, 22(12), 2758-2767. [16] Blais C., Robidoux S., Risko E. F., & Besner D. (2007). Item-specific adaptation and the conflict-monitoring hypothesis: A computational model. Psychological Review, 114(4), 1076-1086. [17] Blanchard T. C., Strait C. E., & Hayden B. Y. (2015). Ramping ensemble activity in dorsal anterior cingulate neurons during persistent commitment to a decision. Journal of Neurophysiology, 114(4), 2439-2449. [18] Botvinick, M. M. (2007). Conflict monitoring and decision making: Reconciling two perspectives on anterior cingulate function. Cognitive, Affective, and Behavioral Neuroscience, 7(4), 356-366. [19] Botvinick M. M., Braver T. S., Barch D. M., Carter C. S., & Cohen J. D. (2001). Conflict monitoring and cognitive control. Psychological Review, 108(3), 624-652. [20] Brown, T. E., & Landgraf, J. M. (2010). Improvements in executive function correlate with enhanced performance and functioning and health-related quality of life: Evidence from 2 large, double-blind, randomized, placebo-controlled trials in ADHD. Postgraduate Medicine, 122(5), 42-51. [21] Cao S. Q., Liu X., & Wu H. Y. (2022). The neural mechanisms underlying effort process modulated by efficacy. Neuropsychologia, 173, Article 108314. [22] Carter C. S., Mintun M., Nichols T., & Cohen J. D. (1997). Anterior cingulate gyrus dysfunction and selective attention deficits in schizophrenia:[15O] H2O PET study during single-trial Stroop task performance. American Journal of Psychiatry, 154(12), 1670-1675. [23] Chiu, Y. C., & Egner, T. (2017). Cueing cognitive flexibility: Item-specific learning of switch readiness. Journal of Experimental Psychology: Human Perception and Performance, 43(12), 1950-1960. [24] Chiu, Y. C., & Egner, T. (2019). Cortical and subcortical contributions to context-control learning. Neuroscience and Biobehavioral Reviews, 99, 33-41. [25] Chiu Y. C., Jiang J. F., & Egner T. (2017). The caudate nucleus mediates learning of stimulus-control state associations. Journal of Neuroscience, 37(4), 1028-1038. [26] Cohen J. D., Dunbar K., & Mcclelland J. L. (1990). On the control of automatic processes: A parallel distributed processing account of the Stroop effect. Psychological Review, 97(3), 332-361. [27] Dosenbach N. U. F., Fair D. A., Miezin F. M., Cohen A. L., Wenger K. K., Dosenbach R. A. T., & Petersen S. E. (2007). Distinct brain networks for adaptive and stable task control in humans. Proceedings of the National Academy of Sciences of the United States of America, 104(26), 11073-11078. [28] Ebitz R. B., Smith E. H., Horga G., Schevon C. A., Yates M. J., McKhann G. M., Sheth S. A., & Hayden B. Y. (2020). Human dorsal anterior cingulate neurons signal conflict by amplifying task-relevant information. bioRxiv. [29] Egner, T., & Hirsch, J. (2005). The neural correlates and functional integration of cognitive control in a Stroop task. NeuroImage, 24(2), 539-547. [30] Feuerriegel D., Jiwa M., Turner W. F., Andrejević M., Hester R., & Bode S. (2021). Tracking dynamic adjustments to decision making and performance monitoring processes in conflict tasks. NeuroImage, 238, Article 118265. [31] Friston K., Brown H. R., Siemerkus J., & Stephan K. E. (2016). The dysconnection hypothesis. Schizophrenia Research, 176(2-3), 83-94. [32] Frömer R., Lin H., Dean Wolf C. K., Inzlicht M., & Shenhav A. (2021). Expectations of reward and efficacy guide cognitive control allocation. Nature Communications, 12(1), Article 1030. [33] Frömer, R., & Shenhav, A. (2022). Filling the gaps: Cognitive control as a critical lens for understanding mechanisms of value-based decision-making. Neuroscience and Biobehavioral Reviews, 134, 104483. [34] Geana A., Barch D. M., Gold J. M., Carter C. S., MacDonald III A. W., Ragland J. D., Silverstein S. M., & Frank M. J. (2022). Using computational modeling to capture schizophrenia-specific reinforcement learning differences and their implications on patient classification. Biological Psychiatry: Cognitive Neuroscience and Neuroimaging, 7(10), 1035-1046. [35] Grahek I., Frömer R., Fahey M. P., & Shenhav A. (2023). Learning when effort matters: Neural dynamics underlying updating and adaptation to changes in performance efficacy. Cerebral Cortex, 33(5), 2395-2411. [36] Grandjean J., D' Ostilio K., Phillips C., Balteau E., Degueldre C., Luxen A., Maquet. P., Salmon E., & Collette F. (2012). Modulation of brain activity during a Stroop inhibitory task by the kind of cognitive control required. PLoS ONE, 7(7), Article e41513. [37] Hebb D. O.(2002). The first stage of perception: Growth of the assembly. In D. O. Hebb (Ed.), The organization of behavior (pp. 60-78). Psychology Press. [38] Holroyd, C. B., & McClure, S. M. (2015). Hierarchical control over effortful behavior by rodent medial frontal cortex: A computational model. Psychological Review, 122(1), 54-83. [39] Holroyd, C. B., & Yeung, N. (2012). Motivation of extended behaviors by anterior cingulate cortex. Trends in Cognitive Sciences, 16(2), 122-128. [40] Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences of the United States of America, 79(8), 2554-2558. [41] Horga G., Maia T. V., Wang P. W., Wang Z. S., Marsh R., & Peterson B. S. (2011). Adaptation to conflict via context-driven anticipatory signals in the dorsomedial prefrontal cortex. Journal of Neuroscience, 31(45), 16208-16216. [42] Horne C. M., Vanes L. D., Verneuil T., Mouchlianitis E., Szentgyorgyi T., Averbeck B., Leech R., Moran, R J., & Shergill S. S. (2021). Cognitive control network connectivity differentially disrupted in treatment resistant schizophrenia. NeuroImage: Clinical, 30, Article 102631. [43] Hosking J. G., Cocker P. J., & Winstanley C. A. (2016). Prefrontal cortical inactivations decrease willingness to expend cognitive effort on a rodent cost/benefit decision-making task. Cerebral Cortex, 26(4), 1529-1538. [44] Howes, O. D., & Kapur, S. (2009). The dopamine hypothesis of schizophrenia: Version III—the final common pathway. Schizophrenia Bulletin, 35(3), 549-562. [45] Ileri-Tayar M., Moss C., & Bugg J. M. (2022). Transfer of learned cognitive control settings within and between tasks. Neurobiology of Learning and Memory, 196, Article 107689. [46] Inzlicht M., Shenhav A., & Olivola C. Y. (2018). The effort paradox: Effort is both costly and valued. Trends in Cognitive Sciences, 22(4), 337-349. [47] Jahn A., Nee D. E., Alexander W. H., & Brown J. W. (2014). Distinct regions of anterior cingulate cortex signal prediction and outcome evaluation. NeuroImage, 95, 80-89. [48] Jessup R. K., Busemeyer J. R., & Brown J. W. (2010). Error effects in anterior cingulate cortex reverse when error likelihood is high. Journal of Neuroscience, 30(9), 3467-3472. [49] Jiang J., Heller K., & Egner T. (2014). Bayesian modeling of flexible cognitive control. Neuroscience and Biobehavioral Reviews, 46, 30-43. [50] Jiang J. F., Beck J., Heller K., & Egner T. (2015). An insula-frontostriatal network mediates flexible cognitive control by adaptively predicting changing control demands. Nature Communications, 6(1), Article 8165. [51] Jiang J. F., Bramão I., Khazenzon A., Wang S. F., Johansson M., & Wagner A. D. (2020). Temporal dynamics of memory-guided cognitive control and generalization of control via overlapping associative memories. Journal of Neuroscience, 40(11), 2343-2356. [52] Kennerley S. W., Behrens T. E. J., & Wallis J. D. (2011). Double dissociation of value computations in orbitofrontal and anterior cingulate neurons. Nature Neuroscience, 14(12), 1581-1589. [53] Kerns J. G., Cohen J. D., Macdonald A. W., Cho R. Y., Stenger V. A., & Carter C. S. (2004). Anterior cingulate conflict monitoring and adjustments in control. Science, 303(5660), 1023-1026. [54] Kolling N., Behrens T. E. J., Mars R. B., & Rushworth, M. F. S. (2012). Neural mechanisms of foraging. Science, 336(6077), 95-98. [55] Kool W., McGuire J. T., Rosen Z. B., & Botvinick M. M. (2010). Decision making and the avoidance of cognitive demand. Journal of Experimental Psychology: General, 139(4), 665-682. [56] Larson M. J., Clayson P. E., & Baldwin S. A. (2012). Performance monitoring following conflict: Internal adjustments in cognitive control? Neuropsychologia, 50(3), 426-433. [57] Leng X. M., Yee D., Ritz H., & Shenhav A. (2021). Dissociable influences of reward and punishment on adaptive cognitive control. PLoS Computational Biology, 17(12), Article e1009737. [58] Lieder F., Shenhav A., Musslick S., & Griffiths T. L. (2018). Rational metareasoning and the plasticity of cognitive control. PLoS Computational Biology, 14(4), Article e1006043. [59] Masís J., Musslick S., & Cohen J. D. (2021). The value of learning and cognitive control allocation. In Proceedings of the 43rd Annual Meeting of the Cognitive Science Society. [60] Matsumoto M., Matsumoto K., Abe H., & Tanaka K. (2007). Medial prefrontal cell activity signaling prediction errors of action values. Nature Neuroscience, 10(5), 647-656. [61] Miller, E. K., & Cohen, J. D. (2001). An integrative theory of prefrontal cortex function. Annual Review of Neuroscience, 24(1), 167-202. [62] Muhle-Karbe P. S., Jiang J. F., & Egner T. (2018). Causal evidence for Learning-Dependent frontal lobe contributions to cognitive control. Journal of Neuroscience, 38(4), 962-973. [63] Nieuwenhuis S., Yeung N., van Den Wildenberg W., & Ridderinkhof K. R. (2003). Electrophysiological correlates of anterior cingulate function in a go/no-go task: Effects of response conflict and trial type frequency. Cognitive, Affective, Behavioral Neuroscience, 3(1), 17-26. [64] Niv, Y. (2009). Reinforcement learning in the brain. Journal of Mathematical Psychology, 53(3), 139-154. [65] Nordahl T. E., Carter C. S., Salo R. E., Kraft L., Baldo J., Salamat S., Robertson L., & Kusubov N. (2001). Anterior cingulate metabolism correlates with stroop errors in paranoid schizophrenia patients. Neuropsychopharmacology, 25(1), 139-148. [66] Olszanowski M., Bajo M. T., & Szmalec A. (2015). A conflict monitoring account of the control mechanisms involved in dual-tasking. Journal of Cognitive Psychology, 27(6), 704-714. [67] Orr J. M., Carp J., & Weissman D. H. (2012). The influence of response conflict on voluntary task switching: A novel test of the conflict monitoring model. Psychological Research, 76(1), 60-73. [68] Otto, A. R., & Daw, N. D. (2019). The opportunity cost of time modulates cognitive effort. Neuropsychologia, 123, 92-105. [69] Pires L., Leitão J., Guerrini C., & Simões M. R. (2018). Cognitive control during a spatial Stroop task: Comparing conflict monitoring and prediction of response-outcome theories. Acta Psychologica, 189, 63-75. [70] Pisauro M. A., Fouragnan E., Retzler C., & Philiastides M. G. (2017). Neural correlates of evidence accumulation during value-based decisions revealed via simultaneous EEG-fMRI. Nature Communications, 8(1), Article 15808. [71] Ratcliff R., Smith P. L., Brown S. D., & McKoon G. (2016). Diffusion decision model: Current issues and history. Trends in Cognitive Sciences, 20(4), 260-281. [72] Rescorla, R. A., & Wagner, A. R. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In A. H. Black & W. F. Prokasy (Eds.), Classical conditioning II: Current research and theory (pp. 64-99). Appleton-Century-Crofts. [73] Rudebeck P. H., Behrens T. E., Kennerley S. W., Baxter M. G., Buckley M. J., Walton M. E., & Rushworth, M. F. S. (2008). Frontal cortex subregions play distinct roles in choices between actions and stimuli. Journal of Neuroscience, 28(51), 13775-13785. [74] Rushworth M. F., Buckley M. J., Behrens T. E., Walton M. E., & Bannerman D. M. (2007). Functional organization of the medial frontal cortex. Current Opinion in Neurobiology, 17(2), 220-227. [75] Sayalı, C., & Badre, D. (2019). Neural systems of cognitive demand avoidance. Neuropsychologia, 123, 41-54. [76] Sayalı, C., & Badre, D. (2021). Neural systems underlying the learning of cognitive effort costs. Cognitive, Affective, and Behavioral Neuroscience, 21, 698-716. [77] Schultz W., Dayan P., & Montague P. R. (1997). A neural substrate of prediction and reward. Science, 275(5306), 1593-1599. [78] Shenhav A., Botvinick M. M., & Cohen J. D. (2013). The expected value of control: An integrative theory of anterior cingulate cortex function. Neuron, 79(2), 217-240. [79] Shenhav A., Musslick S., Botvinick M. M., & Cohen J. D. (2020). Misdirected vigor: Differentiating the control of value from the value of control. PsyArXiv. [80] Shenhav A., Musslick S., Lieder F., Kool W., Griffiths T. L., Cohen J. D., & Botvinick M. M. (2017). Toward a rational and mechanistic account of mental effort. Annual Review of Neuroscience, 40(1), 99-124. [81] Shenhav A., Straccia M. A., Musslick S., Cohen J. D., & Botvinick M. M. (2018). Dissociable neural mechanisms track evidence accumulation for selection of attention versus action. Nature Communications, 9(1), Article 2485. [82] Silvetti M., Seurinck R., & Verguts T. (2011). Value and prediction error in medial frontal cortex: Integrating the single-unit and systems levels of analysis. Frontiers in Human Neuroscience, 5, 75. [83] Simon, J. R., & Rudell, A. P. (1967). Auditory S-R compatibility: The effect of an irrelevant cue on information processing. Journal of Applied Psychology, 51(3), 300-304. [84] Stroop, J. R. (1935). Studies of interference in serial verbal reactions. Journal of Experimental Psychology, 18(6), 643-662. [85] Suri R. E., Bargas J., & Arbib M. A. (2001). Modeling functions of striatal dopamine modulation in learning and planning. Neuroscience, 103(1), 65-85. [86] Sutton R. S.,& Barto, A. G. (1990). Time-derivative models of pavlovian reinforcement. In M. Gabriel & J. Moore (Eds.), Learning and computational neuroscience: Foundations of adaptive networks (pp. 497-537). The MIT Press. [87] Sutton R. S.,& Barto, A. G. (2018). Reinforcement learning: An introduction MIT Press An introduction. MIT Press. [88] Vassena E., Deraeve J., & Alexander W. H. (2020). Surprise, value and control in anterior cingulate cortex during speeded decision-making. Nature Human Behaviour, 4(4), 412-422. [89] Verguts, T., & Notebaert, W. (2008). Hebbian learning of cognitive control: Dealing with specific and nonspecific adaptation. Psychological Review, 115(2), 518-525. [90] Wen T. Y., Geddert R. M., Madlon-Kay S., & Egner T. (2023). Transfer of learned cognitive flexibility to novel stimuli and task sets. Psychological Science, 34(4), 435-454. [91] Westbrook A., Lamichhane B., & Braver T. (2019). The subjective value of cognitive effort is encoded by a domain-general valuation network. The Journal of Neuroscience, 39(20), 3934-3947. [92] Wilk H. A., Ezekiel F., & Morton J. B. (2012). Brain regions associated with moment-to-moment adjustments in control and stable task-set maintenance. NeuroImage, 59(2), 1960-1967. [93] Yücel M., Pantelis C., Stuart G. W., Wood S. J., Maruff P., Velakoulis D., Pipingas A., Crowe S. F., & Tochon-Danguy H. J., & Egan G. F. (2002). Anterior cingulate activation during Stroop task performance: A PET to MRI coregistration study of individual patients with schizophrenia. American Journal of Psychiatry, 159(2), 251-254. [94] Zhang, L., & Gläscher, J. (2020). A brain network supporting social influences in human decision-making. Science Advances, 6(34), Article eabb4159.
PDF(1077 KB)
