Dynamic Reinforcement Learning Modeling and Strategy Evolution in Pigeon Sequential Decision-making

NAVIGATE

Table of Contents

STATISTICS

Viewed91

Downloads107

Dynamic Reinforcement Learning Modeling and Strategy Evolution in Pigeon Sequential Decision-making

PDF下载 (107)

[1]LI Zhihui,MA Ying,SHANG Zhigang,et al.Dynamic Reinforcement Learning Modeling and Strategy Evolution in Pigeon Sequential Decision-making[J].Journal of Zhengzhou University (Engineering Science),2026,47(XX):1-7.[doi:10.13705/j.issn.1671-6833.2025.05.026]

Copy

Journal of Zhengzhou University (Engineering Science)[ISSN 1671-6833/CN 41-1339/T] Volume: 47 Number of periods: 2026 XX Page number: 1-7 Column: Public date: 2026-09-10

Title:: Dynamic Reinforcement Learning Modeling and Strategy Evolution in Pigeon Sequential Decision-making

Author(s):: LI Zhihui¹; 2 ; MA Ying¹; 2 ; SHANG Zhigang¹; 2 ; YANG Lifang¹; 2; 3∗; 1. School of Electrical and Inform ation Engineering, Zhengzhou University, Zhengzhou 450001, China; 2. Henan Key Laboratory of Brain Science and Brain Computer Interface Technology, Zhengzhou 450001, China; 3. The Affiliated Encephalopathy Hospital of Zhengzhou University, Zhumadian 463000, China

Keywords:: sequential decision-making; reinforcement learning; pigeons; learning strategies; Model-Based; Model-Free

CLC:: Q811. 211

DOI:: 10.13705/j.issn.1671-6833.2025.05.026

Abstract:: To maximize future rewards, organisms must flexibly adjust their learning strategies within complex environments. To investigate how learning strategies dynamically evolve during sequential decision-making, we used pigeons—a model species with robust cognitive capabilities—in a two-step sequential decision-making task. Behavioral data were collected throughout the entire learning process, from initial exploration to proficient performance. We developed two dynamic reinforcement learning (RL) models: a reward prediction error-driven Model-Free (MF) model and a state-transition relationship-driven Model-Based (MB) model. Using experimental data, we fitted these models and systematically analyzed the dynamic changes in key learning parameters, including learning rate (reflecting the speed of new information acquisition), discount factor (indicating the valuation of future rewards), and the inverse temperature parameter (representing choice certainty). Model comparisons revealed that pigeons predominantly utilized an MB strategy in early learning stages, focusing on acquiring relationships between states to form accurate value representations. With accumulated experience, pigeons progressively shifted toward the MF strategy, directly utilizing established value predictions for decision-making. Furthermore, analysis of model parameters showed that the learning rate gradually decreased, while both discount factor and inverse temperature increased over the learning period. These changes indicate that pigeons progressively place greater emphasis on future rewards and decision certainty, illustrating a natural shift from environmental exploration to exploitation of acquired knowledge. This study not only elucidates the mechanisms underlying adaptive strategy adjustments in biological systems during sequential decision-making but also provides valuable biological insights for parameter optimization in artificial reinforcement learning models.

References:: [1] DAYAN P, DAW N D. Decision theory, reinforcement learning, and the brain[J]. Cognitive, Affective, & Behavioral Neuroscience, 2008, 8(4): 429-453.
[2] GUPTA N, AHIRWAL M K, ATULKAR M. Development of human decision making model with consideration of human factors through reinforcement learning and prospect utility theory[J]. Journal of Experimental & Theoretical Artificial Intelligence, 2024, 36(7): 1003-1019.
[3] SUTTON R S, BARTO A G. Reinforcement learning: an introduction [M]. London, England: The MIT Press, 2018.
[4] MILLER K J, VENDITTO S J C. Multi-step planning in the brain[J]. Current Opinion in Behavioral Sciences, 2021, 38: 29-39.
[5] DEHAENE S, SIGMAN M. From a single decision to a multi-step algorithm[J]. Current Opinion in Neurobiology, 2020, 62: 155-166.
[6] 张倩倩. 面向人机序贯决策的混合智能方法研究[D]. 合肥: 中国科学技术大学, 2021.
ZHANG Q Q. Research on hybrid intelligent method for man-machine sequential decision-making[D]. Hefei: University of Science and Technology of China, 2021.
[7] MATTAR M G, THOMPSON-SCHILL L S, BASSETT D S. The network architecture of value learning[J]. Network Neuroscience, 2018, 2(2): 128-149.
[8] 王东署, 杨凯. 基于状态转移学习的机器人行为决策认知模型[J]. 郑州大学学报(工学版), 2021,42(6): 7-13.
WANG D S, YANG K. Behavior decision-making cognitive model of mobile robot based on state transfer learning[J]. Journal of Zhengzhou University (Engineering Science), 2021,42(6): 7-13.
[9] 蒲慕明. 跨学科开启头脑风暴促进学科交叉与融合[J]. 科学通报, 2023, 68(35): 4749-4750.
PU M M. Initiate interdisciplinary brainstorming, promote cross-disciplinary integration in neuroscience[J]. Chinese Science Bulletin, 2023, 68(35): 4749-4750.
[10] Huang J, Zhang Z, Ruan X. An improved dyna-Q algorithm inspired by the forward prediction mechanism in the rat brain for mobile robot path planning[J]. Biomimetics, 2024, 9(6): 315.
[11] Rescorla R A. A theory of pavlovian conditioning: variations in the effectiveness of reinforcement[J]. Current Research & Theory, 1972, 64-99.
[12] SUTTON R S. Learning to predict by the methods of temporal differences[J]. Machine Learning, 1988, 3(1): 9-44.
[13] 李琳, 李玉泽, 张钰嘉, 等. 基于多估计器平均值的深度确定性策略梯度算法[J]. 郑州大学学报(工学版), 2022, 43(2): 15-21.
LI L, LI Y Z, ZHANG Y J, et al. Deep deterministic policy gradient algorithm based on mean of multiple estimators[J]. Journal of Zhengzhou University (Engineering Science), 2022, 43(2): 15-21.
[14] 师黎, 陶梦妍, 李志辉. 鸽子强化学习过程中内部学习状态的动态建模研究[J]. 科学技术与工程, 2017, 17(13): 120-125.
SHI L, TAO M Y, LI Z H. Dynamic modeling of internal cognitive status of pigeon in the process of reinforcement learning[J]. Science Technology and Engineering, 2017, 17(13): 120-125.
[15] DAW N D, NIV Y, DAYAN P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control[J]. Nature Neuroscience, 2005, 8(12): 1704-1711.
[16] DOLL B B, DUNCAN K D, SIMON D A, et al. Model-based choices involve prospective neural activity[J]. Nature Neuroscience, 2015, 18(5): 767-772.
[17] MOMENNEJAD I. Learning structures: predictive representations, replay, and generalization[J]. Current Opinion in Behavioral Sciences, 2020, 32: 155-166.
[18] ESBER G R, SCHOENBAUM G, IORDANOVA M D. The rescorla-Wagner model: it is not what you think it is[J]. Neurobiology of Learning and Memory, 2025, 217: 108021.
[19] YANG L F, JIN F L, YANG L, et al. The hippocampus in pigeons contributes to the model-based valuation and the relationship between temporal context states[J]. Animals, 2024, 14(3): 431.
[20] VENDITTO S J C, MILLER K J, BRODY C D, et al. Dynamic reinforcement learning reveals time-dependent shifts in strategy during reward learning[J]. bioRxiv, 2024: 2024.02.28.582617.

Similar References:

Memo

Last Update: 2026-01-15