Social cognition evaluation framework and question-answering benchmark for large language model

doi:10.3969/j.issn.1000-5641.2026.04.011

Abstract

Abstract:

Considering the increasing interactions between large language model (LLM) and humans, the assessment of their proficiency in social cognition, a core component of understanding mutual perspectives, is crucial. Drawing from human social cognition theories, this study created an evaluation framework rooted in the theory of mind (ToM), which focuses on the ability to infer the mental states of others. By employing this framework, a social cognition assessment for LLM was developed and presented as a benchmark for multiple-choice questions, inspired by psychological experiments. By applying this benchmark to various commercial models and examining the development of LLM's ToM abilities, our findings indicate that while LLM exhibit nascent ToM skills, there is substantial room for improvement. Furthermore, our study revealed similarities between the development of ToM in LLM and humans.

Key words: large language model (LLM), theory of mind, social cognition, question-answering benchmark

CLC Number:

TP39

Yiming MA, Xin LIN, Qin NI, Yangze YU, Ciping DENG, Liang HE. Social cognition evaluation framework and question-answering benchmark for large language model[J]. J* E* C* N* U* N* S*, 2026, 2026(4): 102-111.

Figures/Tables 7

Fig.1

Fig.2

Table 1

Fig.3

Fig.4

Table 2

Table 3

References 20

1	OpenAI, Achiam J, Adler S, et al. GPT-4 technical report [PP/OL]. V6. arXiv (2024-03-04)[2024-10-19]. https://arxiv.org/abs/2303.08774.
2	Langley C, Cirstea B I, Cuzzolin F, et al.. Theory of mind and preference learning at the interface of cognitive science, neuroscience, and AI: A review. Frontiers in Artificial Intelligence, 2022, 5, 778852.
3	Wang W S, Wang L T, Zhang C Y, et al.. Social interactions for autonomous driving: A review and perspectives. Foundations and Trends in Robotics, 2022, 10 (3/4): 198- 377.
4	Premack D, Woodruff G.. Does the chimpanzee have a theory of mind?. Behavioral and Brain Sciences, 1978, 1 (4): 515- 526.
5	Grant E, Nematzadeh A, Griffiths T L. How can memory-augmented neural networks pass a false-belief task? [C]//Proceedings of the Annual Meeting of the Cognitive Science Society. 2017, 39: 427-432.
6	Le M, Boureau Y L, Nickel M. Revisiting the evaluation of theory of mind through question answering [C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, 2019: 5872-5877.
7	Nematzadeh A, Burns K, Grant E, et al. Evaluating theory of mind in question answering [PP/OL]. arXiv (2018-08-28)[2024-10-19]. https://arxiv.org/abs/1808.09352.
8	Bubeck S, Chandrasekaran V, Eldan R, et al. Sparks of artificial general intelligence: Early experiments with GPT-4 [PP/OL]. V5. arXiv (2023-04-13)[2024-10-19]. https://arxiv.org/abs/2303.12712.
9	Kosinski M. Evaluating large language models in theory of mind tasks [PP/OL]. V7. arXiv (2023-02-04)[2024-10-19]. https://arxiv.org/abs/2302.02083.
10	Trott S, Jones C, Chang T, et al.. Do large language models know what humans know?. Cognitive Science, 2023, 47 (7): e13309.
11	Baron-Cohen S, Leslie A M, Frith U.. Does the autistic child have a “theory of mind”?. Cognition, 1985, 21 (1): 37- 46.
12	Perner J, Wimmer H.. “John thinks that Mary thinks that…” attribution of second-order beliefs by 5- to 10-year-old children. Journal of Experimental Child Psychology, 1985, 39 (3): 437- 471.
13	Bennett M, Galpert L.. Children’s understanding of multiple desires. International Journal of Behavioral Development, 1993, 16 (1): 15- 33.
14	Repacholi B M, Gopnik A.. Early reasoning about desires: Evidence from 14- and 18-month-olds. Developmental Psychology, 1997, 33 (1): 12- 21.
15	Killen M, Lynn Mulvey K, Richardson C, et al.. The accidental transgressor: morally-relevant theory of mind. Cognition, 2011, 119 (2): 197- 215.
16	Sap M, Rashkin H, Chen D, et al. Social IQa: commonsense reasoning about social interactions [C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, 2019: 4463-4473.
17	毛媛媛, 林欣, 倪琴, 等.. 面向心智理论发展的视频问答研究. 华东师范大学学报(自然科学版), 2025 (6): 46- 52.
18	Beaudoin C, Leblanc É, Gagner C, et al.. Systematic review and inventory of theory of mind measures for young children. Frontiers in Psychology, 2019, 10, 2905.
19	Yott J, Poulin-Dubois D.. Are infants’ theory-of-mind abilities well integrated? Implicit understanding of intentions, desires, and beliefs. Journal of Cognition and Development, 2016, 17 (5): 683- 698.
20	Moghaddam S R, Honey C J. Boosting theory-of-mind performance in large language models via prompting [PP/OL]. V3. arXiv (2023-04-26)[2024-10-19]. https://arxiv.org/abs/2304.11490.

心智理论	问题数	社会功能	问题数	语言	问题数	来源	问题数
意图	12427	社会情绪	16135	中文	30842	生成	39765
愿望	8964	社会行为	24591	英文	18971	爬取	10048
信念	22078	其他	9087
混合心理	6344

LLM	In	DI	IF	De	DD	MD	DC	Be	CFB	LFB	IFB	SB	Mul
Ernie-bot	68.29	72.79	75.00	89.62	79.31	66.08	57.39	59.59	56.15	60.20	62.61	45.94	67.50
Spark	84.87	78.67	84.48	95.28	86.20	83.62	77.58	79.14	80.00	75.51	77.57	82.88	84.06
ChatGLM	81.28	77.20	88.79	87.73	88.50	83.62	77.58	78.72	75.38	83.67	89.71	80.18	81.56
ChatGPT	80.25	74.26	71.42	85.84	80.45	81.89	72.41	77.30	80.00	80.61	69.15	72.97	82.50
GPT-4	87.94	84.55	94.82	100	93.10	86.20	84.48	88.44	84.61	82.65	87.85	79.27	90.93
qwen	78.53	81.40	80.73	100	92.41	84.26	78.50	74.87	81.60	75.28	73.73	71.15	86.49
Llama3	84.06	80.88	95.69	100	94.25	86.09	74.78	83.70	82.03	78.57	84.11	60.00	84.74
GPT-4+FS	90.51	85.29	93.97	100	89.66	91.38	86.21	91.34	95.35	88.78	95.33	81.08	93.75
平均	81.97	79.38	85.61	94.81	87.99	82.89	76.12	79.14	79.39	78.16	80.01	71.68	83.94

LLM	EI	EE	MiE	HE	ME	BI	BE	EL	WL	Hu	Ir
Ernie-bot	74.36	55.81	72.72	60.37	53.61	75.15	64.61	56.60	61.94	77.31	39.51
Spark	86.64	80.62	88.88	77.35	74.85	86.58	79.95	83.01	82.45	88.65	75.00
ChatGLM	86.64	75.19	79.79	74.52	72.45	85.67	84.53	73.58	81.57	78.35	66.93
ChatGPT	84.27	79.84	76.76	86.79	71.25	83.23	76.58	71.69	78.07	77.31	68.54
GPT-4	96.58	84.49	90.90	86.79	81.43	92.68	87.76	70.75	92.10	94.84	77.41
qwen	92.18	81.45	92.30	78.00	64.29	83.78	77.33	68.32	83.65	87.91	70.80
Llama3	93.04	81.89	83.67	79.25	83.23	92.02	82.13	70.09	81.42	86.60	67.74
GPT-4+FS	96.89	95.35	90.90	92.45	89.22	94.51	89.74	79.25	92.10	95.88	81.45
平均	88.83	79.33	84.49	79.44	73.79	86.70	80.33	71.66	81.66	85.86	68.42

[1]	Xinyue LYU, Xinli HUANG. Cold-start problem of federated recommendation systems based on conditional variational autoencoder [J]. J* E* C* N* U* N* S*, 2025, 2025(6): 53-62.
[2]	Sijia ZHAO, Fanyu HAN, Wei WANG. Research on the GitHub developer geographic location prediction method based on multi-dimensional feature fusion [J]. J* E* C* N* U* N* S*, 2025, 2025(5): 1-13.
[3]	Dexin HE, Fanyu HAN, Wei WANG. Application and evaluation of large language models in open source project topic annotation [J]. J* E* C* N* U* N* S*, 2025, 2025(5): 14-24.
[4]	Mingdong YOU, Jiaheng PENG, Fanyu HAN, Wei WANG. A DTA based activity evaluation method for high star GitHub repositories [J]. J* E* C* N* U* N* S*, 2025, 2025(5): 140-150.
[5]	Shengyu ZHAO, Wei WANG, Fanyu HAN, Jiaheng PENG, Lan YOU. Open source evaluatology: A framework and methodology for evaluating open source ecosystems based on evaluatology [J]. J* E* C* N* U* N* S*, 2025, 2025(5): 151-161.
[6]	Xiaowei CHEN, Wei WANG, Fanyu HAN, Guanglei BAO, Fei DONG, Hao HUO, Chen LIU. OSS Insight: A platform for open source ecosystem spatiotemporal data analysis and insights [J]. J* E* C* N* U* N* S*, 2025, 2025(5): 170-182.
[7]	Ruiyang PANG, Xuesong LU. Interactive data structure and algorithm visualization based on AI agents [J]. J* E* C* N* U* N* S*, 2025, 2025(5): 32-42.
[8]	Linna XIE, Xuesong LU. Student employment prediction for digital jobs based on behavior in open-source communities [J]. J* E* C* N* U* N* S*, 2025, 2025(5): 87-98.
[9]	Yufei FU, Zesong XU, Anhao FENG, Tongquan WEI. Online grant prediction for mobile networks based on traffic perception [J]. J* E* C* N* U* N* S*, 2025, 2025(4): 1-14.
[10]	Jiefang CHEN, Chang HUANG. Parallax image mosaic based on matching points and thin plate splines function model optimization [J]. J* E* C* N* U* N* S*, 2025, 2025(4): 15-27.
[11]	Xingxing XU, Chang HUANG. Container lock pins recognition based on a self-built dataset with high completeness [J]. J* E* C* N* U* N* S*, 2025, 2025(4): 28-37.
[12]	Zheng HU, Chang HUANG. A high robust fast video steganography algorithm in lossy channels [J]. J* E* C* N* U* N* S*, 2025, 2025(4): 38-48.
[13]	Jimi HU, Weibing WAN, Feng CHENG, Yuming ZHAO. Rule extraction and reasoning for fusing relation and structure encoding [J]. J* E* C* N* U* N* S*, 2025, 2025(1): 97-110.
[14]	Houlong FAN, Ailian FANG, Xin LIN. Knowledge graph completion by integrating textual information and graph structure information [J]. J* E* C* N* U* N* S*, 2025, 2025(1): 111-123.
[15]	Yiping LIANG, Luwei XIAO, Linlin WANG. Label-perception augmented causal analysis of mental health over social media [J]. J* E* C* N* U* N* S*, 2025, 2025(1): 124-137.