大模型时代的数据可视化理解

doi:10.3969/j.issn.1000-5641.2026.05.010

摘要/Abstract

摘要：

多模态大模型的发展推动可视化图表从静态信息载体转变为模型理解、推理、生成和安全评估的重要对象. 围绕大模型时代的可视化与可视分析研究, 本文系统梳理了可视化图表理解和误导性可视化分析两个方向的研究进展. 前者关注模型如何从位图、矢量图和 D3 图等不同输入形态中恢复图表结构、数据关系和视觉编码, 并进一步支持图表问答、视觉定位、风格化设计以及标题与描述文本生成; 后者关注图表在数据选择、视觉编码、文本解释和传播语境中可能造成的误导, 重点讨论误导性可视化的类型划分、视觉语言模型评估、自动检测、可解释分析、交互式纠正与安全防御. 现有研究表明, 图表理解和误导性可视化分析共同依赖数据、视觉、文本和语义之间的可靠对齐. 未来应进一步发展统一图表表征、细粒度证据定位、可信生成、人机协同纠错以及面向大模型安全的可视化防御机制.

关键词: 数据可视化, 图表理解, 多模态大模型, 误导性可视化, 可视分析

Abstract:

The development of multimodal large models has transformed visualization charts from static information carriers into important objects for model understanding, reasoning, generation, and safety evaluation. This review summarizes recent advances in visualization and visual analytics in the era of large models, focusing on two major directions: visualization chart understanding and misleading visualization analysis. The former investigates how models recover chart structures, data relations, and visual encodings from different input forms, including bitmap images, vector graphics, and D3 charts, and further supports chart question answering, visual grounding, style design, and the generation of titles and descriptions. The latter examines how misleading charts emerge from data selection, visual encoding, textual explanation, and communication context, with emphasis on taxonomy, evaluation of visual language models, automatic detection, explainable analysis, interactive correction, and safety defense. Existing studies indicate that both chart understanding and misleading visualization analysis rely on reliable alignment among data, visual structures, text, and semantics. Future research should develop unified chart representations, fine-grained evidence grounding, trustworthy generation, human-AI collaborative correction, and visualization defense mechanisms to enhance the safety of large models.

Key words: data visualization, chart understanding, multimodal large models, misleading visualization, visual analytics

中图分类号:

TP37

王长波, 宋思程. 大模型时代的数据可视化理解[J]. 华东师范大学学报（自然科学版）, 2026, 2026(5): 119-132.

Changbo WANG, Sicheng SONG. Data visualization understanding in the era of large models[J]. J* E* C* N* U* N* S*, 2026, 2026(5): 119-132.

图/表 4

参考文献 59

1	Savva M, Kong N, Chhajta A, et al. ReVision: automated classification, analysis and redesign of chart images [C]//Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology. ACM, 2011: 393-402.
2	Luo J Y, Li Z K, Wang J P, et al. ChartOCR: data extraction from charts images via a deep hybrid framework [C]//2021 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2021: 1916-1924.
3	Liu F Y, Eisenschlos J, Piccinno F, et al. DePlot: one-shot visual language reasoning by plot-to-table translation [C]//Findings of the Association for Computational Linguistics: ACL 2023. Association for Computational Linguistics, 2023: 10381-10399.
4	Li Z, Li D, Guo Y K, et al. ChartGalaxy: a dataset for infographic chart understanding and generation [PP/OL]. V5. arXiv (2025-10-16)[2026-07-09]. https://doi.org/10.48550/arXiv.2505.18668.
5	Bostock M, Ogievetsky V, Heer J.. D³ data-driven documents. IEEE Transactions on Visualization and Computer Graphics, 2011, 17 (12): 2301- 2309.
6	Siegel N, Horvitz Z, Levin R, et al. FigureSeer: parsing result-figures in research papers [C]//Computer Vision–ECCV 2016. Cham: Springer, 2016: 664-680.
7	Jung D, Kim W, Song H, et al. ChartSense: interactive data extraction from chart images [C]//Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 2017: 6706-6717.
8	Masson D, Malacria S, Vogel D, et al. ChartDetective: easy and accurate interactive data extraction from complex vector charts [C]//Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. ACM, 2023: 1-17.
9	Chen C, Lee B, Wang Y H, et al.. Mystique: deconstructing SVG charts for layout reuse. IEEE Transactions on Visualization and Computer Graphics, 2024, 30 (1): 447- 457.
10	Snyder L S, Heer J.. DIVI: dynamically interactive visualization. IEEE Transactions on Visualization and Computer Graphics, 2024, 30 (1): 403- 413.
11	Chen C, Bako H K, Yu P H, et al.. VisAnatomy: an SVG chart corpus with fine-grained semantic labels. IEEE Transactions on Visualization and Computer Graphics, 2026, 32 (1): 560- 570.
12	Xu Z Z, Wall E. Exploring the capability of LLMs in performing low-level visual analytic tasks on SVG data visualizations [C]//2024 IEEE Visualization and Visual Analytics (VIS). IEEE, 2024: 126-130.
13	Zou B C, Cai M, Zhang J R, et al. VGBench: evaluating large language models on vector graphics understanding and generation [C]//Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2024: 3647-3659.
14	Satyanarayan A, Moritz D, Wongsuphasawat K, et al.. Vega-Lite: a grammar of interactive graphics. IEEE Transactions on Visualization and Computer Graphics, 2017, 23 (1): 341- 350.
15	Harper J, Agrawala M. Deconstructing and restyling D3 visualizations [C]//Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology. ACM, 2014: 253-262.
16	Yang C, Shi C F, Liu Y X, et al. ChartMimic: evaluating LMM’s cross-modal reasoning capability via chart-to-code generation [PP/OL]. V2. arXiv (2025-02-28)[2026-07-09]. https://arxiv.org/abs/2406.09961.
17	Xie L, Lin Y N, Liu C, et al.. DataWink: reusing and adapting SVG-based visualization examples with large multimodal models. IEEE Transactions on Visualization and Computer Graphics, 2026, 32 (1): 824- 834.
18	Kafle K, Price B, Cohen S, et al. DVQA: understanding data visualizations via question answering [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2018: 5648-5656.
19	Masry A, Long D X, Tan J Q, et al. ChartQA: a benchmark for question answering about charts with visual and logical reasoning [C]//Findings of the Association for Computational Linguistics: ACL 2022. Association for Computational Linguistics, 2022: 2263-2279.
20	Methani N, Ganguly P, Khapra M M, et al. PlotQA: reasoning over scientific plots [C]//2020 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2020: 1516-1525.
21	Kantharaj S, Do X L, Leong R T, et al. OpenCQA: open-ended question answering with charts [C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2022: 11817-11837.
22	Carbune V, Mansoor H, Liu F Y, et al. Chart-based reasoning: transferring capabilities from LLMs to VLMs [C]//Findings of the Association for Computational Linguistics: NAACL 2024. Association for Computational Linguistics, 2024: 989-1004.
23	Zeng X C, Lin H C, Ye Y L, et al.. Advancing multimodal large language models in chart question answering with visualization-referenced instruction tuning. IEEE Transactions on Visualization and Computer Graphics, 2025, 31 (1): 525- 535.
24	Wei J X, Xu N, Zhu J N, et al. ChartMind: a comprehensive benchmark for complex real-world multimodal chart question answering [C]//Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2025: 4555-4569.
25	Poco J, Mayhua A, Heer J.. Extracting and retargeting color mappings from bitmap images of visualizations. IEEE Transactions on Visualization and Computer Graphics, 2018, 24 (1): 637- 646.
26	Shi Y, Liu P, Chen S J, et al.. Supporting expressive and faithful pictorial visualization design with visual style transfer. IEEE Transactions on Visualization and Computer Graphics, 2023, 29 (1): 236- 246.
27	Satyanarayan A, Heer J. Lyra: an interactive visualization design environment [J]. Computer Graphics Forum, 2014, 33(3): 351-360.
28	Kim N W, Schweickart E, Liu Z C, et al.. Data-driven guides: supporting expressive design for information graphics. IEEE Transactions on Visualization and Computer Graphics, 2017, 23 (1): 491- 500.
29	Liu Z C, Thompson J, Wilson A, et al. Data Illustrator: augmenting vector design tools with lazy data binding for expressive visualization authoring [C]//Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 2018: 1-13.
30	Xia H J, Henry Riche N, Chevalier F, et al. DataInk: direct and creative data-oriented drawing [C]//Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 2018: 1-13.
31	Ren D H, Lee B, Brehmer M.. Charticulator: interactive construction of bespoke chart layouts. IEEE Transactions on Visualization and Computer Graphics, 2019, 25 (1): 789- 799.
32	Ma R X, Mei H H, Guan H H, et al.. LADV: deep learning assisted authoring of dashboard visualizations from images and sketches. IEEE Transactions on Visualization and Computer Graphics, 2021, 27 (9): 3717- 3732.
33	Mittal V O, Moore J D, Carenini G, et al.. Describing complex charts in natural language: a caption generation system. Computational Linguistics, 1998, 24 (3): 431- 467.
34	Choi J, Jo J. Intentable: a mixed-initiative system for intent-based chart captioning [C]//2022 IEEE Visualization and Visual Analytics (VIS). IEEE, 2022: 40-44.
35	Kantharaj S, Leong R T, Lin X, et al. Chart-to-text: a large-scale benchmark for chart summarization [C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2022: 4005-4023.
36	Tang B, Boggust A, Satyanarayan A. VisText: a benchmark for semantically rich chart captioning [C]//Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2023: 7268-7298.
37	Mahinpei A, Kostic Z, Tanner C. LineCap: line charts for data visualization captioning models [C]//2022 IEEE Visualization and Visual Analytics (VIS). IEEE, 2022: 35-39.
38	Wang H W, Hoffswell J, Thazin Thane S M, et al.. How aligned are human chart takeaways and LLM predictions? A case study on bar charts with varying layouts. IEEE Transactions on Visualization and Computer Graphics, 2025, 31 (1): 536- 546.
39	Wainer H. How to display data badly[J]. The American Statistician, 1984, 38(2): 137-147.
40	Lo L Y H, Gupta A, Shigyo K, et al. Misinformed by visualization: what do we learn from misinformative visualizations?[J]. Computer Graphics Forum, 2022, 41(3): 515-525.
41	Lisnic M, Polychronis C, Lex A, et al. Misleading beyond visual tricks: how people actually lie with charts [C]//Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. ACM, 2023: 1-21.
42	Lisnic M, Lex A, Kogan M. “Yeah, this graph doesn’t show that”: analysis of online engagement with misleading data visualizations [C]//Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, 2024: 1-14.
43	Pandey S, Ottley A. Benchmarking visual language models on standardized visualization literacy tests [J]. Computer Graphics Forum, 2025, 44(3): e70137.
44	Alexander J, Nanda P, Yang K C, et al. Can GPT-4 models detect misleading visualizations? [C]//2024 IEEE Visualization and Visual Analytics (VIS). IEEE, 2024: 106-110.
45	Lo L Y, Qu H M.. How good (or bad) are LLMs at detecting misleading visualizations?. IEEE Transactions on Visualization and Computer Graphics, 2025, 31 (1): 1116- 1125.
46	Bharti S, Cheng S Y, Rho J, et al. CHARTOM: a visual theory-of-mind benchmark for LLMs on misleading charts [PP/OL]. V3. arXiv (2025-06-29)[2026-07-09]. https://doi.org/10.48550/arXiv.2408.14419.
47	Chen Z X, Song S C, Shum K, et al. Unmasking deceptive visuals: benchmarking multimodal large language models on misleading chart question answering [C]//Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2025: 13756-13789.
48	Tonglet J, Zimny J, Tuytelaars T, et al. Is this chart lying to me? Automating the detection of misleading visualizations [C]//Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). ACL, 2026: 8823-8844.
49	Kim M H, Song Y M, Kim Y, et al. Automated pipeline for detecting and analyzing misleading visual elements [C]//2025 IEEE 18th Pacific Visualization Conference (PacificVis). IEEE, 2025: 346-351.
50	Akhtar M, Subedi N, Gupta V, et al. ChartCheck: explainable fact-checking over real-world chart images [C]//Findings of the Association for Computational Linguistics: ACL 2024. Association for Computational Linguistics, 2024: 13921-13937.
51	Tonglet J, Tuytelaars T, Moens M F, et al. Protecting multimodal large language models against misleading visualizations [C]//Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). ACL, 2026: 8329-8349.
52	Zhang Y J, Li Y F, Sheng R, et al. Navigating the mirage: a dual-path agentic framework for robust misleading chart question answering [PP/OL]. arXiv (2026-03-30)[2026-07-09]. https://doi.org/10.48550/arXiv.2603.28583.
53	Das A K, Mueller K.. MisVisFix: an interactive dashboard for detecting, explaining, and correcting misleading visualizations using large language models. IEEE Transactions on Visualization and Computer Graphics, 2026, 32 (1): 134- 144.
54	Ortiz-Barajas J G, Tonglet J, Gupta V, et al. ChartAttack: testing the vulnerability of LLMs to malicious prompting in chart generation [PP/OL]. V3. arXiv (2026-06-04)[2026-07-09]. https://doi.org/10.48550/arXiv.2601.12983.
55	Zhang P Y, Li C H, Wang C B.. VisCode: embedding information in visualization images using encoder-decoder network. IEEE Transactions on Visualization and Computer Graphics, 2021, 27 (2): 326- 336.
56	Fu J Y, Zhu B, Cui W W, et al.. Chartem: reviving chart images with data embedding. IEEE Transactions on Visualization and Computer Graphics, 2021, 27 (2): 337- 346.
57	Ye H Y, Li C H, Li Y, et al.. InvVis: large-scale data embedding for invertible visualization. IEEE Transactions on Visualization and Computer Graphics, 2024, 30 (1): 1139- 1149.
58	Ye H Y, Chen J T, Zhang S Z, et al.. VisGuard: securing visualization dissemination through tamper-resistant data retrieval. IEEE Transactions on Visualization and Computer Graphics, 2026, 32 (1): 1295- 1305.
59	Song S C, Zhang Y J, Chen Z X, et al.. VizDefender: unmasking visualization tampering through proactive localization and intent inference. IEEE Transactions on Visualization and Computer Graphics, 2026, 32 (6): 4720- 4730.