Abstract:

Multimodal interaction has become an important research direction in artificial intelligence. The audio modality carries rich temporal dynamic information and environmental semantics, playing a significant role in scenarios such as intelligent assistants, virtual digital humans, and autonomous driving. However, most existing surveys focus on the visual modality and lack a systematic review of audio-centric cross-modal interaction, making it difficult to comprehensively present the technical roadmap and developmental bottlenecks in this field. Taking audio as the dominant modality, this paper focuses on three major interaction lines—audio-text, audio-static image, and audio-dynamic video—and extends them to audio-driven three-dimensional motion generation. It systematically summarizes representative methods, technical routes, and application progress developed in recent years. On this basis, the paper analyzes the current development status and main limitations of different research directions from the perspectives of data foundation, time-series modeling, evaluation systems, and adaptability to real scenarios, with particular emphasis on the bottlenecks faced by multimodal audio datasets in terms of scale, annotation quality, and acquisition cost. Overall, audio-driven cross-modal interaction has achieved significant progress in unified representation learning, generative modeling, and real-scenario applications, and continues to advance toward stronger multi-granularity alignment capabilities, higher-quality data support, more robust evaluation systems, and more efficient deployment methods.

Key words: audio cross-modal interaction, cross-modal retrieval, multimodal learning, audio-visual alignment, data analysis