Multimodal CoT Prompting

Multimodal Chain-of-Thought Prompting (MCoT) is an extension of the Chain-of-Thought (CoT) method to tasks involving multiple data types (modalities). In MCoT models, language and other modalities, such as vision or tabular data analysis, participate in a unified, step-by-step inference process to solve complex problems.

This approach emerged with the development of Multimodal Large Language Models (MLLMs), which are capable of simultaneously processing text, images, audio, and video. MCoT enables models to generate interpretable, step-by-step explanations that integrate information from different sources, thereby enhancing their accuracy and transparency.

Background: From Textual to Multimodal CoT

Chain-of-Thought (CoT) in Text

Initially, the Chain-of-Thought (CoT) method was proposed by Google researchers in 2022 for text-based large language models (LLMs). The idea is to prompt the model to generate a sequence of intermediate reasoning steps before providing the final answer. Adding examples of step-by-step solutions to the prompt (few-shot prompting) significantly improved the ability of LLMs to solve tasks requiring arithmetic, logical, and commonsense reasoning, and increased the overall accuracy and reliability of the models.

Transition to Multimodality

The success of textual CoT spurred efforts to extend it to multimodal scenarios. With the emergence of MLLMs like Microsoft's Kosmos-1, which are trained on both text and images simultaneously, it became possible to integrate CoT logic with multimodal perception. Experiments showed that such models can employ step-by-step reasoning, considering both textual and visual inputs, demonstrating the feasibility of combining logic and perception.

Key Approaches and Methodologies

Since 2023, several methods have been proposed to implement Multimodal CoT.

Two-Stage Multimodal-CoT (Zhang et al.)

One of the first methods, proposed in 2023, uses a two-stage scheme:

1. Rationale Generation: In the first step, the model generates a textual chain of thought (rationale) based on multimodal information (e.g., text and an image).
2. Answer Formulation: In the second step, the model provides the final answer based on the generated rationale.

This separated approach allowed a model with fewer than 1 billion parameters to achieve state-of-the-art performance on the ScienceQA scientific dataset, surpassing even the large GPT-3.5 model. A reduction in hallucinations was also noted.

Compositional CoT

Presented at the CVPR 2024 conference, this method focuses on visual-textual tasks and proposes generating a structured representation of the image as an intermediate step. First, the MLLM generates a scene description in the form of a scene graph, identifying objects and the relationships between them. This structured description is then included in the prompt for the final answer. This approach allows the LLM to more deeply consider the compositional relationships between objects and improves performance on tasks such as complex scene description and visual question answering.

Duty-Distinct CoT (DDCoT)

This method, presented at NeurIPS 2023, proposes dividing responsibilities among different system components:

• The language model is responsible for logical reasoning and information integration.
• The visual subsystem (a computer vision model) is responsible for recognizing the content of the image.

This "binary prompting" enables "critical thinking": the LLM evaluates and utilizes visual information obtained from a specialized vision module. The DDCoT approach has enabled the generation of more general and explainable reasoning and has significantly increased accuracy on multimodal scientific QA tasks.

Other MCoT Variants

Other approaches adapted for specific modalities are also being actively developed:

• Dual CoT: A parallel bidirectional reasoning scheme.
• Audio-CoT: An adaptation of chain-of-thought for tasks related to audio and speech.
• Video-of-Thought: A technique for step-by-step analysis of video data.

Applications and Results

Multimodal CoT prompting has demonstrated effectiveness in numerous areas where the integration of diverse information is required.

• Education and Scientific QA: Enables systems to answer questions with diagrams and illustrations by providing a detailed explanation of the solution (e.g., on the ScienceQA dataset).
• Autonomous Driving and Robotics: Helps to sequentially interpret data from LiDAR, sensors, and cameras, improving scene understanding and decision-making for agents.
• Embodied AI: Provides more reliable action planning for systems interacting with the physical world based on visual and textual cues.
• Medicine and Healthcare: Combining medical images (e.g., X-rays) with textual descriptions improves diagnostic accuracy and the explainability of AI conclusions.

Challenges and Future Directions

Despite significant progress, the multimodal use of CoT remains a complex research problem.

• Lack of Labeled Data: Training models to generate correct multimodal reasoning requires large datasets with detailed explanations, which are labor-intensive to create.
• Flexibility and Generalizability: Methods tuned for one type of task (e.g., text + image) may not transfer well to other combinations of modalities.
• Optimal Integration: It remains an open question how to best integrate different modalities into a unified reasoning process so that it genuinely enhances the model's understanding rather than simply lengthening the output.
• Standardization and Evaluation: There is a need to develop standardized benchmarks for the objective evaluation and comparison of different MCoT approaches.

Achieving multimodal AI that approaches general intelligence capabilities will require further innovations in MCoT methods that account for the specifics of how different sensors perceive the world.

Links

• Multimodal CoT Overview in Prompting Guide
• "Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey" — a detailed scientific review

Further Reading

• Zhang, Z. et al. (2023). Multimodal Chain-of-Thought Reasoning in Language Models. arXiv:2302.00923.
• Mitra, C. et al. (2024). Compositional Chain-of-Thought Prompting for Large Multimodal Models. CVPR 2024. PDF.
• Zheng, G. et al. (2023). DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models. arXiv:2310.16436.
• Huang, S. et al. (2023). Language Is Not All You Need: Aligning Perception with Language Models (Kosmos-1). arXiv:2302.14045.
• Wang, Y. et al. (2025). Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey. arXiv:2503.12605.
• Ma, Z. et al. (2025). Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Models. arXiv:2501.07246.
• Li, J. et al. (2024). DCoT: Dual Chain-of-Thought Prompting for Large Multimodal Models. OpenReview:0saecDOdh2.
• Ma, Z. et al. (2025). ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models. arXiv:2506.21448.
• Zhang, M. et al. (2023). Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition. PDF.
• Mitra, S. et al. (2024). ThinkVideo: High-Quality Video Reasoning with Chain of Thoughts. arXiv:2505.18561.
• Wu, Y. et al. (2024). MINT: Multi-modal Chain of Thought in Unified Generative Models. arXiv:2503.01298.