Multimodal large language models

From Systems Analysis Wiki
Jump to navigation Jump to search

Multimodal Large Language Models (MLLMs) are a class of artificial intelligence models capable of processing and generating information across various modalities, including text, images, audio, and video[1]. Unlike unimodal language models, which work exclusively with text, MLLMs integrate information from different sources to solve complex content understanding and generation tasks.

The core concept behind MLLMs is the creation of a unified vector representation (embedding) for different modalities. This allows the model to establish semantic connections between, for example, an image and its textual description[2]. A key breakthrough that laid the foundation for modern MLLMs was the use of contrastive learning to align visual and text representations in a shared feature space, as implemented in the CLIP model[3].

History of Development

Early Period (2013–2020)

The conceptual foundations of multimodal AI were laid in 2013, when researchers from Stanford demonstrated the feasibility of zero-shot learning using word vector representations[4]. In 2016, the FAIR (Meta AI) team showed the effectiveness of using natural language descriptions to train computer vision models, achieving 11.5% accuracy on ImageNet without direct training[5].

The CLIP Era (2021)

A revolutionary moment was the release of the CLIP (Contrastive Language-Image Pre-training) model by OpenAI in January 2021. The model, trained on 400 million image-text pairs, demonstrated the ability to classify images without specialized training on specific tasks. CLIP became the foundation for many subsequent MLLMs[6].

Scaling and Innovation (2022–2024)

Following the success of CLIP, numerous key models emerged:

  • Flamingo (DeepMind, 2022) — an 80-billion-parameter model that showed outstanding few-shot learning capabilities.
  • BLIP (Salesforce, 2022) — a unified architecture for understanding and generation.
  • GPT-4V (OpenAI, 2023) — the first commercial multimodal model of its scale.
  • LLaVA (Microsoft, 2023) — a popular open-source alternative to GPT-4V.
  • Gemini (Google, 2023) — a natively multimodal architecture, designed from the ground up to work with different data types.
  • GPT-4o (OpenAI, 2024) — a model capable of processing text, audio, and video in real time with low latency[1].
  • Claude 3.5 Sonnet (Anthropic, 2024) — a model with improved capabilities for visual information analysis.

Architectural Approaches

Dual-Encoder Architecture

This uses separate encoders for each modality, which project data into a shared representation space. A prime example is CLIP, where a vision transformer processes images and a text transformer processes language data. Its advantages include modularity and computational efficiency, while its disadvantage is limited cross-modal interaction[7].

Encoder-Decoder Architecture

A single encoder processes the multimodal input, and a decoder generates the text output. The Flamingo model uses a Perceiver Resampler mechanism to handle variable-length visual inputs and cross-attention layers. This approach enables rich inter-modal interaction but requires significant computational resources[8].

Alignment Architecture

This approach uses frozen, pre-trained encoders connected via a small, trainable alignment module. For example, BLIP-2 uses a Q-Former (Querying Transformer) as a lightweight bridge between a frozen vision encoder and a language model, requiring significantly fewer trainable parameters[9].

Key Models

GPT-4V / GPT-4o (OpenAI)

The GPT-4 family of models is estimated to have up to 1.8 trillion parameters (in a mixture-of-experts architecture). The GPT-4o model, released in May 2024, supports real-time processing of text, images, audio, and video. On the MMMU benchmark, it achieves 69.1% accuracy[10].

Gemini (Google)

A natively multimodal architecture, trained from scratch on text, images, audio, and video. Gemini 1.5 Pro supports a context window of up to 10 million tokens and outperforms GPT-4 on 30 out of 32 popular benchmarks[11].

Claude 3 (Anthropic)

A family of models (Haiku, Sonnet, Opus) with a context window of up to 200,000 tokens. Claude 3 Opus scores 58.5% on the MMMU benchmark. To enhance model safety, the Constitutional AI approach is used[12].

LLaVA (Open-Source Model)

Combines the CLIP vision encoder with the Vicuna language model. Variants with 7, 13, and 34 billion parameters are available. The model achieves 85.1% of GPT-4's relative performance on synthetic tasks[13].

Application Areas

  • Visual Question Answering (VQA): Allows users to ask questions about visual content.
  • Document Analysis: Modern MLLMs can process up to 2,000 pages per minute.
  • Medical Imaging: Models like Med-PaLM M (Google) analyze medical images and clinical data.
  • Robotics: Models like RT-2 (Google DeepMind) enable robots to understand visual environments and execute commands given in natural language.

Current Limitations

  • Hallucinations: The hallucination rate in generated content is estimated to be between 27–46%. Models may describe non-existent objects or misinterpret visual information[14].
  • High Computational Requirements: Training and using MLLMs require significant computational infrastructure.
  • Data Bias: Underrepresentation of demographic groups, languages, and cultures in training data leads to systematic errors.

See also

Literature

  • Radford, A. et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.00020.
  • Alayrac, J.-B. et al. (2022). Flamingo: a Visual Language Model for Few-Shot Learning. arXiv:2204.14198.
  • Li, J. et al. (2022). BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. arXiv:2201.12086.
  • Li, J. et al. (2023). BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. arXiv:2301.12597.
  • Liu, H. et al. (2023). Visual Instruction Tuning. arXiv:2304.08485.
  • Driess, K. et al. (2023). PaLM-E: An Embodied Multimodal Language Model. arXiv:2303.03378.
  • Brohan, A. et al. (2023). RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. arXiv:2307.15818.
  • Yue, X. et al. (2023). MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. arXiv:2311.16502.
  • Tsimpoukelli, M. et al. (2021). Multimodal Few-Shot Learning with Frozen Language Models. arXiv:2106.13884.
  • Singhal, K. et al. (2023). Med-PaLM 2: Towards Expert-Level Medical Question Answering with Large Language Models. arXiv:2305.09617.
  • Yin, S. et al. (2023). A Survey on Multimodal Large Language Models. arXiv:2306.13549.

References

  1. ↑ 1.0 1.1 “A Comprehensive Guide to Multimodal LLMs”. Encord Blog. [1]
  2. ↑ “A Survey on Multimodal Large Language Models”. ACM Computing Surveys. [2]
  3. ↑ Radford, A., et al. “Learning Transferable Visual Models From Natural Language Supervision”. arXiv:2103.00020. [3]
  4. ↑ DeOldify, J. “Zero-Shot Learning by Predicting Attributes”. arXiv:1312.5650. [4]
  5. ↑ “Learning from captions: A milestone in visual language understanding”. OpenAI Blog. [5]
  6. ↑ “Understanding CLIP”. Stanford CS231n. [6]
  7. ↑ “Multimodal LLMs: The Complete Guide”. Viso.ai. [7]
  8. ↑ “The Architectures of Multimodal Language Models”. Determined AI. [8]
  9. ↑ “Understanding BLIP-2: The New Vision-Language Model”. Clarifai Blog. [9]
  10. ↑ “MMMU: A New Benchmark for Multimodal LLMs”. Encord Blog. [10]
  11. ↑ “Google Gemini: A Deep Dive”. DaveAI Blog. [11]
  12. ↑ “Introducing the Claude 3 Family”. Anthropic. [12]
  13. ↑ Liu, H., et al. “Visual Instruction Tuning”. arXiv:2304.08485. [13]
  14. ↑ “Hallucinations in Multimodal Large Language Models”. arXiv:2308.08726. [14]
Retrieved from "https://systems-analysis.ru/eng/index.php?title=Multimodal_large_language_models&oldid=2757"