In-Context Learning

In-Context Learning (ICL) is a fundamental capability of large language models (LLMs) to learn new tasks "on the fly" using only examples (demonstrations) provided within the context (prompt) of a query. A key feature is that this adaptation process occurs without updating the model's weights (parameters), meaning without traditional fine-tuning.

This mechanism allows models to exhibit remarkable flexibility, solving tasks for which they were not specifically trained. ICL has become one of the key breakthroughs that has made large language models so powerful and versatile.

How It Works

A precise understanding of how ICL works remains an active area of research; however, there are several leading theories that explain this phenomenon.

Transformer as a Meta-Optimizer

One popular theory suggests that the Transformer architecture learns to implement learning algorithms within its forward passes during pre-training. When the model receives a prompt with examples, it implicitly performs a form of optimization to solve the presented task by adjusting its internal states (activations) rather than its weights.

Bayesian Inference

Another theory views ICL as a form of Bayesian inference. A model pre-trained on vast amounts of data has a prior understanding of numerous concepts. The examples in the context serve as evidence that allows the model to update its posterior probability distribution over the latent concept. In other words, the examples help the model "understand" which specific task, out of the thousands it knows, needs to be solved at that moment.

Types of In-Context Learning

Depending on the number of examples provided, ICL is divided into three main types.

• Few-shot Learning: This is the most common and balanced approach. The model is provided with a few (typically 2 to 10) demonstration examples.

Example (sentiment classification):

Text: "What a beautiful day!"
Sentiment: Positive

Text: "I hate being stuck in traffic."
Sentiment: Negative

Text: "This movie was rather average."
Sentiment:

Expected output:

Neutral

• One-shot Learning: The model is given only one example. This is often sufficient to set the output format and significantly improve performance compared to the zero-shot approach.

• Zero-shot Learning: The model is not provided with any examples, only an instruction or a description of the task. In this case, the model relies entirely on the knowledge acquired during its pre-training.

Practical Applications

The proper application of ICL allows for solving a wide range of tasks without costly development and fine-tuning.

• For creative and stylistic tasks (e.g., generating code in a specific style, writing text in the manner of a particular author):
  • Few-shot Learning is recommended.
  • Examples help the model grasp the required style, format, and output structure.

• For simple tasks with clear instructions (e.g., translation, summarization, answering simple questions):
  • Zero-shot Learning is often sufficient.
  • Modern models handle such tasks quite well if they were part of their pre-training.

• For tasks where the output format is crucial (e.g., generating JSON, entity extraction):
  • One-shot or Few-shot Learning is recommended.
  • Even a single example can clearly define the required response structure, preventing formatting errors.

Advantages and Disadvantages

Comparison of Advantages and Disadvantages of ICL

Advantages	Disadvantages
Flexibility and Speed: Instant adaptation to new tasks without the need for retraining. Resource Efficiency: Does not require data collection, labeling, or the computational resources needed for fine-tuning. Accessibility: Allows users without ML expertise to configure models using simple text examples.	Context Window Limitations: The number of examples is limited by the model's maximum context length. Sensitivity to Examples: The result is highly dependent on the quality, order, and format of the provided demonstrations. High Inference Costs: Long prompts with many examples increase the cost and time of generation. Security Risks: Providing confidential information as examples can be insecure.

Comparison with Other Paradigms

ICL vs. Fine-tuning

Fine-tuning modifies the model's weights, "imprinting" new knowledge into it. This makes the model an expert in a narrow domain but reduces its overall flexibility. ICL, in contrast, does not change the weights and is more flexible, but it may underperform on highly specialized tasks that require deep domain knowledge.

ICL vs. RAG (Retrieval-Augmented Generation)

Both methods extend the model's context, but for different purposes:

• ICL uses examples to teach the model how to perform a task (demonstrating a skill).
• RAG uses retrieved information to provide the model with facts needed for the response (providing knowledge).

In practice, ICL and RAG are often combined to achieve the best results.

Further Reading

• Brown, T. B. et al. (2020). Language Models are Few-Shot Learners. arXiv:2005.14165.
• Dai, D. et al. (2022). Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers. arXiv:2212.10559.
• Panwar, M.; Ahuja, K.; Goyal, N. (2024). In-Context Learning through the Bayesian Prism. arXiv:2306.04891.
• Müller, S. et al. (2021). Transformers Can Do Bayesian Inference. arXiv:2112.10510.
• Garg, S. et al. (2022). What Can Transformers Learn In-Context? A Case Study of Simple Function Classes. arXiv:2208.01066.
• Min, S. et al. (2022). Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?. arXiv:2202.12837.
• Wang, X. et al. (2023). Explaining and Finding Good Demonstrations for In-Context Learning. arXiv:2302.13971.
• Xie, S. et al. (2024). A Survey on In-Context Learning. arXiv:2301.00234.
• Yu, Z.; Ananiadou, S. (2024). How Do Large Language Models Learn In-Context? Query and Key Matrices of In-Context Heads Are Two Towers for Metric Learning. arXiv:2402.02872.
• Wibisono, K. C.; Wang, Y. (2024). From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and When. arXiv:2406.00131.
• Chan, J. K. et al. (2022). Data Distributional Properties Drive Emergent In-Context Learning in Transformers. arXiv:2205.05055.
• Hahn, M.; Goyal, N. (2023). A Theory of Emergent In-Context Learning as Implicit Structure Induction. arXiv:2303.07971.