Encoder-only models
Encoder-only models are a class of large language model (LLM) architectures based exclusively on the encoder part of the Transformer architecture. Unlike models that use a decoder or a full encoder-decoder architecture, these models specialize in Natural Language Understanding (NLU) tasks.
The flagship model and pioneer of this approach is BERT (Bidirectional Encoder Representations from Transformers), developed by Google in 2018.
Concept and Architecture
The core idea behind encoder-only models is to create deep, contextualized representations (embeddings) for each token in an input sequence. Thanks to the self-attention mechanism in the Transformer, each token can "see" and interact with all other tokens in the sequence, allowing the model to capture rich context.
A key feature is bidirectionality: the representation of each token is formed based on both the left and right context simultaneously. This fundamentally distinguishes them from autoregressive decoder-only models (like GPT), which are inherently unidirectional.
Architecturally, the model is a stack of identical encoder layers. Each layer consists of two main sub-layers:
- Multi-Head Self-Attention: Computes a contextualized representation for each token.
- Feed-Forward Network: Applies a non-linear transformation to each token's representation.
At the output, the model generates a sequence of vectors of the same length as the input sequence, where each vector is a rich representation of the corresponding input token.
Pre-training Tasks
To teach the model to understand language in a bidirectional context, special self-supervised pre-training tasks are used:
Masked Language Modeling (MLM)
This is the primary and most important task for encoder-only models, first introduced in BERT.
- How it works: A small percentage of tokens (usually 15%) in the input sequence is randomly hidden (masked). The model's task is to predict the original values of these masked tokens using their surrounding bidirectional context.
- Objective: This task forces the model to learn deep semantic and syntactic relationships between words.
Next Sentence Prediction (NSP)
This task (also from the original BERT) was designed to teach the model to understand relationships between sentences.
- How it works: The model is given a pair of sentences and must determine whether the second sentence is a logical continuation of the first in the original text.
- Status: Later research (e.g., in the RoBERTa model) showed that NSP is less effective than MLM, and it is often replaced by other tasks or removed entirely.
Applications
Encoder-only models are not designed for free-form text generation, as they lack an autoregressive decoder. Their strength lies in text analysis and understanding. The model's output vector representations are used to solve a wide range of NLU tasks:
- Text Classification: For tasks like sentiment analysis or topic detection, the representation of the special `[CLS]` token, which is added to the beginning of each sequence, is used. Its final vector aggregates information about the entire sequence.
- Token Classification: For tasks like Named Entity Recognition (NER) or Part-of-Speech (POS) tagging, the vector representations of each individual token are used.
- Question Answering: In tasks where the answer is a snippet from a given text (extractive QA), the model is trained to predict the start and end tokens of the answer.
- Embedding Extraction: Encoder models are often used as general-purpose text encoders to obtain high-quality embeddings for sentences or documents, which can then be used in search engines or for semantic similarity tasks.
Key Models and Their Evolution
- BERT (2018): The pioneer of the architecture, setting new records on numerous NLP benchmarks.
- RoBERTa (2019): "A Robustly Optimized BERT Pretraining Approach." Showed that BERT's performance could be significantly improved through longer training on more data and by abandoning the NSP task.
- ALBERT (2019): "A Lite BERT." A model with significantly fewer parameters due to techniques like embedding factorization and cross-layer parameter sharing.
- DistilBERT (2019): A smaller version of BERT created using knowledge distillation, which is faster and lighter while retaining most of the original's performance.
- ELECTRA (2020): Introduced a more efficient pre-training task, replaced token detection, where the model learns to distinguish original tokens from "fake" ones generated by a small generator model.
- DeBERTa (2020): Introduced a "disentangled attention" mechanism that separately encodes content and the relative positions of tokens.