Attention Mechanism
An attention mechanism is a neural network component that allows a model to dynamically weigh the importance of different input elements when producing an output.
The attention mechanism was introduced by Bahdanau, Cho, and Bengio in 2014 to address a limitation of encoder-decoder architectures for sequence-to-sequence tasks like machine translation. In a standard encoder-decoder model, the encoder compresses the entire input sequence into a fixed-length context vector, which becomes a bottleneck for long sequences. The attention mechanism solves this by allowing the decoder to access the entire sequence of encoder hidden states, computing a weighted sum of these states at each decoding step. The weights are learned and indicate which parts of the input are most relevant for generating the current output token.
This computation involves three key components: a query, a set of keys, and a set of values. The query is derived from the decoder’s current hidden state, and the keys are derived from the encoder’s hidden states. A compatibility function (often a dot product or a small neural network) computes a score between the query and each key. These scores are normalized using a softmax function to produce attention weights, which are then used to compute a weighted sum of the values (typically the same as the keys). This weighted sum, called the context vector, is fed into the decoder along with the previous output to generate the next token.
Attention mechanisms have since evolved into several variants, including self-attention (where queries, keys, and values all come from the same sequence) and multi-head attention (which runs multiple attention operations in parallel). The Transformer architecture, introduced in 2017, relies entirely on self-attention and has become the foundation for many state-of-the-art models in natural language processing and other domains. The mechanism is differentiable, allowing it to be trained end-to-end with backpropagation.
Why it matters
Attention mechanisms are fundamental to modern deep learning because they enable models to handle long-range dependencies and variable-length inputs effectively. They have been crucial for breakthroughs in machine translation, text summarization, image captioning, and speech recognition. By providing interpretable weight distributions, attention also offers insights into model decision-making, which is valuable for debugging and trust. The mechanism’s flexibility has led to architectures like Transformers, which power large language models and have transformed natural language processing and beyond.
First appeared
Bahdanau, Cho, Bengio, 2014.
Related terms
FAQ
How does it work?
The mechanism computes a weighted sum of input values, where weights are determined by the relevance of each input element to a query. A compatibility function scores the query against each key, and softmax normalizes these scores into attention weights. The weighted sum of values produces a context vector that highlights important information.
What is the difference between soft and hard attention?
Soft attention computes a differentiable weighted sum over all input elements, allowing gradient-based training. Hard attention selects a single element stochastically, which is non-differentiable and requires reinforcement learning or other techniques for training. Soft attention is more commonly used in practice due to its simplicity and efficiency.
When should I use self-attention versus cross-attention?
Use self-attention when modeling relationships within a single sequence, such as in language models or text classification. Use cross-attention when combining information from two different sequences, such as in machine translation where the decoder attends to the encoder’s output. Many architectures, like Transformers, use both in different layers.