Key Concepts, Self-Attention, Multi-Head Attention
Attention is a mechanism that permits neural networks to deal with totally different components of the enter sequence when processing data. It is a vital part of the transformer structure, enabling the mannequin to seize the sequence’s dependencies and relationships between totally different parts. For textual content sequences, the weather are token embeddings.
In a transformer mannequin, consideration is computed by way of the self-attention mechanism.
Disclaimer: I’m purposely not touching masks as that is solely related for the decoder a part of the Transformer, which goes to be tackled in Part 2. My principal aim right here is to clarify the fundamentals of the eye mechanism.
But let’s begin with first issues first…
In the context of consideration mechanisms, every factor within the enter sequence is related to a question, key, and worth vector.
Imagine you’re attending a convention the place a number of audio system give shows. Each presentation corresponds to a token within the enter sequence. Now, let’s break down the important thing, question, and worth on this context:
- Key: The key represents the content material or context of every presentation. It captures the primary concepts, themes, or related data related to every speak. Think of the important thing as a abstract or illustration of the important thing factors of every presentation.
- Query: The question represents the precise matter or query you’re interested by or wish to deal with throughout the convention. It might be a selected space of curiosity or a selected topic you’re inquisitive about. The question displays your present context or the side you wish to discover additional.
- Value: The worth accommodates the detailed data, insights, or data offered by every speaker throughout their presentation. It encompasses all the precious content material of every speak, together with info, examples, explanations, and concepts.
Hold on to this instance as we discover the eye mechanism.
