Positional Encoding

What is Positional Encoding?

Positional encodings are mechanisms to inject information about the relative or absolute position of tokens in the sequence, as the self-attention mechanism is invariant to order.

Where did the term "Positional Encoding" come from?

Fundamental part of the original Transformer architecture.

How is "Positional Encoding" used today?

Evolved into variants like RoPE (Rotary Positional Embeddings) and ALiBi which allow for better length generalization.

Related Terms

• Attention Mechanism
• Transformer Architecture
• Large Language Model (LLM)