CLS Token (Classification Token)

What is CLS Token (Classification Token)?

A special learnable vector added to the beginning of the input sequence. The final state of this token is used as the aggregate representation of the entire image/text for classification tasks.

Where did the term "CLS Token (Classification Token)" come from?

Introduced in BERT, adopted by ViT.

How is "CLS Token (Classification Token)" used today?

Common pattern in encoder-only models.

Related Terms

• Vision Transformer (ViT)
• Foundation Model