Entropy / Information Gain

What is Entropy / Information Gain?

Entropy measures disorder or uncertainty. Information Gain, used in Decision Trees, measures the reduction in entropy achieved by splitting data on a specific feature.

Where did the term "Entropy / Information Gain" come from?

Information Theory (Shannon).

How is "Entropy / Information Gain" used today?

The logic behind how trees decide to split.

Related Terms