Shannon entropy

Background

Shannon entropy measures the average uncertainty (in bits) of a discrete distribution. A fair coin has 1 bit of entropy; a certain outcome has 0. Entropy is the foundation of information theory and shows up everywhere in ML — it is the impurity criterion for decision-tree splits and the basis of cross-entropy loss.

Problem statement

Implement shannon_entropy(p) for a discrete probability distribution:

$$H(p) = -\sum_{i:\,p_i>0} p_i \log_2 p_i$$

Skip zero-probability terms (treat $0\log 0 = 0$). The result is in bits.

Input

• p — np.ndarray (K,), a probability distribution (non-negative, sums to 1).

Output

A float: the entropy in bits.

Examples

Example 1

Input:  p = [0.5, 0.5]
Output: 1.0

Explanation: a fair binary choice needs exactly one bit: $-(0.5\log_2 0.5 + 0.5\log_2 0.5) = 1$.

Constraints

• Use $\log_2$ so the answer is in bits.
• Ignore entries where $p_i = 0$ (their contribution is 0, and $\log 0$ is undefined).
• Entropy is maximized ($\log_2 K$) by the uniform distribution and minimized (0) by a point mass.

Notes

• Switching to natural log gives entropy in nats; the only difference is the log base.
• Entropy is the expected "surprise" $-\log_2 p_i$; rare events carry more bits of information.