Cohen's kappa score

Background

Cohen's kappa ($\kappa$) measures agreement between two raters — or a prediction against ground truth — corrected for the agreement expected by chance. Two raters labelling at random still agree sometimes, so raw accuracy overstates real agreement; $\kappa$ subtracts off that chance baseline. $\kappa = 1$ is perfect agreement, $0$ is chance-level, and negative means worse than chance. It is the honest metric for imbalanced or multi-rater labelling.

Problem statement

Implement cohens_kappa(y1, y2) for two label sequences:

$$\kappa = \frac{p_o - p_e}{1 - p_e}$$

where $p_o$ is the observed agreement (fraction of matching labels) and $p_e$ is the chance agreement $p_e = \sum_c p_1(c)\,p_2(c)$, with $p_r(c)$ the fraction of rater $r$'s labels equal to class $c$.

Input

• y1 — array-like of labels from rater 1.
• y2 — array-like of labels from rater 2, the same length.

Output

Returns a float (typically in $[-1, 1]$; $1$ = perfect agreement).

Examples

Example 1

Input:  y1 = [1, 0, 1, 1, 0], y2 = [1, 0, 0, 1, 0]
Output: 0.6154

Explanation: observed agreement $p_o = 4/5 = 0.8$. Chance agreement $p_e = (0.6)(0.4) + (0.4)(0.6) = 0.48$, so $\kappa = (0.8 - 0.48)/(1 - 0.48) = 0.6154$.

Constraints

• $p_o$ is the fraction of positions where the two labels match.
• $p_e = \sum_c p_1(c)\,p_2(c)$ over all classes that appear in either sequence.
• If $p_e = 1$ (both raters constant on the same class), return $1.0$ to guard the $0/0$.

Notes

• $\kappa$ can be negative when raters agree less often than chance would predict.
• The chance correction is what makes $\kappa$ informative on imbalanced data, where plain accuracy looks high simply because one class dominates.