METEOR score

Background

METEOR scores machine translation by aligning unigrams between candidate and reference, then combining a recall-weighted F-mean with a fragmentation penalty that punishes scrambled word order. Unlike BLEU it rewards recall and ordering explicitly, correlating better with human judgement at the sentence level.

Problem statement

Implement meteor_score(reference, candidate, alpha=0.9, beta=3, gamma=0.5). Lowercase and tokenise both, count matched unigrams $m$ (clipped), then:

$$P = \frac{m}{|\text{cand}|}, \quad R = \frac{m}{|\text{ref}|}, \quad F_{\text{mean}} = \frac{P\,R}{\alpha P + (1-\alpha)R}$$ $$\text{Pen} = \gamma\Big(\frac{\text{chunks}}{m}\Big)^{\beta}, \qquad \text{METEOR} = F_{\text{mean}}\,(1 - \text{Pen})$$

where chunks is the number of contiguous runs of matched words in the candidate. Round to 3 decimals.

Input

• reference — str.
• candidate — str.
• alpha, beta, gamma — float: METEOR parameters (defaults 0.9, 3, 0.5).

Output

Returns a float in $[0, 1]$ (rounded to 3 decimals).

Examples

Example 1

Input:  meteor_score("Rain falls gently from the sky", "Gentle rain drops from the sky")
Output: 0.625

Explanation: 4 unigrams match (rain, from, the, sky); $P=R=4/6$ gives $F_{\text{mean}}=0.667$. The matches form 2 chunks in the candidate, so $\text{Pen}=0.5(2/4)^3=0.0625$ and METEOR $=0.667\times0.9375=0.625$.

Constraints

• Lowercase + whitespace tokenise; match unigrams with clipping (each reference word usable once).
• $F_{\text{mean}} = PR/(\alpha P + (1-\alpha)R)$; chunks = contiguous runs of matched candidate positions.
• Penalty $= \gamma(\text{chunks}/m)^\beta$; final score $= F_{\text{mean}}(1-\text{Pen})$, rounded to 3 dp.
• Return 0 if there are no matches or either text is empty.

Notes

• The chunk penalty is METEOR's signature: the same matched words score lower when scattered (many chunks) than when contiguous (few chunks).
• The default $\alpha=0.9$ weights recall heavily — in translation, covering the reference matters more than terseness.