TD(0) value update

Background

TD(0) is the simplest temporal-difference learning rule for policy evaluation: it estimates the value function $V(s)$ of a fixed policy from experience, without a model and without waiting for an episode to end. After each transition it nudges $V(s)$ toward a bootstrapped target — the observed reward plus the discounted estimate of the next state's value.

Problem statement

Implement td0_update(V, s, r, s_next, done, alpha, gamma) performing one TD(0) update and returning the new value table:

$$\delta = r + \gamma\,(1-\text{done})\,V(s') - V(s), \qquad V(s) \leftarrow V(s) + \alpha\,\delta$$

Only V[s] changes; return the (copied) updated array.

Input

• V — np.ndarray (n_states,), current value estimates.
• s, s_next — int, current and next state indices.
• r — float, observed reward.
• done — bool, whether s_next is terminal.
• alpha, gamma — float, learning rate and discount.

Output

An np.ndarray (n_states,): a copy of V with V[s] updated.

Examples

Example 1

Input:  V = [0, 0, 0], s = 0, r = 1.0, s_next = 1, done = False, alpha = 0.5, gamma = 0.9
Output: [0.5, 0, 0]

Explanation: TD error $\delta = 1 + 0.9\cdot V[1] - V[0] = 1$. The update is $V[0] \mathrel{+}= 0.5\cdot 1 = 0.5$.

Constraints

• The target drops the $\gamma V(s')$ term when done is True.
• Only V[s] is modified; the rest of the array is unchanged.
• Return a copy (do not mutate the caller's array in place).

Notes

• TD(0) bootstraps: it learns a guess from a guess ($V(s')$), so it can update online, mid-episode, unlike Monte-Carlo which waits for the full return.
• The TD error $\delta$ is exactly the signal used by actor-critic methods and is the building block of TD($\lambda$).