Great paper from Salesforce AI Research on RL for multi-turn tool use.

The finding is that you want to train the one call where the action changes the outcome, instead of spreading reward across the whole trajectory.

(bookmark it)

When reward depends on later turns, much of its variation comes from what happens downstream.

Critical-State RL uses nested sampling to separate the reward variation caused by the current action from that noise, then trains only the selected call with contextual-bandit updates.

On BFCL v4 missing-function tasks, training the selected turn adds about 14 points. Training the other candidate turn leaves accuracy flat or lower.

Paper: https://academy.dair.ai/papers/critical-state-rl-diagnosing-trainable-states-for-multi-turn-tool-use-2609.24985