Second-step state values were updated as:2V(s)←(1−αQ)V(s)+αQrIn models that included value forgetting this was implemented as:3Qmf(c′)←(1−fQ)Qmf(c′)4V(s′)←(1−fQ)V(s′)Action-state transition probabilities used by the model-based system were updated as:5P(s|c)←(1−αT)P(s|c)+αT6P(s′|c)←(1−αT)P(s′|c)In models that included transition probability forgetting this was implemented as:7P(s|c′)←(1−fT)P(s|c′)+0.5fT8P(s′|c′)←(1−fT)P(s′|c′)+0.5fTAt the start of each trial, model-based first step action values were calculated as:9Qmb(c)=∑sP(s|c)V(s)Models that included model-free values for first step motor actions (e.g., left→top), updated these as:10Qmo(c,st−1)←(1−αQ)Qmo(c,st−1)+αQ(λr+(1−λ)V(s))Motor level model-free value forgetting was implemented as:11Qmo(m′)←(1−fQ)Qmo(m′)Where m′are all motor actions not taken.
