to the behavioral data, and ran the logistic regression analysis on data simulated from both models (Figures 2D–2I). The RL agents used in these simulations included forgetting about actions not taken and states not visited, as RL model comparison indicated this greatly improved fits to mouse behavior (see below). Data simulated from a model-free agent showed a large loading on the outcome predictor (i.e., rewards were reinforcing) but little loading on the transition predictor or transition-outcome interaction predictors (Figure 2E). In contrast, data simulated from the model-based agent showed a large loading on both outcome and transition predictors (i.e., both rewards and common transitions were reinforcing) (Figure 2H) and a smaller loading on the interaction predictor. Therefore, in our data the transition predictor loaded closer to the model-based strategy, and the interaction predictor loaded closer to the model-free strategy.Table 2RL and Logistic Regression Model Variables and ParametersVariables and ParametersDescriptionLogistic Regression Model PredictorsBias: top/bottomchoose top-pokeBias: clockwise/counterclockwisechoose top if previous trial ended at left poke, bottom if at rightChoicerepeat choiceCorrectrepeat correct choiceOutcomerepeat rewarded choiceTransitionrepeat choice followed by common transitionTransition-outcome interactionrepeat choice followed by rewarded common and non-rewarded rare transitionsRL Model Variablesrreward (0 or 1)cchoice taken at first step (top or bottom poke)c′choice
