@NicoleCRust
So for #2 the Bellman equation solves a problem---estimate expected future reward without directly estimating the future---that is not faced by the brain. Insofar as the brain has an explicit temporal memory of the continuous past (and it certainly does), it is straightforward to construct a direct estimate of the distant future via simple Hebbian associations. We didn't know that in the 80's when Sutton and Barto were working on this problem or the mid-90's when the dopamine mapping was made. This paper makes these arguments in much more depth: https://arxiv.org/abs/2302.10163
(will try to write answer to #3 later, time for dinner)
NicoleCRust