In The Last of Us Remastered's commentary with actress Ashley Johnson and director Neil Druckmann, in the scene where Joel found Ellie in the abandoned house, Ashley stated that she improvised by shoving him out of frustration, because she felt emotional reading for the scene where Joel is planning to leave her to Tommy.

Druckmann stated that the team wanted him to remove this scene where Joel is being too cold to Ellie, but he refused because he felt it was important to keep in the scene that Ellie is "being so vulnerable and [Joel is] having these feelings. He's trying to shut it down". Druckmann also commented that when Joel said to Ellie "You're not my daughter", he stated that while that quote is almost an insult, it's the opposite of what he actually feels.
Contributed by ProtoSnake
Video