He runs away not because he doesn't care, but because he cares so much that he can't deal with the pain.

Running is a response. Four's complete inability to reply to Leela at the end of Invasion of Time is a prime example. At the end of LotTL and GitF, I just didn't see a response.

As you say, we clearly got different reads on issues that may come down to actors' delivery, which is always going to remain subjective.