but somewhere, at some time, the rose-colored pearle visions fell off and they wouldn't fit again

That's exactly it, right there in a nutshell. I'm sure if I hadn't been witness to some of his more, how shall I put this, not exactly sane moments, then perhaps I wouldn't hate the fact that he's in my shiny new fandom. But alas, it is what it is.

But dude, Barrowman is still hot. Watch the clips.