The affection of a cold man will never seem particularly affectionate to the rest of us. I've always seen him as being true to form, also, in the main. There has only ever been one thing in his life that he has broken form for and that's Isolde. Marrying an Orlesian when he did was not true to his cold political form; putting the King's bastard into a Templar uniform wasn't either. Both of those things he did for her.