Or maybe, just maybe, he felt he'd never scorned you. Sometimes it's good to listen to what people say, not make things up to insert into their words to try to feel justified or prove a point. Honestly your response rings more of a guilty conscience than his.