Vorenus is that he hates himself so much right that he can't understand why anyone else would love him, and the kinder and gentler Pullo said "Brother," the more it was salt in his wounds. I was surprised (and pleased) he didn't kill Pullo when he finally had the chance after all.