Either you love each other well or he's abusive. One or the other. And it seems pretty clear from your description that he's abusive.
Just leave already. As far as we know, you don't have kids or any shared assets with this guy, so I'm not sure what the hold up is...