You've done all the right things, talking to your pastor, trying to make this work. He sounds immature and selfish. If he's not going to put the effort into this marriage, it won't work. You can die trying but it still won't work. You don't deserve or need to live like that. He abandoned your marriage. I think it's time for you to move forward and find happiness in your life. God bless you.