My advice is to keep praying and stay close to God. And pray for your husband, whom the enemy has blinded and is trying to destroy.
Have you talked to your pastor? I certainly do not agree that you should just divorce because he is the "wrong" man for you. God honors your commitment to each other. He does not honour a pastor who contradicts the Bible! IMHO