It takes a lot to learn to do it, but you can just forgive them and ignore it. After all, if you are following what the Lord has told you to do, all they are doing is proving what they think of His work. Of course, if you think there is any point to what they say, you should pray about if there is any truth to it.