I think its a fear of man issue. If a humble godly bloke who has taken enormous effort in todays society to stay pure for his wife (which is AWESOME!) has people around him accuse him of being gay etc he should stand his ground and be secure in Jesus as his identity and not feel shame for what hes worked to achieve.
Also, I find most people who harras you of anti cultural things you do are actually jealous of you. I deal with it all the time, its just another grain of salt in the beach of "religious weirdo" "old fashioned" and "Amish" insults I get. Blah blah blah...
I once dated a guy who came from a family that accused you of being homosexual if you graduated highschool. His family culture was filled with things like visiting a prostitute once you became legal, stealing something, smoking something illegal, only watching tv programs if it had decent violence in it and drinking bourbon every night. My ex on the other hand was a bit of black sheep, he was respected, he graduated highschool, went to college, became a black belt kung fu instructor and only drank Jim Beam on public holidays lol. Not to mention he dated me - what his family referred to as a "goody two shoes christian priss." The taunts I got were nothing compared to his. And you know what he did, he hardened up, ignored them and continued doing what he was doing, being an 'abnormal' yet respectable citizen of earth. And what made his family even more horrified was that he at least waited until he was in love with a woman before giving up his virginity at age 21 in comparison to giving it to a prostitute at age 16.