The only problem I saw here is this..."I kept silent".
Why keep silent if you can say something?
How can a problem be resolved if everyone will just keep silent every time they feel something not right in a given situation or when someone says something offensive to them?
This is a very common problem I saw in the Church really. When a brother or sister said something offensive to them, then it's either they would stop going to church, or they would start saying lots of negative things behind that person's back, and so the problem gets even more complicated. Instead of dealing the problem right away they chose of making it worse.
Matthew 18:15 says, if thy brother shall trespass against thee, go and tell him his fault between thee and him alone: if he shall hear thee, thou hast gained thy brother.
It's just the mindset they have that if you are single then you have less obligations, less responsibilities and you have more time compared to married ones. If I was that single woman I'd just politely tell them my situation and I can't, I don't think their minds are that narrow not to understand my situation.
If nobody would tell those people what is wrong or what you feel about it, then they will keep doing what they're doing and you will keep feeling what you're feeling right now.