Unfortunately, in America, if a man is over a certain age and still single, many people in society (including, and sometimes, especially in churches) view him as:
1. Homosexual.
2. Desperate for female company.
3. Possibly dangerous around children.
Now of course, women might have their own issues as well, but unfortunately, a single man, say around age 40 and over often has to fight off several unflattering stereotypes.
But at the same time, unfortunately, there are reasons why those cautions need to be in place.
I had a co-worker who was a single man in his mid-40's who would often stop and talk to customer's children, usually kneeling down so that he could talk to them at eye level, and sometimes bypassing any conversation with the parent. Some of the parents would complain about this to management.
I honestly don't think he meant any harm, but if I had children, I would probably have gotten them away from him immediately, because even if it verges on paranoia, I'd rather be safe than sorry, whether in a public place or in a church.
(I'm not saying at all that single men in church have wrong motivations, but yet, I'm not sure how to work around the fears that society reinforces either, because in some cases, it's made out to be common sense rather than paranoia.)