I once joined a life group when I was living in Michigan and it was all older men. I was in my 30's and the youngest in the group was probably in his 50's. It was little uncomfortable since I thought it would be nice to be in a group with people in my age group. It ended up as one of best life group as I learned and received plenty from the group. Then, joined another life group, now in California, and it was all women, youngest in her 50's. I really thought it was a mistake. Then, again, decided to stick around and got so much out of it. Realize, and remind yourself again why you are there at church. Sure, it's distracting and you feel alone, conscious. I get it. But at the same time, push those feeling aside just little bit and just focus on why you are there in the first place. You never know what you will get out of it, what you will find by sticking to THE reason you are there.