Jesus spent his time with the worst of people, yet was never like them. He was with prostitutes, but was never a prostitute. He was with drunkards, but never a drunkard. He was with tax collectors, but never a tax collector. In fact he spent so much time with these people, he actually had a reputation for it and was thought to be all these things himself. Hence the fuss the Pharisee's etc would hold about it. It's not wrong being with these people, often at school or whatever we don't have the choice. But we can always make the most of situations. Challenge people. Ask them why they spend all their time and money on things that cannot satisfy them? And exalt Christ. Be an example in speech, life, love faith and in purity. Reflect Christ and surely they will be drawn to you, for you are that fragile clay jar, with a treasure that they do not have themselves. And the best part is, that treasure you contain is free.
You're not wrong in feeling that way. I think we all feel it sometimes. But you have to remember if we are in need of a break, it's a break from the world we need and not God. So yes it's always a benefit to spend a weekend or so out of town with some fellow Christians. But I think a good majority if that time should also be spent praying over each other and fellowshipping with each other for that refreshing and somewhat preperation for going back into the world, and to be a light in the uttermost darkness that Jesus was, and also calls you to be.
May God's grace be with you
.