It's a hassle to have hair in all the 'unwanted' places. I know women find it manly, but body hair is very unhygienic and is a breeding ground for bacteria that create body odour, especially in humid climate regions. (Too much information, eh?)
Hair can sometimes also be a nuisance. Like, when it grows on your ears. Believe it or not, you can see such men in India. :\