So? There being ONE man out there thinking that way means that ONE man might one day rape someone
(same if it was a girl)
If ONE man said "I want to kill you"
would you think that was OK because most didn't?
Nobody has said anything like that... it's NOT ok to physically assault someone, male or female.
But to look at the other side of your question... If ONE man said "I want to kill you"... is it ok to assume that ALL men want to kill you, because that one man did?
Just because some men are sexual pigs does not mean that all men are. And, as has been mentioned, our society tends to ASSUME guilt on the part of the man, if an accusation is made.
Remember the girl that accused that university soccer team of gang rape a few years back? Those young men were castigated in the press, found guilty in the eyes of much of our society, because people ASSUMED they had done it... because they were "young men"... and we ALL know what young men do....
Turns out the girl made the whole thing up... after weeks (months?) of sticking to her heart-rending story..