While I'm not sure that men are necessarily more visual than women, I don't believe that visual attraction is sinful (so I would disagree with what the person said to you in PM, Rachel). The reason is that there is so much beauty in God's creation, and God gave us eyes to see that beauty and brains to process it so that we might give Him glory for that beauty and splendor we see in it. I believe the same is true of the human body, and more specifically I believe that God created woman to be beautiful in the sight of man. There is nothing sinful about being beautiful or attractive, nor is there anything sinful about acknowledging beauty or feeling attraction. Song of Solomon is full of visual imagery about the beauty of the bride. That's why I don't believe that there is sin in being a visual person in terms of how you initially sense that someone is attractive.
Where it gets into the realm of sin is when you allow your eyes to stare and do double-takes in a way that is inciting lust in your heart and mind. Also, it is indeed sinful to only care and focus on the physical in a person with no care whatsoever for their soul. But that doesn't make visual attraction a result of the fall.
As far as modesty and lust goes, I think this is something that each gender needs to police for themselves. Men need to be worried and concerned about keeping their eyes, mind, and heart in check and hold each other accountable to that. Women need to consider what they are wearing and what their heart is behind wearing it.
Where I think things go wrong with men telling women or vice versa is that it comes across accusingly. Women don't want to hear from men that they need to dress more modestly. And if men are focused on honoring the Lord and the women around them and the women are focused on honoring the Lord and the men around them, we have a double safeguard against lust. And when someone fails to hold up their end of the bargain, that's on them.
Does it make it easier for me as a man to not see women with barely anything on or with clothes that emphasize certain body parts? Yeah. Do I have a lot of respect for women who choose to dress modestly? Totally. But that doesn't give me an excuse to lust after anyone, and I would never personally ask a woman to dress more modestly lest I be accused of making an excuse for myself or look as if I'm attempting to justify sin. What I will do is continue to ask the Lord to continue to redeem me from my sinfulness, that I would guard my eyes, mind, and heart so as to honor Him and the women around me.
I thought this article explained some things well and it's a message I think all men should heed:
Young Men, Sex, and Urge Ownership (And Why It's Not the Girl's Problem)