Much of the "savage" world prior to colonialization did not wear tops. The women in South America, Africa, and the Polynesian islands were topless. Breasts were not considered sexual in these places during these times. The men also wore thongs. There are still a few places like this. However, Western men and women found that going topless was too sexual. While I think that going topless is extreme, I think the burden of guilt falls on the person feeling lust not the person showing extra skin (unless the person is being intentional/provocative).