Yeah, that's how I feel with some womens clothing too and I'm struggling with it. I'm REALLY skinny for a male, not sure if I'm underweight to be honest. The lowest waist size they make for male super skinny jeans are too big for me, but womens jeans fit my body much better and is MUCH more comfortable. Also with underwear, panties/knickers are just wayyyy more comfortable than boxers. But I'm just afraid with wearing them cos I'm afraid of sinning. Like, I love them because of the comfort and because they fit me better than the male versions do, but I'm afraid to wear them in case I sin and that's why I don't.
Being faithful to God is just more important to me than wearing comfortable clothes or clothes that fit. I'm not sure if it is sin, but just in case. I wish it wasn't sin, but I don't know for certain if it is.