What you were wearing and what he was wearing are two different entirely thing.
Some dress in a suit to draw attention; just as some dress below modest means to draw attention.
There was nothing wrong with what you were wearing as it is modest apparel.
Look at how Paul instructs the women to dress.
1 Timothy 2:[SUP]8 [/SUP]I will therefore that men pray every where, lifting up holy hands, without wrath and doubting.[SUP]9[/SUP]In like manner also, that women adorn themselves in modest apparel, with shamefacedness and sobriety; not with broided hair, or gold, or pearls, or costly array;
If, in like manner that women adorn themselves in modest apparel, then that speaks to the men as well. Do men wear make up? No. Then neither should the women in shamefacedness.
But if everybody, and I do mean everybody, is wearing their Sunday's best, then if you have a suit, wear it then; just to blend in and not draw attention to yourself, but don't join the crowd in being standoffish just because a newcomer comes in not dressed in their Sunday's best.
What retailers ask for suits may be ridiculous to the poor, and so I have no qualms to speak against one wearing modest clothing that they wear everyday when they go out in public.
Hebrews 13:1Let brotherly love continue.[SUP]2 [/SUP]Be not forgetful to entertain strangers: for thereby some have entertained angels unawares.