Of cause clothing matter to God, Especially as Ministers, scruffy dressing could offend people, and the Bible says give no offence in anything.
The Mormons and JWs put some Christians to shame, as they always dress smart.
Those scruffy ministers have no respect for God, their ministry or the people they are ministering to.
It's very different if one is poor and can't afford to buy clothes.
When I first got saved, is was a long haired hippy who only had scruffy clothes, I just knew I had to have a hair cut, No one told me to, I just knew, and I felt bad for not being able to dress smart. And God knew this, So He led someone to buy clothes for me, What a transformation.
And that's what being a Christians does, It transform's people.
The love of God will transform those who love Him, they would want to go to Church well dressed.