If Peter wasn't the pastor, then why did Paul waste his time arguing with him, and then going for agreement?
Here again, the word "pastor" is being applied with the modern understanding of a titled role, not a first-century understanding which is likely more of a ministry specialty.
I mentioned earlier (not sure if this thread or another) that I suspect the modern concept of "counselor" is closest to the biblical gifting of "pastor". I may be incorrect in this, but counseling is certainly one area where modern pastors are called to minister - and not all "pastors" are gifted in counseling. Personal experience testifies only too loudly of this.