You are skipping something.Jesus says the Spirit Himself teaches believers directly:
“He will guide you into all truth” (John 16:13),
“He will teach you all things” (John 14:26), and John says
“you have no need that any man teach you” in the sense of false teachers because “His anointing teaches you” (1 John 2:27).
That does not eliminate pastors — it eliminates dependence on men as the final authority.
John was written to his disciples - the future Apostles.
They had to have a unique walk with the Spirit because the NT would not be in writing for years to come.