From that time many of His disciples went back and walked with Him no more.
These disciples who came to Him, being drawn by the Father, did turn away from Him and followed Him no more.
Being drawn to His does not = eternal salvation.
Eternal salvation is for those who obey Him.
And having been perfected, He became the author of eternal salvation to all who obey Him. Hebrews 5:9
No one can "obey" until they become saved/born-again - to do so is a fruit of the Spirit. Until they do become born-again, the eyes of their understanding remains darkened.
[Eph 1:17-18 KJV]
17 That the God of our Lord Jesus Christ, the Father of glory,
may give unto you the spirit of wisdom and revelation in the knowledge of him:
18 The
eyes of your understanding being enlightened; that
ye may know what is the hope of his calling, and what the riches of the glory of his inheritance in the saints,
They may have been with Him but they hadn't been drawn to Him - that's why they left Him. Walking with Him is not the same as having been drawn to Him. Spiritually speaking, they were unable to hear or to learn. The "therefore" in verse 65 makes this
clear because by verse 65, Jesus is excluding those who left from those who were truly drawn.
[Jhn 6:64-66 KJV]
64 But there are
some of you
that believe not. For Jesus knew from the beginning
who they were that believed not, and who should betray him.
65 And he said,
Therefore said I unto you, that no man can come unto me, except it were given unto him of my Father.
66 From that [time] many of his disciples went back, and
walked no more with him.
[Jhn 6:45 KJV] 45 It is written in the prophets, And they shall be all taught of God.
Every man therefore that hath heard, and hath learned of the Father, cometh unto me.