Actually, it was the ones who couldn't take Him literally who left. The ones who took Him at His word, that the bread really was His Flesh, and the wine His Blood, stayed.
Let's take a step back. In the past, when Our Lord spoke in parables, when there was confusion over His meaning, He took the time to clarify things (Matt. 16:5–12). Here, when He is talking about eternal life, He doesn't go off and say, "Guys, I was being figurative." Instead, He says, "Truly, truly, I say to you, unless you eat the flesh of the Son of man and drink his blood, you have no life in you; he who eats my flesh and drinks my blood has eternal life, and I will raise him up at the last day. For my flesh is food indeed, and my blood is drink indeed. He who eats my flesh and drinks my blood abides in me, and I in him" (John 6:53–56).
Quite a bit different than in the past, almost like He's emphasizing that this really is His Flesh and His Blood. And the reaction of the disciples implies bewilderment over this statement, "Many of his disciples, when they heard it, said, ‘This is a hard saying; who can listen to it?'"
At that statement He tells them that they won't comprehend the truth except by way of the Spirit, without that grace, their fleshly minds can't grasp the concept.
"It is the Spirit that gives life, the flesh is of no avail; the words that I have spoken to you are spirit and life" (John 6:63)
And then still there were some who simply couldn't accept His literal words and left, "After this, many of his disciples drew back and no longer went about with him" (John 6:66).
Remember, Our Lord said, "This is my blood." He didn't say, "This represents my blood." He knew what He meant and said exactly that.