so what and to whom is the falling away to you?
Well certain letters speaking about a turning away from the truth or the falling away from the believe.
For the time will come when they will not endure sound doctrine; but after their own lusts shall they heap to themselves teachers, having itching ears;
"And they shall turn away their ears from the truth, and shall be turned unto fables."
2.Tim. 4,3-4
Matthew 24,11
"And many false prophets shall rise, and shall deceive many."
Matthew 24,24
"For there shall arise false Christs, and false prophets, and shall show great signs and wonders; insomuch that, if it were possible, they shall deceive the very elect."
All this verses are adressed to believers in Christ! So for me its clear that Christians are meant.