The Elect cannot be literally all men, in the sense of all-who-ever-lived-and-ever-will-live. Else the word Elect would make no sense. The Elect are the Church, consisting of believing jew and gentile. Chosen by God in Christ before the foundation of the world.
Another way to put it is that if you are in the Body of Christ (the Church), you are in the Elect. The Church was chosen by God in Christ before the foundation of the world. Individuals enter into the Church by faith in Christ. And that is a free offer to all mankind!
-JGIG