The "second man" is a metaphor. It has nothing to do with genealogy or chronology.
Adam (the first man) was a type of Christ and he brought a curse upon humanity and creation. In contrast, Christ as "the second man" did away with the curse and brought light and immortality to life by the Gospel.