The genealogies paired with the history recorded, places people were at certain times, the lifespans recorded, etc, can all be used to make a legitimate estimate of when man came into being regardless of their intended purpose for being there. The consensus among historians puts that at 6,000-10,000 years ago. You dismiss the idea of days actually meaning days in Genesis 1. This concept is fairly new and uses an interpretation of the bible based off of the flawed scientific cosmological model and origin of life hypothesis (Darwinian evolution falls dreadfully short of evidence to call it a scientific theory).
Genesis 1 can't be referring to an allegorical amount of time because it's made clear that a full day cycle took place with the words "and evening and morning were the # day". It denotes a full day cycle, not epochs of time. Using 2 Peter 3:8 or Psalm 90:4 to interpret the creation story doesn't make sense within their contexts. 2 Peter 3 doesn't refer to creation in any sense, and while Psalm 90 vaguely references creation, it still doesn't apply "a thousand years" to the time span it took place in. The psalmists also makes clear what we understand as actual day and night cycles in the same chapter.
If the creation account should be understood as allegorical, then scripture loses its foundation and why not take it to it's fullest extent and say all the characters mentioned were just allegorical stories to teach us lessons? The flood didn't really happen, how could it have? If God meant for us to interpret "a day" in the account as an epoch of time, I think he would have been more clear. He apparently meant for us to understand it as a literal day since there is the mention of evening and morning each day denoting a full day cycle.
It's sad to see so many Christians adhering to evolution and scientific cosmology, neither of which have ever been substantiated. And as was mentioned already, the order of events during the creation causes any other interpretation than that of a literal day to fall short.