Not so fast!
Genesis 11 discusses this very thing.
The so-called "Tower of Babel" was something so bad that God
Himself put a stop to it and dispersed the crowd pretty thoroughly.
Whatever it was could have been a threat to God Himself.
If "time travel" were possible, someone could theoretically
alter history to foil the death, burial, and resurrection of Christ.
Nothing can be a threat to God. What was evil about the Tower of Babel was that God commanded that people spread out and populate the world at that time. And they wanted to stay and make a name for themselves. They wanted to be famous- which is self-centered and prideful. They wanted to show that God is no big deal because even they can literally go up to His level.
God knew, as we now know, that the tower could never be high enough to reach God, and God is omnipresent anyways. It was the intentions in their hearts that He was displeased with.
(And no, I don't believe there are portals. Jesus appeared in a locked room, Philip suddenly vanished and appeared somewhere else after baptizing the Ethiopian eunuch. I believe miracles happen in a natural way- a supernatural way. Whereas magic is all a hoax- a trick.)