it says they had left the temple. And as they were walking.
That's not what it says. It says Christ alone was exiting temple and disciples came to him. That's a meeting place right at temple. Christ gives a prophecy and then they will walk to mount as a group.
Mat 24:1
And Jesus went out, and departed from the temple:
and his disciples came to him for to shew him the buildings of the temple.
Mat 24:3 And as he sat upon the mount of Olives, the disciples came unto him privately, saying, Tell us, when shall these things be? and what shall be the sign of thy coming, and of the end of the world?
On the mount they ask, "Tell us, when shall these things be?" Read carefully because Christ does not answer this question. He never says when anything takes place. He only answered about the signs of his coming and end of the word plus he gives signs that come prior to start of the Great Tribulation. He never says when any of this takes place, presumably based on something related to Mat_24:36 .