As Somedisciple asked. Why not? I think it can only be referring to a physical building because of the connection to Mark 13, seeing the abomination of desolation standing where it ought not to be.
{ What connection? }
Even if we were to concede that you are correct and the AoD already occured, it would still lead one to question how could 2 Thess 2:4 be fulfilled by the church? The church is comprised of true believers only, not fake ones, the fake christians arent the temple of God.
{ Neither is any physical building.
} The living stones are built up to the spiritual temple.
{ Exactly. } That is why it can only be fulfilled by a physical temple
{ That is why it cannot be fulfilled by a physical temple. }