Deut 24:1-4 pretty much forbid exactly that scenario for the nation of Israel. So the question is, is this sexual immorality that is forbidden under the new covenant as well or do we treat it more like the laws that say the priests couldn't marry a divorced woman?
The bigger question in my mind would be: this person left you once for a better option, are you really going to give them an opportunity to do the same thing to you again?