i think it isn't a selfish as long as you let him to meet with your son & keep their relationship going well, then maybe you need to talk to him that you need more times to make sure that you are ok with that situation. so you need to pray for it so you can find the answer for it.