There is nothing wrong with how you feelings,but sadly the real father has rights to see his son. I understand that you love your son(you are
his father in every way,other than biologically) and that you want the best for him. Just make sure that he knows you love him,that you are there
for him,and stay with him through all things,no matter what.
I don't know whether the real father loves him or not,or if he is a good man or not. But just try to explain to him how you feel,and see how it goes.
Also pray,and hard,stay faithful,and trust God always-He knows best.
God Bless!