Point one: Phrasing the question this way kind of opens the door to condemnation. You have these desires regardless of how ok it is or isn't. I would think a more productive question to ask is I have these desires, how do I handle them in a way that honors God, myself, and other people? I can't see much good coming from people telling you that it's not ok to have desires that you don't know how to get rid of.
Point two: For a more accurate definition of sex. Let's assume your son is still a minor, what behaviors would you consider sexually inappropriate for an adult (of either gender) to initiate with your son? If that is your definition of sexual behavior in one case, why should the definition of what behaviors are sexual change just because it's you and another consenting adult? Sex is so much more than just the act of male / female intercourse.