I say yes, with an explanation.
People are imperfect and can be too easily swayed or sidetracked. Since we are not perfect, neither is our love. I think, ideally, the answer would be no, but in a fallen reality the answer is yes. Humankind does not, in it's current state, possess the capacity to be so limited.
Help me understand what you're saying......
I think you are saying that "with God it is possible, however, without God it is impossible for a man to only love his wife and not other women."
Is this what you are saying?