For men a big part of that is physical contact. "That is why a man leaves his father and mother and is united to his wife, and they become one flesh. " Genesis 2:24 .
It's not that big a part of it. You are making a mountain out of a mole hill.
I "need" to talk to my wife every day. I don't "need" sex with her every day.
But even if I can't talk with her every day we are still "one flesh".
And until you really give up your notion of sex being the "end all, be all" of marriage....you are not going to experience a happy marriage.
Thats all there is to it.
Your spouse is a person...full of hopes, dreams, feelings, thoughts, and feelings. And unless you have intellectual, emotional, and spiritual intimacy with her you cannot have a successful marriage.
You have two people cohabitating using each other as a masterbatory aide without the intimacy I have been explaining.