To the OP, let me relate two incidents of a marriage (from where I come from )that did not start off well but ends well due to the perseverance and faith of one of the spouse. I don't know if you can relate to it or not but going through your post, I felt for you and as a believing Christian I would not want you to opt for a divorce either. I believe everything is possible in and with God's help. So, just wanted to relate this incident to let you know that you are not alone and at the same time, wanted to remind you of the hope we have in Christ if we persevere as the two marriages in the testimony will reveal.
1) The first testimony is about a man who became a God fearing evangelist now. This man in his youth days was a nominal christian and an alcoholic. After getting married, he did not change his ways but became even physically abusive towards his wife after a couple years of marriage. His wife was helpless and did not know what to do and so, sought the counsel of other Christians. The wife after seeking counsel from other fellow Christians realised that only God could change/transform her husband (as divorce is not an option) and so, she started praying for him every day and night and she would also wake up regularly at midnight and pray for her husband with sorrowful heart and tears while her husband sleep soundly like a log. This continue for about a little more than a year and by then they already had two kids. Then one night, her sorrowful prayer and tears woke her husband up and he was convicted of all his ill-behavior and ill-treatment towards his wife. To quote him, "I woke up hearing her soft cry in the death of night and listen to her praying for me. I was like, why did she prayed and cryed for a loser like me and why didn't she left me despite me ill-treating and abusing her? Then, it pierced my heart to see the depth of her love and sacrifice she has put up for our family and I just didnt know (at the time) what made me to kneel besides her and pray along with her (a man who hardly remain sober and prayed before) but I was full of tears repenting that night itself and vow never to drink, gamble, smoke and physically abused her again." From then on, their marriage life began to change for the better and he became involved in church ministry wholeheartedly, became a full time evangelist later on. Its been about thirty years now since that incident and they are now one of the most admired Christian couple serving the Lord.
2) The second testimony is about a Reverend and his estranged wife. This reverend is a well known figure in my home state for his faithfullness in the Lord. After almost twenty years of marriage, his wife (who was otherwise perceived as faithful in the Lord) eloped with another man out of the blue, leaving behind not only her husband but also her children. The whole town was shocked and the incident became the talk of the Christian community in the state. Then, after a lapse of about 3-4 months, the wife finally repented and returned back to her husband who welcomed her back into their marital life (insiders had it that the reverend constantly prayed for his wife during all these time). Everyone was again shocked by the forgiveness attitude displayed by the reverend in welcoming back his wife. It seems the reverend told people close to him that if Christ could forgive his filthy sins, who was he to condemn his repentant wife? It seems the reverend didnot blame his wife rather he blame the devil that took control of her over a moments weakness. This incident took place in the mid 1990's and their marriage still remain strong to be seen.
I am relating this two story in the hope that it in some way might help in elevating your burden that you seems to be going through at the moment.