My heart aches to read the pain behind your post. But, as you know, being a mom is all about putting our babies above our own needs. You are doing the right thing by trying to get along with him and allowing him to spend as much time as he can with your children. They love you BOTH and need you BOTH.
I do think it's important for you to limit YOUR contact with him while God is healing your heart, especially when he's around you and your children. Children are very sensitive to the moods of their parents and can be very defensive of their moms when they think someone is hurting her.
My prayer is that God will intervene in this situation and bring restoration that is best for everyone involved, especially your precious babies.