Hold strong. U are just trying to keep things going.
Suggest that you don't try to please others. Focus on you. If they don't like you, either side, it's their issue.
U are accountable to God, not these folks.
As for your husband, state what you expect, issue a deadline for change.
Plan...