Undecided

I recently just got married in march. I've been with my husband for 3 years before we got married and we have had some issues prior us being together. I'm still in love with my ex, which is my fist born dad. Then I got pregnant again with my husband. I ended up having our son early at 30 weeks and we have been through a lot. So long story short. I kinda pushed us into getting married and now that we are married. It's just bad. We fight all the time about the same thing, plus I'm still in love with my ex.
Right now I'm really ready to just get a divorce because I'm not happy and I rather just be single instead of me being married to him and loving another man. I feel as though it is not fair to him or our marriage. I constantly tell him that and it seems like he understands but idk what to do. 
Do I need to stay and fight for my marriage and try to get over my ex to make sure our relationship lasts. Or should I get a divorce and be single so nobody's feelings will ever get hurt in the long run.