Looking for advice

Me and my boyfriend been together a little over a year we have a baby together she is 2month old. I have 2 other kids with a different man. And he has 1 other kid with a different lady.

Well before we found out I was pregnant we did not live together and our relationship was great! But then we found out I was pregnant and we moved In together all happened within 5 months. Ever since then our relationship has been on the rocks constantly fighting about his other daughter and finances.

I am starting to think that maybe we need to live separately again but stay a couple. I think we moved to quickly and need to take huge step back And see if our relationship gets better or not. Any thoughts?