Any thoughts ??

So I have two children who are from the same father .. It took us 1 month of trying when I got pregnant with our first child (I lost my virginity with him ) .. Then when our daughter was about 6 months old we had an accident while having sex (we was using condoms so we won't get pregnant) and well the condom broke and I became pregnant again ... Unfortunately he abandoned us when my youngest was only 1 year old .. Now my daughter is 7 and my son is 6 .. I'm in a new relationship and we've been trying to conceive for the past 6 months and nothing 😢😢😢😢 my boyfriend does not have kids .. I'm just so confused and sad .. I became pregnant pretty quick with my kid's father and now its taking forever !! Any thoughts ?