Losing my kids, please give advice not critcism

My ex and I divorced three years ago. I had four kids at the time. The oldest was a freshman in college. The others were 10, 12, and 16 and all lived with me. I met someone soon several months after my divorce and quickly became pregnant while I had an IUD. During my whole pregnancy and the first year of my babies birth I lived alone with all of my kids. About 6 months ago I finally moved in with my boyfriend and the father of my baby. All of my kids were supposed to live with me, with the exception of the son I have in college. I painted and fixed up all of their rooms in our new home. Well, the now 19 year old and 15 year old stopped staying here because my bf had certain rules such as not being out past a certain hour and turning off TVs and not making noise in the kitchen past 11. They live with their father now. My 13 year old stays here about Half of the time and at his dads the other half of the time. I know it’s mostly because he misses his siblings and wants to be with them too. Sometimes I feel guilty and feel like I should just leave my bf and have all of my children again. I feel like as if I traded a man and had his baby for my other kids. I need them and have terrible guilt. But at the same time I think one day they will look for their happiness and not care to leave me alone. I’m not sure what to do. I do feel like my bf can be strict. But I also know their father lets them do whatever they please and doesn’t set down any ruleS. I feel like I’m loosing my kids and I need them. What should I do?