advice please 😭

Hi I'm only 10 weeks and 3 days pregnant but suffer a lot with anxiety I tend to worry about things that might not happen a lot but anyways I'm pregnant with my second baby to my boyfriend who I have been with for over a year, I also have a 19 month old to an ex partner. My ex partner gave me a very hard time was verbally mentally and physically abusive in our relationship and I was too scared to leave until my daughter was 5 months old because if he had hurt her I never would have forgave myself. I went in to premature labour with her at 31 weeks and the early labour was put down to stress (from him) she was in hospital for 6 1/2 weeks and his behaviour was shocking he was very possessive waned joint custody of our daughter even though we were in a relationship and his family were the same to cut a long story shor he now only sees her once a week in a contact centre as I am pregnant again to my boyfriend who is more like a daddy to my daughter than my ex ever has been I'm just concerned of things like this happening again. I need advice on how to talk to people without things without being rude for instance my boyfriend has a very big family where as I only have 5 others, I hate thinking about when I have this baby as I don't want everyone visiting at once and passing a newborn baby round as I don't think that's fair but how do I explain this