Sending baby signals

So I have two kids already from a previous relationship. I have been dating this guy for months now but known him about 9 years. Well this makes the third guy now that can’t stop thinking about having a baby with me and I’m scared. Three kids is way to many for me and I did make that a point in many conversations and he seemed good with just the two. He keeps bringing up pregnancy and having another child. I mean he cuts it short but I can tell from knowing him that he’s dropping major hints. I’m not saying it’s a terrible thing to want but I get sad because everyone I find immediately starts thinking about a baby. Not only the two kids but I struggled with infertility with both and that was a hard time. Just feel down and more to life than constantly thinking about having a baby.