To try for a baby or not?

I'm 40 yo with a child from previos relationship. My DS is 16, I love him dearly of course but I never had an easy time with him. As I was married to his father I was bullied by all in laws family including MDH and in 3 years of marriage lost all of my confidence or self respect. At a time I found enough strength to leave that toxic situation but as I was rebuilding myself , trying to survive ,  I hardly had enough space and energy to actually enjoy my motherhood fully. 
Now I am in a new loving relationship with the most amazing guy I ever met! He does not have kids, but wants us to try. I dont want to upset him and said yes to trying  knowing that , if I had a child i would love her and it could be my second ( and last ) chance to be a happy mother to a baby. But now as my child is grown, I really enjoy my freedom , I don't want sleepless night, lunch boxes , school nights and be tied to a house for years again. 
Your thoughts?