Is it wrong to feel shitty that my x is hanging with my parents :(

So my x- husband and I have a child together and obviously him and I arnt together anymore. I am pregnant with my soon to be husband. My x lives out of state and when he comes to visit, he visits my parents and goes out to dinner with them. He is a 36 year old grown man that also brings his daddy with him on the 6hr drive. Him and I have been separated for 3 years (I couldn't file sooner since he would threaten suicide) and he think it's "wrong" that I moved on before the divorce was final even though we have been apart for 3 YEARS I was 18 years old when I married him and he is 11 years older. I fee so shitty and my parents don't get that it hurts that they go out together and take my daughter and leave me out of it :( this sucks  am I wrong to feel this way ?