Boyfriend was married before

So my boyfriend is 10 years older than me, I don’t really care but sometimes I feel like he expect me to be like his ex. We already live together I already confronted him and tell him that if he doesn’t feel comfortable with me he can leave before we have kids or married(We are TTC)the whole deal is when we have a discussion he is like you should do this and this is how I like it im use it like that before or I always bought this when I had my other apartment. ( when he was married with his ex)Thats when he pissed me off and tell him if he wants it like before he can go and look for her so she can do it. He stop talking and tell me that he is sorry and he doesnt mean to compare me with someone else. I dont know if I’ve just jealousfor nothing or I should be worry.

He was divorced 4 years before we met and his ex already married with someone else and as far I know she is already having a baby.