I need some opinions...

So today my boyfriend and I were talking about going to my grandma, he asked wether I wanted him to go and this is pretty much how the conversation went.( Keep in mind I’m a stay at home mom to our almost one year old and have been since he’s been born.)

Bf: So do you want me to go.?

Me: Well grandma invited you so it would probably hurt her feelings if you don’t go.

Bf: Well I take care of her granddaughter all the time so I feel like I’ve paid my dues.

Me: Well you know you can leave at anytime.

BF: I can’t because I can’t trust you to take care of my son.

Me: Well why would you laid down and make a baby with someone you didn’t think could take care of you child.?

Bf: Well that’s not exactly what you think about at that time.

Me: We were intentionally trying and have unprotected sex for over a year.

I’m so hurt and confused. Idk what to do or say honestly. How could you say that to the person who takes care of your child 24 hours a day most day.? What do you guys think I should do.? Or what would you do in this situation. I haven’t brought it back up to him and I didn’t really say anything after that, as I was getting myself and my son ready.