first mother day -- relationship problem?

I'm starting to feel like I may becoming jealous and angry at the wrong stuff. Last Mother's Day I was pregnant with my son ... me and the father weren't together so I didn't expect anything. This year I am pregnant again and actually dating the child's father. We do not stay together but I asked him could we all go to the zoo for Mother's Day and he said "no, I'm going to be with my family"... I'm beyond pissed now... is this even a legit reason to be upset or are my hormones getting beat of me? It almost seems like he only asks me to come around when he's horny and then uses the baby as bait by saying ," come over so I can cuddle with your belly" or "your keeping my kid from me"..