Getting some feelings off of my chest

I just feel like my fiancé doesn't ever want to spend time with me or our daughter anymore. I feel like he would do anything to just get away from me. Like today I told him I want to go walking in the park. But once my grandpa says anything about going fishing, he just forgets about what I want to do and does what he wants. I don't know maybe it's my hormones, but I'm just feeling really down. I just feel like he rather go fishing then spend time as a family. I can't talk to my grandma or grandpa about it because they will take his side saying he needs his day to himself but he always goes fishing when they bring it up. I can't go cause I have my daughter and have to take care of her. I'm just feeling down and annoyed. Just not feeling like a family like we used to be.