Boyfriend troubles

My boyfriend and I have been dating for about 5 1/2 months things are going great! We have so much in common, my family adores him, he is supportive in my interest and everything else. We’ve already been on crazy trips. So things are pretty serious. But I’m stuck in a thought loop because idk how I should feel about this situation. There will be times where I don’t want to have sex, but he does. When I say no he gets upset and will ignore me for a few hours and then later on say sorry. I know that’s not okay but idk how to explain to him that sometimes I just don’t wanna have sex. I need some advise on this situation. Is our relationship toxic? Is that a red flag? Please help.