Sex.. Not doing what I said I'd do..

Should I feel bad for not giving in and giving him sex like he wants? I'm in my second trimester and I have zero to no sex drive.. But he seems to want it all the time.. Sometimes I tell him that we can do it if he lets me sleep for awhile but I end up not doing it and he gets mad.. I feel really bad but if I don't want it should I tell him so or hive in so he doesn't get mad at me? I don't know if it's the pregnancy or if it's me but I don't feel like we are close at all.. I feel so far apart and drifted from him.. I wish I could make it better and be the way we were.. 😔 any suggestions on what I should do?