Don't want to get pregnant again, feel bad for husband 😔

So I'm ~8 weeks PP with my first baby. I'm on the mini pill, which I know is a little less effective than combo pills. I LOVED being pregnant but my birth went horribly and honestly made me feel like I didn't want another kid AT ALL for a few weeks. Well I'm back to wanting a second one at some point, but just not right now.

I take my pill at the same time every day, never missed one, etc. etc. My period did come back before my 6 week appointment though - so I know I've already (probably) ovulated at least once. I literally always think "what if the pill doesn't work for my body, and I just keep ovulating anyway? What if it doesn't make any changes to my uterus?" - pretty much always finding something to worry/make me feel like the bc doesn't work.

My husband has been super understanding with me and wears condoms when we do end up having sex, but I feel bad because more often I just try to avoid it. I do desire sex and I miss having it with him, but I'm so scared of pregnancy. I've already talked to him about not wanting to be pregnant again, but I do think he's feeling a little rejected 😞 We went from fucking like bunnies our entire relationship (including pregnancy) and while a change in sex life is to be expected after a baby, ours has been DRASTIC - only because I don't want to get pregnant again. I just feel bad for him😭

Has anyone else experienced this? Did it ever get better - if so, what did it take?