New mom & nervous to have sex

My son is 4 months old but was born 7 weeks early via emergency c-section. He is doing amazing and besides spending 3 weeks in the NICU and being small you would never know. He is our first baby so I am getting about 5 hours asleep a day and worried about him eating enough and sometimes I get easily stressed. My husband and I have only had sex twice since he's been born. He has been completely understanding and great. But the longer we go without it, the more nervous I get about doing it. I want to but I'm so tired and worried that it'll hurt or his views about me have changed since seeing me give birth. I've even lost weight and smaller now than I was before I was pregnant. It's irrational I know. But please tell me I'm not the only one to feel this way. I need some encouragement and suggestions please.