Pushing him away...

Since having our baby who is 11 week old I keep dreaming that my boyfriend is having an affair, with 2 different women and they are always the same 2. It's getting to the point where I can't even speak to him some days because i genuinely feel it's happening, I feel like I'm pushing him away to prepare myself for the worst. It doesn't help that there isn't any intimacy between us really, we never have date night or anything like that. I really don't know why I am so paranoid because he literally is either at home with us or at work which he is only there 4 days and then has 6 days off and is with us and when he is at work he rings me from the works phone a few times through the day. Through the day we have both children so our time is always taken up with them, then of a night time we sit on separate sofas and within minutes he's asleep, no conversation. I'm 24, he is 37...I feel like I'm going mad. Money isn't an issue so there is no reason why we can't have 1 night a month date night but I feel like he just doesn't want to have that time with me, we went for a weekend away just after baby was born and we where in at 10pm because he was tired and when we were out together there was no conversation, it was just awkward. I feel like we are together for the baby that's it, it's not a miserable relationship, through the day when my 5 year old is home it's all laughs but the between me and him there's no chemistry. I don't know what to do.