Relationship with older man

Michelle
I've dated men that were older than me before but getting older, he is the oldest man I've dated. (I'm 33, he's 43) I'm used to being able to have sex pretty well whenever I want. But with him, we had sex A LOT for the first couple of months. As soon as my son and I moved in with him, I'm lucky to get sex twice a month. It's either that he's too tired, sore or just wants to play games. But if I give a BJ, he's all about that.
Now, I KNOW that he loves me and I KNOW he's not cheating. He doesn't have time to cheat and I've checked up on stories, because of past relationships, everything has checked out. So I'm not worried about that. But I've talked to his sister in law about the issue because she has known him since he was 19. And she brought up the thought of it being him just being an older man and I shouldn't be worried.
Now, I know he would LOVE to have kids because we have talked about it but he doesn't think that he can have kids because he has been married 3 times and has never had kids but he has never been tested. I've been trying to time our sex here lately because I don't want to tell him that I'm trying to get pregnant because I don't want him to get disappointed if I don't and I don't want him to think I am disappointed because we don't get pregnant. 
But my main thing is, has anyone dated an older man in their 40's and if you noticed a significantly lower sex drive and what did you do to fix it? Or is it just something I need to get used to?