Getting frustrated with my man!!!

My man is becoming a real pain! He's started talking loads about us starting our family the last couple of months and so we started trying but then when we've been having sex he's pushing me off him as he cums. My last 2 periods have been 2 or 3 days late each time, when I've always normally same day every month. The frustrating part is when he gets disappointed and miserable when my period starts, because he says he was hoping I was pregnant. I can't win with him, how can I make him realise what he needs to do if he really wants us to start a family. Not sure how much longer I can put up with his on off attitude to having family.