What to do?!?

So my husband and I have been "trying" to conceive since shortly after our son was born, and he will be 4 in December. My husband constantly makes comments about wanting to have another baby, but he won't have sex with me, particularly if he has any idea it's near my time of ovulation. Usually when it's around that time, he goes to "hang with his brother" all night long. I have mentioned this to him, and told him of my doubts that he's not really wanting another baby and he says it's all because of me, and I assure you, it's not. I've begged him to go to a fertility doctor with me just to make sure everything is ok, and he refuses saying that the problem is with me. I'm not sure what to do, and I absolutely want my son to have a sibling to grow up with. My husband had two kids prior to our marriage with 2 different women, but he promised me that he would love to have more kids with me. I'm 31 and my clock is ticking, and I'm clueless as to what to do or what to think.