starting to give up

I'm probably over reacting but I need a rant as nobody knows we are ttc. We started ttc over a year ago and of course after many tests, scans, ovulation sticks I still haven't fell pregnant at all. I'm currently on my 3rd round of clomid 50mg but not holding much hope anymore. My OH has a son who's 3 and didn't know about until he was 1.5 years old. Every time I try talk about our future he doesn't want to talk about it, anything I see and say "I can't wait to we have a little one to buy this" he says "oh why don't you buy it for my son?" He's already stated "he can get someone pregnant first try but not me" I got tested and still don't know if I have pcos because my tests were done on wrong cd and I have a low thyroid which my doctor seems to think why I'm not conceiving but when saying that to my OH he said "your thyroid is only one of the reasons" I don't get what he means? I don't know if there's anything else wrong with me, even the doctors can't say that! He can't be bothered to get a semen analysis done or pay for tests etc I've paid for everything. Then tonight we were talking about how big his friends family has expended (step kids) and I said jokingly "you've only the one we may get a move on" he said "one is enough for me" then said he didn't mean to say it like that. He's already stated his son will always mean more Because he's first born and they'll always have a closer bond (confuses me because he wasn't there through the pregnancy, birth etc) 
I don't want to have a baby who's not loved as much by their father. Should I just leave?