Infertility causes serious relationship issues

I need advice! Ive been actively trying to get pregnant for 3 years now. My son is from a previous relationship and loves my boyfriend more than his own absent father. My boyfriend has started rejecting my son because of my problems getting pregnant. He wants one of his own and since its not happening he doesn't want to deal with my son. My son has started noticing the difference in my boyfriend and now is clinging more to me than ever before. Which I dont mind but he used to have such a great bond with my boyfriend I was the outsider in their lives. But I loved that. Everything is so different we cant stop fighting. He told me to take my son to his dads so it could just be the 2 of us. Which I would never do in a million years. Do I stick it out a little longer and hope for everything to back to the loving relationship we used to have? or take my son an go back to my parents house and cut my losses?