What next to do?

Me and my husband for over six years been trying to have a baby and still nothing I'm getting frustrated about it and I feel he is too. I got myself check out and they said I'm okay to have kids and the doctor even said to me why I'm I here cause I have three older kids from someone before. My husband don't have any kids, but he told me one of his ex girl was pregnant for him, I find it hard to believe cause she never had his child. I think she lie to him about that. The thing is I feel like he wants to leave me to go and try it with someone else to see if he can have a baby cause he say maybe me and him is not a match. What should I do?? Please help me find some answers.