He's joking but still...

My SO and I have been tcc for awhile now.. he always makes the joke "I'm mexican so I'm really fertile" I know he's just doing it to get a laugh out of me.. but then I think maybe he's saying that and it makes me feel like he's blaming me for not getting pregnant. He already has one child so I know he can for sure have kids. But am I just letting my emotions get to me or what...? Just feeling really down about it