Upsetting

My boyfriend always says how his other son (by another female) has nothing to do with me bc he's my my kid and all this stuff knowing I pick him up from day care if needed, let him use my car to take his son places, etc.. Mind you his child's mother wished death on my baby during my last pregnancy before my misscarraige and much more... Just tonight he once again said his son has nothing to do with me and honestly I've had it to the point I feel if he really is gonna be an ass then my child will not be around his son or his family. I went through so much with him just to always be so rude. It's very upsetting, but who I'm around is who my child will be around so if I'm not around his son my child won't be either and if it gets to that point he won't be around my child either.