Sleeping with my daughters father cousin

So I have a daughter and her father is not in her life, she sees him maybe 2 a month. So while working at a job I started dating a guy. It wasn’t until 6 months into the relationship when we found out that he was related to my daughters father. The family isn’t close at all. He introduced me into his mother and grandmother and his entire side of the family. They all know who I am and say don’t worry about what they other side of the family say because they exclude themselves from them. My daughters father doesn’t even know his own cousin by name. They are not close cousins they are like 3rd cousins. But should I feel bad? We have a child on the way and he picks up all the responsibilities of what his cousin lacks. Any advise would be helpful!