losing my boyfriend over family

Samantha • I am me.
So I am 22 and he's 23 and we have a one and a half year old son together.
I grew up with my dad not being around for me to talk to or just be there when o needed him the most so I don't have a relationship with him now as an adult. I have struggled with self image issues and forming relationships with people I usually avoided befriending women and was more hanging out with guys because I found it easier to talk to them but once I started dating I realized I like to find a way to ruin my relationships so I guess I'm not able to have long lasting relationships. I currently have zero male figures in my life because my grandpa and step dad passed away who were the only figures I had and my grandma has a husband but I don't talk to him much and can't stand him because he stole money from my great grandma and ran out on my grandma but they got back together.
Now my current boyfriend I really love and we haven't really had any issues except for his family who he said liked me but just this last year I found out his siblings had been trying to get us to break up so now I barely talk to them but the rest have seemed to be kind but he's getting mad at me because I have a hard time talking to his dad I mean when his dad asks me something I answer of course but if I have a question or need something my bf will tell me to ask his dad and I will say nvm I don't need it and he gets so mad at me! I feel broken and like we shouldn't stay together and I'm probably better off just being alone because of these issues I have.
Does anyone have any idea as to why I'm afraid to talk or ask questions to his dad unless he talks to me first?