I had a baby by an idiot

Every time my baby goes with her dad I feel like he tries to keep her from me. When ever I call he always says she's sleep or gets angry and says why are you calling, I know how to take care of my child leave us alone I don't need supervision! which really upsets me because I'm not trying to supervise them I'm just a first time mom who wants to hear my baby voice. She's only a few months old. But on the other hand when she's with me he calls all day and we talk on the phone for hours at a time. I feel like he's showing out in front of his girlfriend that's why he don't want me to call him. I really don't even know how to handle this situation