18 Month Old Favors Daddy

Does anyone else have a toddler who favors daddy over them, and always has? My 18 month old son has favored his dad since the day he was born, and ever since he has never let me hold him for more than 5-10 seconds without fussing and kicking/pushing off of me. The only time in the first year of his life when he would let me hold him was while nursing. I’m just starting to feel like he will never like me and it’s upsetting. I work out of the home and I’m gone about 10-11 hours a day and he stays at home with daddy so I’m wondering if that has something to do with it. I’m 27 weeks pregnant now with a little girl and am trying to figure out what I did wrong and how not to repeat it so maybe she will actually want something to do with me. :(