Scared he doesn’t like our baby😓

Me and my partner recently welcomed our baby boy into the world. My partner was great the first few days but after me and baby coming home and us all having broken sleep he is saying that he ‘doesn’t do babies’ and ‘will prefer him when he’s older’ am I right to be upset by this? I just look at my sons face and feel hurt for him😭