How to address this problem to my boyfriend?

Me and my partner We have a beautiful newborn 8 weeks old now. I love my baby so much and my partner my boyfriend would think it’s funny to whisper in baby’s ears that whenever I leave the room or whatever oh mummy don’t love you , mummy abandoned you etc negative words - whenever I hear these whispered onto our baby I am heartbroken as why would he do this ? He is doing this because he thinks it’s funny?

I tell him to stop and then he would reply saying babies don’t understand it. So why he would continue to say !!!! It angers me so much not sure how to approach it

When I tell him to stop the more he does it of