Inappropriate or Overreacting?

So, I kind of want to take a step back from all the posts about recent current events for a minute. So humor me for a bit.

Earlier today, some maintenance workers came to my apartment to clean out the dryer vents. I was holding my 5 month old daughter and she was staring at the guys because they were strangers. One of the guys says to my baby "oh you like men already? I can tell!" The comment skeeved me out because I'm paranoid. I thought it was inappropriate. But then I'm thinking maybe I'm just having an overreaction because of my past, and stories that get pushed on the news and that I shouldn't be so serious. So what do you all think? Was the comment inappropriate or am I an overreacting mom? Let's discuss.

EDIT: I know the guy meant no harm. I'm more skeeved about the phrasing itself.