does this make me crazy?!

I addressed a concern with my boyfriend about joking with his coworker that he loved her. (She helped him get something at work done so he was able to have the next day off and his response when she told him he could have the day off was "I love you.") am I wrong to feel like those words should only be said to be even if they're just in joking to someone else? He says it to his guy friends too and even that bugs me. Am I crazy?!😭