Baby doesn't seem attached to me

Hi all I hate to sound complainy but I'm feeling really down. My baby boy is healthy and wonderful and will be 4 month old this Sunday. Today he laughed out loud for the first time with his grandpa. I feel so down because no matter what I can only get a smile from him. He doesn't seem to notice me when breastfeeding and I'm afraid he associates me with just feeding and putting him to sleep (which he fights).

I play and engage with him every time he's awake (SAHM) during tummy time and talk to him lots, smile at him, kisses him. I don't know why he doesn't see to want me to even avoids eye contact sometimes than his grandpa who's only visiting from out of town.

I want to be the bigger person but at this moment I'm feeling so hurt and also a bit possessive of my baby. I don't want anyone else to be with him or touch him and feel like he's this 'blank slate' that no one else might taint.

I feel awful for writing this but this is how I'm feeling and I just feel down. Anyone went through similar feedings and am I just crazy?