Any moms experience this

My son is currently in the nicu, he was born a week ago at 31 weeks. Well I got to hold him for the first time a couple of days ago and it was amazing he was so calm but today when I held him he was so fussy they had to put him back in his incubator. I got so emotional like scared that he doesn't like me. If he's in the nicu this whole time and I only held him twice so far Im scared that he's not going to know who I am.. 😢 I go everyday to see him but I can't pick him up like I want to. I pump and bring them the milk and they feed it to him through a tube so I don't even have that bond with him yet .. Any other moms go through this ? Its just so hard. I feel like all these other woman are there all the time tending to his every need ( the nurses ) and I can't even hold him like I want to