Feeling like a burden...

I have a 3 yr old son who is with me every other week. My boyfriend works nights from 430pm to 430am. Every day when my boyfriend wakes up (later than my son and I)

my son loves him like crazy and always wants his attention. The moment my boyfriend walks out our bedroom door when he wakes up my son is immediately at his feet wanting his attention! I can tell that it annoys him and he says we will never be a burden to him but I feel like my son annoys him because sometimes he will even go sit in the bedroom and play on his phone because my son just won't leave him be... he has no children and he spends more time with my son then my sons dad does which is why hes so attached to him... Advice?