Baby crying through feedings

I breastfeed & bottle feed. The past few days I noticed my son gets very frustrated when I breastfeed he’ll keep pulling off the breast, pretty much fight me & start crying. So I decided to take a break & mostly bottle feed, but still give him the nip here n there. He will eventually latch on to my nipple after a few minutes of fighting him. Today however, I noticed while I was feeding him with a bottle that midway through he started getting really fussy. He pulled away, cried for a minute until I offered the bottle back to him. Then it kept happening until he tired himself out. At the next feeding we tried breastfeeding & this time he kept fighting until he just gave up. I went to shower & I guess my boyfriend was trying to feed him & he would keep wanting the bottle & then not wanting it like he was doing with my breast. I took over & at first it was fine until he started crying. I again offered him the bottle, he would take it & then a few seconds later he would start crying. He eventually tired himself out again & now he will only sleep if he’s laying on my chest. Would any of you know from experience what could be causing this? He is 2 weeks & 2 days old so could this be a possible growth spurt?