Am I doing something wrong?

I recently lost my virginity to my boyfriend, I was also his first. We've had sex twice and I've given him a blowjob too. But hes never finished and while giving him the blowjob I felt like he just wanted me to stop after awhile. He was making noise so it couldn't be that bad. Could I be not good enough to get him off?