Lower sex drive than my SO

My boyfriend has a very high sex drive and I feel like I let him down a lot which I feel horrible about. I find him incredibly attractive, I just have a lower sex drive than him (I take medication that has this side effect so that could be it.

Now he thinks I'm afraid to talk to him about it because I don't say anything until he leaves and I text him apologising or saying I want him.

Do any of you have any suggestions for sex drive or how to deal with differences in a relationship?