Sex has dropped

Me and my boyfriend used to have sex several times a day in the beginning of our relationship. If I went a day or two not wanting sex he would become a little moody but be okay. Now he is the one who will not want sex and says I need to enichiate sex if I want it. He has also made up some excuses here and there about how he needs time for his penis to recover because the muscles are too sore to have sex. Should I be worried that he is no longer interested in me?