My boyfriend won't invite me over

I've been dating this guy for about 7 months now and he is my first serious boyfriend, so he's the first guy I've let my parents meet and the first guy I've invited over. I was very nervous at first inviting him over because I come from a loud family and he comes from a very quiet family, it's just him and his mom, knowing this I was worried he wouldn't like that but he loves coming over but he has never invited me over. Throughout our relationship he's been to my house A LOT and has conversations with my parents but I have never had a real conversation with his mom. We've talked when she would come pick me up but it was small talk. I've confronted him asking why he hasn't invited me over and if I'm a problem but he says he would love to have me over and he would but he still hasn't asked me to come over. Any advice on what I should do?