My boyfriend stays at his mom all the time!

So a little back story me and him has been together for a year now and we share a apartment together we live 5 minutes from his parents . Every day he don’t work he wakes up and says let’s go to moms and spends basically all day there, everyday he works after he gets off he goes to his moms and asks me to meet him there and stays there until 10-11pm . I don’t care about him seeing his mom and dad but it’s literally every single day almost all day long for him to sit there and do absolutely nothing.. Everytime I say something about it he gets defensive and says I have a close relationship with my parents I’m sorry that you don’t.. am I overreacting?? Should I be upset he’s at his parents ALL the time?? What should I do??