What can I do about it?

I’m sharing my son with his dad bi weekly. My son is 6. What bothers me is that my son wants to go to his dad all the time (reason for it is because there’s a bunch of kids there and they spoil them with toys and candy) even while in our home. I’m the disciplinary, communicative, responsible for his school and homework, buys him necessary things, buy toys and candy only when earned it kind of parent. His dad on the other hand is a bad influence, when he comes back from his dads house he doesn’t know how to be respectful, does not listen, thinks I can buy him all the toys he wants, doesn’t eat the food I cook (his dad buys him fast food) and so on. How can I explain to my son that his dad buys his love and not really care about him? Or should I wait and hope he will grow out of it? I’m not trying to compete, I’m concerned that when he grows up he might want to live at his dads permanently and basically live the life his dad lives. It’s sad to think about