Why is it such a big deal??

Why is it such a big deal for a 20 year old guy to be dating a 16 year old female? Let's be clear. This is my situation and everyone judges it saying all he wants is sex. He doesn't! He is the most caring and compassionate guy that I've ever met! All he wants to do is cuddle. It's really no different from a 28 year old dating a 40 year old.