Does age really matter?

My boyfriend an I have known each other for 2 years but just started dating 8 months ago. He is 16 and I am 18. I love him so much and I'm so glad he is my first boyfriend. I want to spend the rest of my life loving him. It's just, our age is getting in they way. I always have to take a chaperone so I will not get into trouble, and I can't talk to him barely because of it. Someone please tell me I'm not crazy