when is the right time?

Me and my boyfriend have been together for around six months and he's not a virgin but I am, their are times that I feel like I'm ready for us to have sex but when it gets to be time I always back out and he's really understanding of it all, any advice?