Sex while having hpv.

My boyfriend and I both have hpv. I got it from him. The thing is he had his shits for hpv when he was younger so he swears that he doesn't have it and that I could of gotten it from my ex. My ex only had sex with one other girl and she was a virgin. My bf on the other hand has slept with a lot of girls before me. Well my boyfriend and I still have unprotected sex and my doctor told me that the type we have our bodies will fight it and it'll go away. But if we are having unprotected sex will it still go away? Or will it get worse?!