Using condoms after 3 years

So me and my boyfriend have been together for 3 years. I’m almost 30 and he’s almost 35. I have a child whose 5 (turning 6 this year) by a previous marriage. To really understand where I’m coming from with this situation, I have to explain that during the 3 years we have been together, we took a break one time and that was last year in March. Come to find out he was talking to his ex and I broke things off. We were still living together, all except for 2 weeks when I stayed with my mom. Long story short, when his ex found out I was still living with him, she eventually got tired of it, Gave him an ultimatum saying he either kicks me out or she’s done. (He never kicked me out) I would t go as far to say they were even in a relationship, more of “friends with benefits”. Anyways the girl messaged me on Facebook not too much longer after they stopped talking. We went back and forth for a while (not arguing) and I asked her if they used condoms (I know kinda an odd question) but I wanted to know bc he used them with me. She never gave me a direct answer. She said “I personally don’t use condoms. But maybe I should now”. So I told my boyfriend in the weeks/months of arguing with him about this ex of his that she told me they didn’t use any condoms at all (just to see what he would say). He told me they did all except one time. So 6 months goes by and we didn’t have sex. AT ALL! He had this heart condition and was on a lot of meds and says “he couldn’t get it up even if he wanted to” bc of the meds. (Also the “one” time he had sex with his ex without protection was bc he couldn’t get off bc of the meds, that’s what he told me). So after 6 months without sex with him, we finally did it! Without a condom, and a couple times after without anything. Then all the sudden, after having sex a couple months without anything he wants to use one! Should I be offended? I mean I’ve been with him for 3 years and I’m actually tired of using them. If I get pregnant then it is what it is. I felt offended in a way bc he had sex with his ex without one at least one time that i know of. I told him if we have to use them then let’s just not have sex. He says I should compromise with him. But I don’t want to. I mean that might sound selfish but we’ve been together for 3 years and have lived with each other the whole time. He’s basically a step father to my son and my mind set is what ever happens, happens.