Am I being Petty?! Need support/advice

So, my husband has this female friend of his that I thoroughly dislike.

When we first started dating, he broke up with me because he said "You should know that I don't think I can love anyone else because I have always loved (females name).

Well, I stuck it out we ended up back together and broke up again because he said he didn't ever want a family and I said well I do. A few months pass we dated other people and cross paths again. He says he realized that he wanted to be with me when I started to date someone else and he didn't want to lose me and our future family.

Fast forward 4 years we've been married for 2. I've never felt comfortable with his friendship with the girl he originally said he wouldn't love anyone else because of. He sees nothing wrong with it at all. He had pined after the girl for years they tried dating it didnt work... and even by the time I came around he was still in love with her.

I know that he is my husband but am I being petty? It's causing problems in our marriage because he continues to be friends with her. He goes over there after I'm in bed to hang out and smoke (i don't smoke so I think that plays into it).

He doesn't tell me... he waits until I'm in bed or says he is somewhere else. I trust him in the sense that I dont think he would physically cheat... but I'm insecure about their relationship. He's shady about it and gets defensive and angry if I ask if he's seen or talked to her.

Am I entirely out of bounds... because this seems shady and inappropriate to me... (i know I shouldn't snoop). Granted not all their conversations are like that but still. Anyways, if it affect us then why isn't he willingly to just not be friends with her... she only talks to him when she's single. It grinds my gears...