Would it be controlling of me to ask my husband to stop talking to this girl?

So I am pretty newly married and husband has this friend who is really inappropriate with him. He doesn’t say anything wrong but she does. She texts him constantly. If she knows we are out on a date she will call him. She does not like me at all. She has came to our house before and ignored me the entire time. She laughs too much at everything he says and it won’t even be funny. He told me he was hungry one day I asked what he wanted to eat and then I went to make it and she got up and told me she would cook for him maybe that’s nothing but it was just weird to me. I have been waiting for him to say something because I do not want to come off like I am controlling but I feel really uncomfortable and I have had a lot of resentment building up because of this. Well I tried talking to him about it last night and he basically told me he doesn’t flirt back with her so it does not matter. But to me it does matter. I don’t know if I should just ask him to stop talking to her or not and I don’t know what I’m going to do if he says no. I don’t want to be irrational and divorce him or something but I also feel very stupid right now and like he doesn’t care about me at all because he is letting her flirt with him and even though he is not doing anything back it’s the fact he has not put a stop to it. Am I being crazy?