Should I be worried??

So I am currently in a relationship with my bf, we’ve been involved together for 4 months (since August). He was in a 3 year relationship previously and had just ended it a week or 2 before pursuing me. His ex attends the same college as us and things are weird. When he knows she is somewhere around he makes me walk far away from him, he says she knows about me just not that we are in a relationship, he says that it would bring unnecessary disrespect his way. When we are together or when I’m with him she will call/text his phone and blow him up several times interrogating him and they are NOT together. When she calls he tells me to shush or leaves the room. He says that I have his word that he will fix this issue but he doesn’t know how to deal with it right now so he avoids it. What should I do?? because I feel like this could be a big impact on our relationship