My boyfriend and his brother's baby mama too close? 🤔😕

Don't know if I'm overreacting or should be concerned but my boyfriend and his little brother's baby mama seem a little too close to me. Today my boyfriend was at his mom's house (helping with some yard work I'm guessing) with them and he told me (just gone call this chick N) "N scared the shit outta me today, I'm sitting on my bed you know playing the game and I feel something coming through my pants and it's her messing with my ass! 😂" and then he said that her response was "oh I thought you was T" (his little brother) MY response was "mmm" I don't like N and it makes me feel uneasy everytime he talks about her when they get around each other. She goes to my boyfriend for everything even with problems in her relationship with his brother. My boyfriend is always talking about her and the way he talks about her makes me feel like maybe he might like her or something, I just don't feel right.... About an hour after he told me about that he goes to say "oh yeah I'm gonna be gone all weekend" (AFTER looking at his phone then hiding it so I can't see who he's talking to, he never goes out on the weekend's or at all so it just seemed odd to me that he wants to go out all of a sudden) I just said "... okay" and his response was "oh you not gone question me about that??" I told him "nope..." and just kept my mouth shut for the past few hour's. He's the type of person that, if he's up to no good or trying to do something he has no business doing, he'll be extra lovely and affectionate towards me. He's not the affectionate type at all so I know when he's up to something... I may be overreacting but my instincts are screaming something doesn't feel right