He told me not to worry but should I?

This is what my boyfriend admitted to me last night, but before this he told me not worry. The only reason this was brought up is because I told him I was considering going on a "double date" with my brother, and he assumed my brother hooked me up with someone (not the case) because it will be between my two brothers and their girlfriends... I was gonna tag along because my mom said I need to get out more. Anyway, he said he would've been sad if I was hooked up with someone else.

I know the whole "you deserve better" thing can have many meanings, none u of them good.... But I am overthinking and just need to know if this is worth worrying about or not.