My brother in law has just told me he fancys me I've been with my husband for 5 years nearly a whole year married we have a baby girl

What the actual fcuk right I've told him he can't feel that way as its not fair on not just me but his neice and brother

How the hell do I tell him to do one but also not affect our relationship? I want to tell my husband