It’s complicated...

So I need some advice I’m 21 and I’ve been hanging out with a friend from back in high school. He is a single dad with a beautiful 2 year old little girl. We’ve been talking for a while and well we aren’t a couple, but we basically act like boyfriend and girlfriend. I lost my virginity to him a while back and we continue to have sex. I have feelings for him, but I know this relationship of ours isn’t anything serious for him because we stated that this was just going to be a casual relationship between us, such as the sex, kisses, and cuddling. (Side note he’s also reluctant to having a relationship For his daughters sake of getting attached) Am I stupid for going on with this knowing that I have feelings for him? What do you guys think?