I think I'm falling for a guy who I'm not suppose to fall for...

so I've been seeing or talking to a guy he is older than me like a lot older and we had both made an agreement to where we would just have sex with each other occasionally. but I messed up and went and dated another guy for about 3 months and broke up with him. And then I went back to dating this older guy because I missed him and he does something to me that I can't explain. 😩 and when he found out that I was that I wanted to be back with him everything was good he flirted with me talk to me on the phone he would send me funny pict9ures and gifs and stuff. but something happened to wear it got out at my job that I was dating this older man because he worked there with me as well and a lot of people questioned it his daughter questioned it because she work there too. and got mad at me to where he didn't speak to me for like 2 weeks and I was devastated I started having panic attacks I couldn't trust anyone at my job I didn't want to go to work but

then two days after the two weeks he didn't talk to me he messaged me again and he wanted to see me and I did so I went to see him but now everything's good but everything was kind of different he doesn't text me as regularly as he did and doesn't talk on the phone with me as regularly, he doesn't send me funny pictures are gifs as he used to,he talks to me at work but it's kind of short and I just kind of feel like he is slowly drifting away from me and I don't want that and this is where I am starting to get the feel of I'm falling for this man even though we both had an agreement to where it was just strictly about sex and my friend told me to keep all my feelings out of it my emotions out of it but I can't and I don't know what to do?