How do I handle this?

Long story short im a single mom with a 5 month old daughter, im living with my mom and stepdad while i go to school. We live in apartments and our neighbors are this guy named joel, his wife and their 2 boys, school aged kids. Well apparently joel texted my mom saying how he'd like to go on walks with me (i go on a walk almost every day after i get back from school with my daughter) to get his kids out too. I think he texted my mom cause he doesnt have my number, but idk. So my mom just texted me and relayed the message, but said she doesnt think its a good idea cause it could cause problems with his wife. He also asked my mom if we would want to go fishing with him. He's a nice guy and i think he might just be wanting some friends, but i do kinda see what my mom is saying when it comes to just me and him walking with our kids. I definitely dont want wife drama. Which is kinda funny cause im actually 100% into girls 😂 people just never guess that cause i have a baby i guess lol. But anyways, how should i handle it if he comes up and asks me? I dont want to be rude, and honestly i could use a friend too i just moved here last year and havent really made any friends yet. But i dont want it to become an issue. I was thinking i could just invite him and his kids to go hiking with my mom and stepdad and i all together on the weekends or something. Does that seem fine? I just dont know how to handle this and what the boundaries are/should be, cause ive never been married or even in a relationship really