Am I wrong?

So I been friends w benefits with this guy for about half a year now. I’m tired of just having someone for sex I wanna be able to go out & do stuff. Is it messed up to just cut him off because he won’t have time for me only when we have sex. We’ve hanged out a few times but it’s always at night. He works from 4 am - 3 pm. He has a 3 yr old daughter he takes care maybe three times a week. He works out everyday after work. Says he’s always busy. Is it wrong for me to ask him to spend some more time with me?