not sure how I feel about this..

I have a fwb. We aren't really friends but actually met each other thru my ex because my ex was trying to fuck with me. I recently moved back to my hometown and we started hooking up about 2 weeks later. it's great. we message each other whenever we feel like cuddling or having sex but lately he's been telling me no. we s omehow moved onto his terms. he will message me usually around 11 thru 2 am to come over. which im okay with but when i do the same he always has an excuse. then when I tell him I can't he doesn't stopb messaging me till I give in. Im not sure how to go about telling him I don't like everything being on his terms. Also he did something shady a while ago but that's a story for another time. any advice ladies?