Baby not rolling over

Krys

So my baby is 5.5 months old and he doesn't seem to be even close to rolling over. He can go from side to side like a champ though. I feel like it might be my fault because I hold him so much but he HATES being put down, especially on his tummy.

I'll put him under his gym, he cries; I take it away and try to just play with toys with him, he cries; I put him on his stomach, he screams. (I feel like I should clarify though that it's not always an instant reaction. It usually takes about 5 minutes and sometimes he'll even go for up to 30 minutes before he starts fighting.)

When I am playing with him on the floor, we practice rolling over so that it hopefully won't scare him. He's almost fully sitting up on his own and he can support his own weight standing up too. It's just the rolling that he seems to be having trouble with. It's like he tries (front to back at least) but then he gets fed up cause he can't and he cries. I just don't know what to do... Any suggestions?