Not allowed to see my parents

My husband told me I'm not allowed to see my parents that live 3 hours away every 3 weeks. That it needs to be every other month and not every holiday. Because it takes time away from him seeing his kids (that he sees everyday) when i take them down to see their grandparents for 2 days. He said it makes him feel like he's not the priority and that I need to grow up and realize that when they die he is who I'll be spending my life with. He also doesn't like them because they don't like him due to us getting in an argument and it turning physical. They advised I file a police report, so I did but ultimately I had it all dropped. He holds a massive grudge against them for advising I file a report because "what they did was vile and in-despicably evil to try and ruin someone's life by sending them to jail" even though he left a bruise on my leg from kicking me. Anyways I ended up crying because they invited me down to their house to celebrate my upcoming birthday and see extended family coming down. I had to tell them I can't come this time -didn't say why to them, but the reason i couldn't was because he said if I go I'm choosing them over him and they're the "nastiest pieces of shit alive", and I already went down there 3 weekends ago. My mom was recently diagnosed with some serious health issues and I would like to see my parents as often as I can before her health declines and before my husband moves us all the way to the West coast where I will never be able to see them again basically. I told my husband that and he called me stupid saying she's not dying and that they were guilt tripping me into seeing them more by coming up with a sob story. He yelled at me for crying and making him the bad guy. So naturally I apologized and now instead of seeing my parents and everyone coming down I have to go help his mom this weekend move things out of her old home into our home because she's now living with us. I continued to cry and he cane back in an apologized and said he loved me but that he still doesn't think I need to go but that it's my choice and If I do go i can only go for one day and go straight there and straight back all in the same day (6+ hours of driving with a 5 year old and 2 year old). And that whatever i choose to do I can't hold it against him because I chose it ultimately. He kept saying i cant hold it against him and then kept saying I love you until Insaid it back.