My boyfriend tries to keep my son from my parents

So here's the deal, my fiance and I currently live with HIS mom who gets to spend time with our 2 week old son everyday. We live about an hour away from MY parents and my boyfriend refuses to drive there for any family gatherings. My mom asks for 1 night a week with our son, but my fiance gets mad and starts a fight everytime I go over there and he wont even go with me. My parents invite him over and he says its childish that my mom is always wanting to see our son...what do I do?!