Boyfriend trouble

I am 32 weeks pregnant I went to my appointment and after to go visit my sister! I am a former smoker but do not smoke since pregnant! Well my boyfriend is now calling me a liar and is mad because I went to visit my sister who smokes! He thinks when I am around her the peer pressure will cause me to smoke which is not gonna happen I don't like the smell anymore but that is my sister and I miss seeing her all of the time! How can he get mad and call me a liar if I didn't do anything wrong? What should I do he won't talk to me or sleep in the same bed?? I'm afraid he will do something stupid to try and get back at me for what he thinks I have done!! Advice I love this guy but this is a bit ridiculous!!