Should I be concerned?

I don't get it. I have been married for 2 years and rarely get sex. My husband doesn't want me touching him in an intimate way. He rarely touches me. He can be very very effectionate and flirty verbally, but not physically.  
When we first started dating I gained 10lbs and he use to be very rude and ridicule me for my weight. After a huge blow up he stopped mentioning my weight. I am now 20lbs heavier since we married and he has not brought up my weight but never shows any physical attraction to me. Should I be concerned? Is there a chance he may find someone else? I am starting to freak out...