gaining weight

So in my first trimester, I lost about 20 lbs and kept it off until a few days ago when I checked my weight. I am now 29 weeks and gained 4 lbs out of no where. Is the 3rd trimester where weight gain is the most prominent? Should I expect to gain all of my weight back? I'm assuming it's just from the baby packing on fat and such.