Trying for a baby!

So my husband and I have decided to stop using birth control and to try for a baby. We aren't in a rush and aren't really pushing for it, but we would like to start trying. 
He is 5 years older than me and I feel like he is more ready than I am. I think it's the right time since he's approaching 30, we've been married for four years, and our parents are getting older. But I can't help but be afraid and to feel like I should refuse sex when I'm ovulating to "protect " myself. Part of me wants to get pregnant, but a big part of me really doesn't and wants to continue enjoying time with just us!  Have you guys experienced this? Is it normal? Am I weird?
Also, I'm terrified of gaining weight during pregnancy and having my body change and not having the thin, young, healthy body I currently have and work hard for. Have you guys experienced this? 
Thanks for your's all I can think about these days!!