Dr told me to lose 20 lbs before ttc. Bummed.

Hi there. So we have lovely 3 yr old and just moved to new home. And want to grow our family. I went for check ups before trying. And OBGYN said to loose 20 lbs before start trying. I'm am on healthy path of excersise and eating right. I have lost 8 lbs since Xmas. But super bummed cause I feel likes it's going to take forever to loose that much weight. I'm 150 lbs and dr wants me to be at 130. I haven't been that since high school. I just sad and frustrated. My mom and others just say to try. But hubby wants to wait so I loose more. So pressure is on me. Just sad about it. Anyone going through this too?