I need help!

Alex
So I am 18 almost 19 and at my heaviest, I was 225 at 5'5. I then decided to lose weight and became a HUGE fitness buff. I lived at the gym and was about 160. I lost that weight in 5-6 months. However, when I got married I ended up gaining some back. I moved to a bigger city and didn't want to go to the nearby gym. And I would rather of gone out with my husband or hung out doing something than work out. So, I got up to 180. I wasn't TOO big but clearly a little overweight again (I had high muscle mass and low body fat). So then, I got pregnant. And I could see myself gaining pounds and losing my body. But I guess it didn't effect me because I couldn't see my weight. I went to the Doctor for my 18 week check up and I was 204 pounds!! 204!! I've never felt so depressed. I feel like all of that work was just flushed down the toilet. And my husband fell in love with a fit women, not a women who is overweight. And I hate when people say "He should love you for you!". You're right. But he isn't obligated to be attracted to me. Because no matter how you cut it, body attraction is important in a marriage and some men just aren't attracted to bigger women. And assuming from his exes, he definitely isn't. I want my body back for me and him! I am 204 right now and haven't worked out since I was 6 weeks pregnant. Is it too late to tone up and try and lose weight over this pregancy? Obviously by doing it correctly and clean eating, not by restricting calories. I'm 20 weeks now and I feel like it may be too late...I want to get down to 180 at least. 
Opinions?