I declined a job offer that I got in my fifth month

I have been unemployed for an year now. I traveled to US with my husband and did not have work authorization for a job. When I got one, I was pregnant 4+ months. Still I tried and got a job within a month from the time I was eligible to work. My start date was mentioned as nov last week and my due was feb second week. With just 3 months in and thought letting the employer know after I join May not work.  I had so many questions in my mind about taking care of baby.. as I was not eligible for FMLA or short term disability I may not get leaves.. what if the manager did not agree for work from home.. so many thoughts and questions... So I declined the job offer. I know the decision I made was for my baby.. still I am feeling depressed on the fact that how good will I be considered in job market after 1 year 6 months of gap. What do you all think?! What I did was right?