My husband doesn't want a baby?

The Little Mermaid
We've been having unprotected sex lately and on Sunday he ejaculated in me knowing the consequences. Today is Wednesday and he said that he is worried that he got me pregnant and wants me to take Plan B. Taking the pill 3 days later is probably not effective anymore. If he was really worried he should have told me to take it the day of or the morning after. 
Personally, I don't mind getting pregnant and having a baby now but if he is not happy then it's hard for me to be happy. I want both of us to enjoy the journey. I don't want to enjoy it alone. 😪 The reason why he is not ready is because his dream is to travel with me. We haven't done any traveling together. He wants to go to Vegas, Miami, Europe etc. I'm 22 and he's 24 so I understand and respect his wishes but now idk what to do... I feel guilty for not taking Plan B earlier and I feel bad this his dreams are gone because of having a child. A child is MY dream and traveling is HIS dream. We're married but on a different page so now I feel so guilty, confused and I can't stop crying. My heart is broke for him 💔😪 I want to see him happy as well because he is the love of my life. I don't want to only accomplish my dream of being a mother and ruin his dreams by not being able to travel or accomplish his goals in life.  😭