Am i being selfish?

My SO and I live about 3,000 miles from his family.

My family has come and visited me, but his family has not come to see him. This has made him depressed because he really misses his mother. His mother is the only one who works, so I understand money is tight for them. My family just left from visiting over memorial day weekend and he is suddenly pushing to go and visit his family. I don't have a problem with going and seeing his family at all, but he is not willing to plan it out very well. He literally wants to jump in the car tomorrow and go.

Reasons why I have a problem with it: 1. I'm 30 weeks pregnant. He doesn't want to stop at a hotel and waste time and money, he wants to sleep in the car when needed. 2. We have a 18 month old and I don't want her trapped in the car for 20+ hours. She gets cranky being in her car seat for an hour! 3. We just spent a lot of money hosting for my family and we still have bills that need paid.

Am I being selfish in wanting to wait a week or 2 to plan it better and see where we sit financially before jumping headfirst into this? We have been fighting about and he is making me feel horrible like I'm keeping him from his family when I'm not at all...