Am I being crazy or would you be uncomfortable too?

First, a little background information. I’m 35 weeks pregnant with mine and my husband’s second child. We’ve been married for 2 years and we have a two year old. We recently (like a month ago) found out that he has a daughter who is 5 years old, going on 6 at the end of July. The mother of his little girl said she was afraid to tell him when she got pregnant and only told him recently because the little girl started asking about a daddy and why she didn’t have one. My husband and the mother never dated, it was kind of a hit it and quit it type thing. He claims he never had feelings for her, and the reason they stopped communicating back then was because she had feelings for him but he wouldn’t date her. Fast forward to now, things have been going great. We really get along with his daughter’s mother and the rest of her family. We’ve sort of become one big family which I am grateful for. I don’t hold any kind of grudge against the family or anything like that. I’ve always said the past is the past and my husband was a person with his own life before I knew him too. However, today when my husband and I dropped his daughter off at her mom’s house after spending the afternoon with her, she asked why my husband and her mom are not married. (Apparently a kid in her class told her mommies and daddies are supposed to be married.) My husband explained to her that it was because he is married to me, but I guess she doesn’t really understand. My husband is a truck driver and left out today to be gone for the week. This weekend when he gets home, we were supposed to go about 2.5 hours away to get some goats that were an Easter gift to his daughter. (They’re all she has talked about/asked for) He called me a little bit ago and said that his daughter’s grandmother had suggested that my husband, the mother, and the little girl go to pick the goats up alone so that they have a chance to explain our situation to her a little better. Honestly it makes me feel a little uncomfortable, but I also feel like I’m in the position that I can’t say that because then I will seem jealous/possessive. I trust my husband, but something about them taking an almost 6 hour trip together makes me uncomfortable. Not to mention, I only get to spend 2 days a week with my husband since he is gone the rest of the week, so taking 6 hours of that time away seems extreme. The grandmother also said she would like to use that time to get together with myself and my mother in law and work on stuff for my baby shower, which is next week.

Please, I would like some honest opinions on the situation. If you think I’m being jealous or possessive, tell me. If not, what do you think, or what would you do? I know the post was long so if you took the time to read it I really appreciate it.