Is there ever truth behind baby’s head being “too big” for a vaginal birth?

I want to preface with the fact that my situation is a little more complicated because of the fact that my baby has mild ventriculomegaly (dilation of the ventricles in his brain). So I went to an MFM and found that the dilation doesn’t have any complications for him and isn’t related to any genetic disorders. They say it’s “his normal” which is great news. But his head is measuring large. The rest of his body is too, but his head is the main concern for birth. The MFM doctor said that I could try for a vaginal birth but that I should be mentally prepared for him not being able to fit and needing a c-section. I’m 36+5 weeks and have another ultrasound coming up in a few days to see how big he is. At the last one, his head was measuring 4+2 weeks ahead circumference-wise which the MFM doctor said was above the 99th percentile. My ob says this upcoming ultrasound will tell us whether or not it’s worth it to try for a vaginal birth but doesn’t seem too optimistic.

This is my first baby and I’d say my frame is only very slightly on the smaller side of average. I always see people on here saying to never believe a doctor saying that baby won’t fit, but is there ever a case where you should believe them?