Should I be worried

My baby father does not want to go to our baby's second ultrasound. We've together for three years. He went to the first ultrasound and we have an ultrasound coming up to determine the sex of our baby and he doesn't want to go because it interferes with his work schedule. One day I told him (at four months in to my pregnancy) that he would be a good father and he said "I don't know about that". So should I be worried?