If you have to cancel a trip with your bf for medical reasons, and he gets mad, is that okay?

I had a trip booked with my bf, but I was later notified to do a medical procedure that would wipe me out for the entire weekend. I told him to cancel because it would be physically taxing and I would be in a lot of discomfort if we went. He got mad. And was also upset about having to pay the non refundable service fee. Is this ok? Just want other people’s perspective