don't want to start food yet

At my sons 4 month appointment the doctor said don't give food yet because he's growing great with just milk. He watches us eat but I don't think that's classified as interested, he watches us do everything. I'm perfectly happy just breastfeeding for another month or 2 and the research I've done supports my decision but my husband and his family are dying to feed him 😒 it's so frustrating. Why do we have to rush him? Just venting.