Doctors reaction off :/

So I'm 40, have 2 adult kids from a young, early marriage. In a new relationship which I now have a 2year old daughter from. (not planned btw) Well my partner is now wanting to try for a boy, so yesterday had dr visit to remove my birth control so we can start trying. Well the dr was a bit taken back at my decision, not supportive at all. Kept telling me all the reasons why getting pregnant at my age was questionable. :( I want to have at least 1 more....am I wrong for wanting?