Am I the one wrong here ?

Hi everyone so I been married for about 5yrs we have a 3yr old and my husband has been wanting another for about a yr (a little longer )but we compromised to wait a few more yrs : I am currently working for the trauma team and I take on call hours at the hospital, it's awesome money but I wouldn't like the stress of it while being pregnant so we agreed to stop trying until I changed department. But lately he seems to always "accidentally" finish inside of me I notice when I get up of course ... he won't tell me and pretends he didn't  finish if I ask ... The last time I got pretty angry and said I will stop having intercourse if he doesn't want to respect my say n he got all offensive and asked why I don't want to have his child if we're married and financially stable that I'm wrong for putting work before family for so long and not caring about his feelings because he's been waiting another for so long, at that very moment a surgeon called me asking an important question about a patient and it doesn't help he's single , young and attractive (my husband has met him before) but my job is to always answer the phone, so my husband stormed out saying I always put work before him and I probably like all the attention from that specific surgeon cus he could of easily looked at the patients record instead of directly calling me . That same night his buddy from work and wife came over for dinner and mentioned they were happy to hear we are trying for our second and good luck I just said thanks and looked over at my husband who purposely avoided my glare .My question is am I wrong for not wanting another ?? I do want another child but I can wait a few more yrs . I love my job & I'm still in my 20s I have no rush . Is it wrong that I love my career and have a passion for it ?? My family (mom and dad) say the same thing that family should be first and they want another grand baby I feel selfish cus I'm married and I know he's been wanting another, & my child wants a sibling to play with  but deep inside I feel it should be my choice and I should have a say . I feel stuck 
Advice pease & thank u