Am I over reacting?

Last night my husband wanted to have sex while I was putting our child to sleep, I didn't tell him no I just asked if he would wait til the baby's sleep and he said ok. Ten minutes later he ask if the baby was sleep yet, I told

him no then he grabbed our child I thought he was about

to rock him to sleep. He puts our child on the end of the bed and proceeded to try take my pants off I told him no and to wait. He got really aggressive with me and started to say I never have time for him anymore (which isn't true). He then just yank pants off aggressively and to try to have sex I told him no but he wouldn't get off of me. Our baby was wide awake and crying but he still wouldn't stop. After he finished he just went to sleep and I was up having to calm a hollering baby and put him to sleep. I feel so bad even though the baby isn't old enough I feel like that shouldn't have happened.