Dirty talk trouble ..

Occasionally, and last night .. When we're having sex he wants me to talk dirty so I do it, if we're home alone I really go for it. But sometimes he has a go at me if I repeat the same things over and over, like hell be like goodnight then (in a jokey way) then we'll carry on but I lie there like 'you're just killing the moment' u know? I feel bad that I'm not that confident, idk why I mean we've been having sex together 3 years. I just wish he didn't have a go at me when we're in the middle of intercorse. It ruins the moment and makes me feel horrible... Sometimes he apologises after and once I even cried and he felt terrible but still does it. Am I in the wrong? What can I do / say to build my confidence? :(