Nipple Stimulation

When should you start to avoid nipple stimulation while pregnant?? Or should you avoid it no matter what in the whole pregnancy? I love when my man plays with my nipples and sucks on them and stuff but im trying to be cautious, thanks:)