Breasts not sore - worried

Chelsea

My first baby my breasts were bigger and sore by week 4. This time I’m 4 weeks or further and my breasts haven’t changed at all. First ultrasound is next week. Should I be worried?