Purse with diaper bag?

Ladies, I usually carry around my diaper bag and a purse. My purse can either be a tote, handbag or wristlet. My husband says I carry around too much and should just cram a few diapers into my purse cause I look silly carrying all that around. How do you guys do it? Silly discussion I know but it got me wondering