Low sex drive

Does anyone have a man with a lower sex drive than they do?? Like my husband told me that he doesn’t want the marriage to be all about sex and also that he feels insecure since his penis is a bit on the smaller side. I personally don’t mind the size coz it works wonders? Any advice coz I wanna have sex almost daily....