Does size really matter?

Have any if you ladies been with a man whos on the smaller size and a man whos more on the average/ bigger range.. if so was there a difference or does size not matter?