Dick sizes

Okay guys im really troubled, today my boyfriend asked me if his dick size was bigger or smaller than my ex. And we literally got into an argument about it because i didnt want to answer his question. I dont think it matters and he shouldnt care so much. Am i wrong for that? I couldnt even give him a straight answer, and i didnt wanna lie to him but i didnt wanna tell him the truth either so i gave him the answer the best way i could. Opinions?