Workplace Attire

My boss asked me if I knew what time of year it was. When I asked him what he meant he laughed and said I wear too much clothes for it being summertime. I told him I get cold easily and that it’s cold in the office opposed to sitting outside all day. He then said maybe by winter time I’ll wear less clothes since it will be warmer in the office building... wtf... this is my outfit today... I work in a business casual setting... tell me I’m not the only one who “wears too much clothes” to work???