Dealing with Criticism

So, my husband and I both work night shift jobs. He works full time on a 2-2-3 schedule, meaning he works 12 hours half the days of the year. I work part time (6) hours on the nights he has off, so we always have one of us at home with our son. Our son also sleeps on our schedule, I get it’s unconventional, but it’s what works for us. We wake up around 1-2 in the afternoon and get him to bed around 1-2 in the morning. We still have ample time for “normal” activities and generally nothing is different from “normal life” other than the time on the clock. We get SO much backlash about “keeping our kid up all night” and the constant reminder of “you have to switch him eventually” which yes of course! We have 2-3 years to get him switched over and switch our schedules to something that works better for a school age child. I’m so sick of explaining that my child gets everything he needs out of life. I’m sick of justifying our “odd” lifestyle. Anyone have anything else similar? Am I REALLY doing my kid harm like people seem to assume?