Has anyone not expanded their family because of the older kids?

So I think we’re really good parents (nobody is perfect of course) but we try really hard. We have two older kids and a baby. Our oldest has always been a bit difficult but we’ve seen doctors and he’s very high strung and compulsive but no diagnosis. He was doing well until Covid hit and we’ve been in lockdown over half the year (no school or activities - were in Ontario Canada). Anyway long story short we had a baby July of 2020 smack dab in the first wave. He has been complaining about how I don’t spend as much time with him and that it’s all his sisters fault and so on…but in reality it’s because I’m home all day with them and he’s difficult so my husband puts them to bed to give me a break. When I go back to work it’ll be me doing it. Anyway so I feel like in the future I might want another baby, but our oldest said that he does not. Of course it’s not his choice but is this a sign that maybe we shouldn’t? I can care for everyone and have time for everyone but it just worries me