Hard to do activities with older child with difficult younger child

I have a 3.5 year old and 2 year old. Honestly I don't do as many activities as I used to do with my older kid because my younger kid can be very difficult. It's his way or the highway or else he throws the worst tantrums. Like he throws himself on the floor, throws his toys, screams. It's just very hard overall to have him in on an activity, especially when it's better suited for my older child. They both still take naps, so it's not like I can do things when the youngest is sleeping. Idk I guess I just feel bad and don't know how to go about it. I hope I'm not alone in this.