Do you wake your baby?

We have a 4month old who has 3 naps a day. His first is usually short and he wakes himself. His second is after dinner and we wake after about 2.5 hrs. His third nap is late afternoon. Sometimes he wakes himself but we wake him if it goes past 5.30 so it doesn't affect bedtime. My husband thinks we wake him too much but my concern is that he could have too much day sleep and not sleep at night. He gets between 3.5-4.5 hrs of day sleep a day. 

Do you wake your baby? What would you do?