Immune system development

I’ve been thinking, and is this whole being self isolating/quarantining going to effect my baby’s immune system.

The reason I ask, is that my husband myself and our baby have not left our house since middle of March. And we haven’t had visitors (we’re being extra careful because A. Baby and B. I’m asthmatic) and I’m worried that he won’t be introduced to a verity of germs and the like which can help teach his immune system to work. And as much as I was dreading him getting a cold, now I’m worried he won’t get one and will be sick later in life because his immune system didn’t learn how to be an immune system.

I know I’m over thinking this, but I worry a lot. And so far the only things that are challenging his immune system is his vaccinations