My cat

My husband says once the baby is born the cat stays out...he is hardly in anyway but its almost winter and last winter he would come in more around dinner time to stay warm and sleep on his special mat then go outside over night. Also if he isn't on his mat he goes back outside as that's our house rules. 
I think if I keep the baby's door closed or put a tight net over the bassinet and obviously supervise my cat I don't see what the problem is because I'm so careful with him in the house even without a baby. 
Plus I can see his mat from the kitchen, lounge room and dining area! 
I know it will end in an argument, not sure what to do! 
Any other ideas? I don't want to neglect my cat either lol
Thanks 😉