Advice please!

I had a pretty heated argument with my SO last night when I voiced a concern I had and have been thinking about for a few days. My SO thinks it's a good idea to have his father watch our son once he is born after I take my maternity leave of course and need to return to work. I was never ok with this but didn't say anything because I wasn't sure how to voice my concerns with out being ignorant but he mentioned last night about maternity leave and how we could leave the baby with his father I instantly blurted out that I do not feel comfortable with that because his dad drinks and smokes cigarettes all day long and I will not be able to focus at work not knowing what is going on with my baby while I'm gone and he is with him. My SO instantly thought I was judging his father and talking crap about him. I explained that I wasn't judging him on his habits I simply don't approve of his habits if he will be doing them while looking over my child. He still doesn't understand my point and thinks I am overreacting but seriously! I don't think I am!!