not agreeing on Dad's family to watch my daughter

So I have my own housekeeping business and for the last few weeks I hired a close friend to watch my newborn daughter(which I never do) because I'm very overprotective with my children this is my 2nd baby the oldest is 8 I'm strict on who watches her and my son so today me and my boyfriend were due in for a housekeeping gig and we weren't able to make it this morning so I'm calling my friend this morning and said I cancelled the gig so you can go home and relax for the rest of the day so as I'm talking to her my boyfriend shouts out in the background"oh my aunt can watch her" so I tell my friend let me call you back lol so I tell my boyfriend that will not ever happen she's not someone I trust or will ever leave my daughter with EVER so he got upset and walked out the door the issue is here my boyfriend's family don't have it all together (mentally) why would I leave my newborn in that type of care?? They smoke, cuss, and do drugs so am I wrong for feeling this way ? I want my daughter in a positive environment not at his aunts house my friend has done a wonderful job with my daughter so far he even wanted his sister who doesn't have custody of her kids watching her please tell me am I wrong for not allowing anyone outside my friend to watch her??