Father in law with a key to our house


I'm a stay at home mom 35 weeks pregnant with a 1 year old my husband works long hours and I'm always home alone. my FIL use to live with us he recently purchased his own place and moved out ( thank god ) My husband wanted him to keep the key to the house for emergency since he lives close by which I really don't care if he has a key or not the thing is his dad now seems to think its okay to drop by for this or that without giving ME a heads up ( he still has to park his other car here but he doesn't use it and still gets some mail ) he stopped by the other day and here's how it happened he rang the door bell and before I could even get to the window to see who it was he was already opening the door with his key I'm talking like 1 sec later He was already walking in and is like I was in the area I hope it's ok that I stopped by ( he wanted to see his grandson real quick ) I think he noticed I was first off shocked and bothered because I know I had a look on my face like mmm okay WTF. I'm a very private person with social anxiety at times okay and if I'm going to have company even family I need to mentally prepare ! Now I know at that point I should of said something but I'm someone who needs to think before and choose my words wisely to not come across mean or rude. I ready don't mind him stopping by to see my son but he needs to freaken text me before even 5 -10 mins is OKAY !!! that he's on his way like WTF I could of been not dressed for company and stuff and my own family would never do that shit so it really bugs me I know for a fact if my dad had a key he would NEVER use it unless it was an emergency !!! And that's my OWN father. What do you guys think of this ??? I'm saying something next time he does it because I'm SURE there will be a next time. I'm worried after my daughter is born because I'm obviously going to be breastfeeding/pumping and with my boobs hanging out all over the house running after my 1 year old I don't need that man walking into my home with no heads up