Am I being unrealistic?

Backstory (You can skip if you just want to answer my question):

My boyfriend and I live in separate households. I have a daughter, and he has 3 boys. Our relationship isn't perfect, but we make it work and we love each other. I'm a teacher, and ATP since I went back to work out time is quite slim and I see him maybe once a week, for an hour maybe two at the time. He has said that he will devote more time, but in the 3 weeks since he said this, nothing has changed. We do talk daily, but overall, we're busy and I get it. He is very big on his, "independence," since he has this history of long-term relationships with previous women and living with them, and I was fine with this because I am too, to a certain extent up until we found out we we're expecting (please don't ask me why, it was NOT a planned ordeal and we protected ourselves, but here we are) and he's been great with coming to appointments and being there. He is at this point in his life where he prefers to be home, stay home, and that's fine, but normally, he doesn't want company as often. His days off are spent with his son he has 50/50 custody with except when the child is at daycare. I respect this because he truly is a great father and QT with children is important. On his rotating weekends, he has them both (the 3rd son is grown). He works 12 hour shifts, and I'm M-F of course, but ever since the beginning of time, I have initiated is spending time.

The Point (details that tie into the question):

Either way, I have been very patient with this man. Well, at 4 months pregnant, we have been having conversations about when the baby gets here. He tells me that he wants to take her to his house and keep her there. I said after an period of time, I was fine with that, but straight out the womb, I'm not comfortable with that being that I will be exclusively breastfeeding being out from March until July. He said he could handle that too and I told him you don't have breast. It became an entire argument of you don't trust me with our baby and I'm an involved father and I should be able to keep my daughter at my house when I want and etc. I told him he was operating in co-parent mode and that I thought we were in a relationship and moving toward marriage and cohabitating together. I understand we're not doing that now, but I don't think there's anything wrong with not wanting my newborn going from house to house that early. He told me I was being ridiculous and unfair to him if the only way he could see our daughter was if he went to my house and stayed over. I know this was long y'all, but I cannot sleep. This is heavy on my mind and I'm so sad bc I'm realizing we are 🚫 going to be a happy family.