Confused

I have never wanted children of my own. Then I met my husband. He is caring, sweet, and takes care of me. He has always wanted children. Since we felt differently about such an important issue, I would not marry him. 6 years into the relationship he said "I would love to have children but I want to marry you more than anything. Let's not have kids. Let's get married and just be happy together." I finally married the man.

5 years into the marriage, I have fallen into a love that is even deeper than before. So much so that I now think a child (or two) is the LEAST I can offer this amazing man. I addressed it with him and he still wants kids so we've been trying to conceive.

However, his younger brother (whom we raised since the brother was 13) moved in with us once again. His brother is now 23. We provide all of his necessities including a vehicle and gas money because he is in college. His brother is VERY opinionated and is constantly commenting or criticizing our relationship. I don't think it's a good idea to go through a pregnancy with someone so critical of us in our house. I want his brother to move back into his own home (he has a house that was given to him by his family) so my husband and I can finally start our own little family since we're finally on the same page but my husband doesn't want his brother to move out. My husband said we need to give his brother 2-3 years so he will graduate and save money to buy a house with his own money.

It makes me sad that my husband put our plans on pause so easily. I feel like starting a family with me is no longer a priority. Am I being unreasonable? I need advice.