Can't make up his mind!!!

My husband and I have been together for almost 6 years. When we first started dating I made it clear that I wanted more children(I have 1 from previous relationship) He took on the role of daddy to my daughter and he is wonderful with her....but he doesn't seem to want anymore. He says if it happens we will deal with it but we just aren't ready yet...I think he's scared...I've been ttc for the last 2 years without him knowing but it's stressful doing it alone. Am I being selfish here??