Kids with Age Gap

Okay so I’m 21 and he is 31. We have talked about having kids. But the thing is he doesn’t want to wait long to start and I wouldn’t mind having kids now. But he feels it is unfair for me to be so young. I was wondering if there was anyone that has kids and there was any extreme difficulties with him being older, or me being so young and having kids. We are very happy but having this discussion a lot lately. And we plan on putting a down payment on a house next year.