Feeling selfish

I'm a first time mom to a wonderful 5 month old boy. I've always wanted to be a mom and have 2 kids close in age. I love giving him all my time and attention but my husband and I are thinking about trying for #2. I keep going back and forth whether or not to do that because I feel so selfish. I feel like having another baby will put my son on the back burner. I wouldn't purposely do that but I have a feeling my attention will shift from him to my pregnancy and there won't be any balance. Am I worrying about nothing or should I wait? Anyone with children close in age have any advice for me??