Not a rant, just need advice

My boyfriend and I have been together over 2 years now. I have 2 kids, and he was 1 from previous relationships. He is absolutely amazing with my kids! I feel horrible, because I just cannot seem to get that "connection" with his son. His real mother has given us a lot of problems since we started dating. Im not sure if thats why, or what. Is this normal or am I just a horrible person, and going to be a horrible step mom?? I hate feeling this way, because I'm not that kind of person. I never let it show, I'm always attentive to him and care for him like he is my own, but at the same time I count down until he goes home. What can i do??