Need advice

My boyfriend and I live together, have for two years. I finally got pregnant after months of trying and was blessed with twins. My boyfriends cousin, who I known before him started coming over a lot. He has kids and a gf that don't really get along. Anyway, it started from him coming two times out the week to staying the night to now not wanting to leave. He says he had no where to go like that and needs somewhere to stay. He's not working and our apartment isn't that big. At first I let it kinda slide, but now it's getting on my nerves. He wants to be here everyday and sleep here. I will tell my bf he needs to leave so we can be alone and stuff or I just want my damn house back. I'm four months pregnant, I want to walk around comfortably. If we tell him kindly to leave he'll linger around our apartment! Stay at the pool talking to people until it closes, or just sit in his car faces our apartment. The other night he kept asking my bf if he could come in and charge his phone. I feel like he's using anything to come back in the house .Now he's upset and I look like the bad guy for kicking someone out with no place to go. I can't feel sorry for him cuz he's not working, he's not even trying to get himself together. He's a grown ass man! Am I wrong for feeling this way or what should I do?