Should I let it go?

I'm seeing this guy who was married before, and he has 1 child. He doesn't talk about his previous marriage which I understand, but it takes him longer to open up to me about anything. When we started I wanted the relationship to be a secret and he wanted people to know about us. But I wasnt ready for people to know yet . Whenever we argue he just avoid me for a couple days then when he would talk to me he always act as if nothing happened. He's the first guy I have had sex with. Now I want go public with him but he doesn't want to. I feel like I'm forcing this relationship. And his friend have this huge crush on me way before I met him and he's aware of that. I'm questioning what it could have be like with his friend and he's so attentive and ready to do anything I want. I don't know what to do. What should I do?