Sex with another man while pregnant

For my whole pregnancy I was single so I decided towards the end to have sex with someone I communicated with for years. In the mist of this we ended up in a relationship after I had my baby. Things between haven been the best. I was told from other people that he will never respect me for the choice I made (having sex with him while pregnant). Has anyone been through this? I think the people are right he will never respect me. I know I was honest but men look at women different because of something like that.