relationship problem's

so my significant other and I have been together for 3 years on and off. I have a 4 year old son he will be 5 this month. his biological dad left to California when I told him I was pregnant. my almost 5 year old knows my significant other as daddy. my son was one and a half when my significant other end i met up again. my significant other and I went to middle school and high school together. a week ago I feel I had to decline his proposal I felt we where not really. my family absolutely hates him but his family loves me. Idk what to do my heart says to love him but my head says to leave him. I need some advice ladies. I want to spend forever with him but I don't want to lose my family.