moving without telling baby daddy

If my child's father has not been in the picture, doesn't get ahold of me at all.. and I haven't heard from him in months since my child was a newborn (he is almost 2 now) 
Could I move out of state without telling him? If we didn't go to court but I have sole custody (I think that's what it's called) or I think it's physical custody.. I'm not sure 
Like would I get I trouble if I moved out of state? Sorry if I worded this wrong - I'm tired and just had this on my mind.