Don't know what to do.

So my boyfriend & I have been invited to go to Oregon in Sept for one of our friends wedding. This will be a week long trip. I have a 6 year old daughter with a previous relationship. Her biological dad had been in & out of her life since she's been born. He took me to court with in the month she was born to get custody, because of his record & the fact that even at the age of 17 (me) & he at the age of 18-19 I was the most stable I had a job & went to school & i was also breast feeding, I ended up getting sole custody & he had visitations he went MIA right after court came back a year later saw her at the most a handful of times & disappeared again, same cycle since. Usually when he comes around I try to work with him for my daughter, I can see she gets excited which makes me happy but also very sad when I see her face of disappointment when he dissapears. This last time he showed up I stopped giving him what he wanted NOW it was my rules when I wanted. (note: my mom is the mediators I never really have in person contact with him only thru a text app where he can call his daughter when ever he wants. He saw her once & called her maybe 10 times now he went MIA again. We want to take her with us to the wedding but someone told me i would have to get his signature authorizing it even though I had sole custody, But because he changes his number every month I can never get in contact with him. Would I be in legal trouble if I just bring her with without his signature?