Help please

Me and my partner have just had an argument as Mother’s Day weekend I’ve said I’m going to my mums for a bit but his told me that his going comic con and his taking our daughter with him whether I like it or not. She’s exclusively breast fed so I don’t feel comfortable her going yet. And it’s my first Mother’s Day with her. His on the sex register so I don’t know where I stand in stopping him taking her. His said this a few times that if we split up he would take her our daughter etc.I know this sounds silly as I know men deserve to have there children alone time but it’s Mother’s Day

Edit- his on the sex register as he slept with a 14 year old when he was 19 he spent 1 year in prison and 10 years on the register for it which his done 5 of them years already. Social deem him no risk whilst his with me