custody??

Here's a little background; I am an 18 year old mother who had a premature baby (33w) she is now 3 months and doing very well. She's breastfed and loves it! She hates bottles. Her father wasn't around at all during the pregnancy (I told him and he left). I worked up until I gave birth and I decided to be a stay at home mom, and do online college classes I graduated high school early and saved up enough money to where it's possible for me to do so. Her dad wants me to give my baby to him (and his mom she's nuts and I doubt it's actually coming from him) every Saturday for 8+ hours and every Wednesday for 8+ hours, however her pediatrician said she shouldn't leave me for more then 6 hours because giving her extra bottles can affect breastfeeding and I would like to strictly breastfeed. He lives with 5 random people in a sober house (he is in recovery) including his girlfriend who is a minor.  He is saying if I don't go along with what he says he will take me to court and get custody of my daughter. Is this possible? Anyone help? We live in Minnesota