Boyfriend wants to use my savings for a down payment

My partner and I were together from 2016-2020. On and off since then. And back together, but more serious this year. He mentioned moving back in together. My lease ends in January. I’m still a little iffy on how I feel about living together again just because I’m worried something will happen and we’ll split again and then I’ll be stuck in a rough spot. We did live together from 2018-2020. We have a 5 year old together. He did pay rent 100% when we previously lived together and I payed for all of our sons things, bedroom stuff, ect. I still continue to pay the majority of things for our son. We have never done child support. He did give me between 300-500/year when we were split up. We talked today about the financial aspect of a house. I have a college savings that I’ve built up. I graduated this year, so I don’t need it for schooling anymore. But he mentioned that we’ll have to use the majority of it, if not all of it for a down payment because he said he won’t be able to contribute much. It kind of rubbed me the wrong way because it’s a lot of money and will completely drain my account. I guess I’m a little worried that he’s going to be relying on me to pay for the majority of it. Then there’s the worry of what happens if we split up since I put so much towards the house. I’m all for going 50-50, but if I’m paying the majority plus our son’s things then I feel like maybe he’s taking advantage of it. He also doesn’t make the best financial decisions, so I worry he’ll think he can his money on whatever and fall back on mine for bills, ect. My mom also borrowed 2.2k from me, so I’m just financially stressed. I have a hard time saying no or stating my opinion to avoid confrontation. In a ways, I feel like I owe it to pay the majority since he paid the rent last time we lived together. Idk. What would y’all do in this situation?