Scared about anal??

Angel
So in my first serious relationship I got pregnant really fast and we caught and he was abusive and I was really horny while prego and he said he wouldn't have sex with me just anal so we did it all the time and it made me cry and I hated every moment and he didn't care how I felt but now I met someone else and I'm married and have been for a year and a half and he wants to do anal and I told him I'm scared and don't know what to do how do I get over the fear that he isn't going to want it all the time my fear is that he is going to want that over my vagina... Am I just crazy?:/