My Husband Is losing His Memory!

For the past several months my husband has been acting super weird. I'm 28 he's 36. We are currently going through <a href="">IVF</a>. It started small. Not remembering little things like where he put his keys or to buy something, but it gradually gotten worse. He seems to forget stuff then remember it the next day. Oh he's remembering a different part of his life. For example. He used to work in construction in his early 20s. He's realtor now. A few weeks ago he was asking me where he his hard hat was. I seriously thought he was drunk. I told him you haven't worked in construction in years. He got mad and started yelling at me. And tried to walk out the house. I tried to stop him and he pushed me. I called his mom and she had to help me with him. Other days he will be fine. There have been times where he's asked me who I am and literally doesn't remember me. Today his memory seems okay but I'm so worried. I want to take him to the hospital but he's insisting it's the stress from work and <a href="">IVF</a>. IDK what to do.