Working out safe?

I have never been a person who works out a lot. Maybe I’ll feel guilty and go running once a month. However, about 2 months ago I joined a gym and I was going about once a week and started to enjoy it. Now I found out I’m pregnant for the second time. My first pregnancy ended in miscarriage. So now I’m paranoid about doing anything but walking. I’m about 4.5 weeks pregnant. Is it safe to go swimming or do the elliptical? Should I just wait until second trimester?