Mixed feelings about possible pregnancy

So my husband and I have only recently been able to try having sex again. My endometriosis has been making it impossible until a couple weeks ago. One of the days we were able to successfully do it, was during my fertile window. I didn't realize I was in my fertile window until after. We used the withdrawal method (which has worked really well for us in the past, my husband knows his body really well, but I'm aware it's not great at preventing pregnancy), and that's when we stopped, after he pulled out. We snuggled for a bit after he had cleaned up and then went about our day.

Anyway, there's a slight chance I'm pregnant, and I have so many mixed feelings about taking a test. I've been having a lot of health issues recently that my doctor's don't know what's going on. It's causing me to collapse and have a hard time breathing, being unable to walk, etc. So needless to say it wouldn't be great timing. We're not ttc by any means, but if it did happen we'd still be happy. It wouldn't be the end of the world, but we'd be concerned because of my health issues. I feel like we're more on the avoiding side of things but we're not trying to avoid it at all costs if that makes sense. Though maybe we should be.

A selfish part of me hopes I'm pregnant because I miss it, and my LO is gonna be 3 in December. I loved being pregnant. But my husband has said he doesn't want another one yet. And I wouldn't want to put the baby at risk because of my health. But the thought of not being pregnant kinda hurts. Some days I get really sad, other days I know I'll be disappointed if I'm not, which is probably the reality. I know it's better for me not to be pregnant right now, but a part of me wants to be. Which I feel really selfish for, but I'm just trying to be honest with myself.

Has anyone been in a similar situation?