Feeling Guilty

Is it wrong of me to feel some what of relief that my S/o mom takes our daughter so we can have sex? I love having our time but I feel guilty because I miss her and just want her away for that time.