canceling genetic counseling...frowned apon?

Besides a miscarriage and a thyroid surgery I've always been relatively healthy. As well as my s/o. I'm 23 years old and my doctor automatically scheduled a genetic counseling appointment. I already had to reschedule once but with my work schedule-won't be able to make this one work either. Truthfully, I would rather go without. But I'm not trying to hear any negative feedback from my drs. I already got the test for Down syndrome and everything went well with that. I have a lot of anxiety about it because I really would rather not go for multiple reasons, but at the same time I don't want it to seem like I just don't care because I most certainly do care about my baby, I just don't feel the absolute need for the test at this time in my life. Has anyone else felt pressured to do this counseling, and denied it? I need to call my doctor today with a final decision lol.