Should I be worried?

I am 24 years old and I am not actively trying to conceive but I am also not using any method to prevent it from happening, my boyfriend and I believe it will happen when it happens. Not to put TMI out there but we have sex at least 4 times a week and have been for a while now, I'm rather concerned that I'm not pregnant. Should I be concerned or is this something I should consult my doctor about? I'm reluctant to consult my doctor because we are not actively trying and if I find out there's something wrong with me it will make me stress but at the same time...