What doctor to start with?

It's only been 6 months TTC but I'm getting worried there could be a problem because I'm 31 and I'm getting impatient. The thing I don't know is, what Doctor do you start with when you want to rule out any issues? I don't have a dedicated OB/GYN, I see my primary care for routine GYN stuff. I don't need a referral to see an OB/GYN, but did most women on here start at their primary cares office and take it from there? Or should I go directly to an actual OB? Thoughts?