TTC crazy

Decided we want another baby after our autistic 10 year old... got him on sheer chance.... now that we've started trying it's taking what seems like a long time and I feel like I am punishing my husband because we aren't having sex "at the right times" and we have only been trying for 6 weeks.... I feel like I am going crazy way too early but I am 32 and had Mirena after my first.... and I felt myself ovulate every 2 weeks when that was in. Am I alone in this feeling?