Need Conceiving expertise!!

I'm 22, me & my husband want to have a baby, we've been having unprotected sex for about a year & a half... The thing is we only really have sex like 4 to 5 times a month. I didn't realize until I looked at my last logs, & for three months out of that year we only have sex during my fertile week for three months & maybe twice for each fertile week in that month. I tried to get worried for the fact that its been a little over a year & I haven't gotten pregnant. It is really that hard or should I be worried. Or is it normal die to our lack of "trying" to have sex everyone her day, i would love advice, I live in a new town & my husband works alot, I don't really have a whole lot of people to confide in! I appreciate it! Thanks :}