fertility troubles

I've been with my partner now for just under 3 years. We haven't really been trying but we've never been careful either and nothing's happened? Should I be concerned?