Feel like giving up

We been TTC for 3 years with no luck. I had a HSG test and the Dr said everything is good no blockages. The boyfriend get a sprem count do, their all good but alittle slow. Now I'm going to start Clomid 50 mg next week for 3 cycles, and I feel like if this doesn't work that I really don't want to go though with this anymore or feel like giving up on having a family. Does anyone else feel like this????