I honestly give up now

Me and my partner have been TTC for a while now and honestly i just give up ! Its getting me down so much we have been together 4 1/2 years and nothing and then his brother has been with a girl a year they have a baby and got engaged today and it just aggravates me so much because they didnt even plan the baby it was just a drunken night and i feel awful for getting annoyed at it but all she does is go out drinking and wants free time yet she talks bad about all the mothers who do the same thing yet she does it. Am i wrong for getting annoyed at it all ?