HEEEEEELP!!!!! Switching breastfeeding to formula at 20 days old?

I'm a first time mom and I'm getting so frustrated with my baby. He is currently 20 days old and he has been very fussy and crying all day long for the past week; it seems like he just want to be sucking to calm down. My husband believes that perhaps he is not getting enough milk or that my milk is not good for him and it's making his stomach hurt. His poop is normal (bright yellow) and clear urine,. we get a pretty decent amount of dirty diapers throughout the day. Also, he farts a lot while I'm breastfeeding him but nothing seems different other than his fussiness for the past week. 
We have tried giving him gripe water to release gas, but he still cries afterwards, which makes me believe that's not the problem. Is this normal? He has no symptoms of something else other than jaundice (Doctors told me his levels are normal according to his age).
I'm not 100% sure about giving him formula now, but I have done everything to calm him down and nothing seems to work other than having him on my breast 24/7.
Has anyone ever gone through this?
Sometimes he would suck for almost an hour, just to cry and suck again an hr or 30 minutes after! It just breaks my heart seeing him like this. 
Please, someone guide me on this!!!!