Nighttime cries

My new baby has been getting fussy at night. He sleeps all day, besides when I feed him, and rarely cries, but come nighttime he won't sleep and is cranky. He acts like he wants to eat over and over. Lastnight he was literally on my breast for hours straight because he would cry and start acting hungry as soon as I finished burping him. I broke and gave him a paci for a little and it worked and he slept for a little bit before crying again. Tonight he's starting to be fussy, after I fed him on both breasts. When I pick him he doesn't stop crying but when his daddy hold him, he's fine. It seems like he's either gassy or just awake from sleeping so much and wants to be held, but not by me. Is this because he smells my breast milk? My mom suggested that. I was so exhausted lastnight and didn't know what to do. Any advice? I'm so afraid this be a nightly thing