Help!!

My baby was born Tuesday , he will get fussy and I will try to nurse, he won’t latch / latch for a second and unlatch, does this mean he’s not hungry or not wanting to latch?? He also can go like and hour to four hours between feedings. I’m worried he’s not getting enough.