I think we moved too fast... idk what to do

I think we moved too fast, maybe that’s the problem. He told me he loved me first during a drunk night, and still tells me weeks later he loves me. I said it back. But some days I feel like the relationship is okay, and others I don’t. I just develop an attitude that I cannot shake. He still is loving and caring but I feel as though when I get in my moods he’s being sneaky. He has a baby mama, and they just ended and then he got with me but somehow I just do not trust it. Even though he is with me everyday and if not he’s at work and we work at the same place.