My bloat went way down around 12-13 weeks and I thought it was a bump. Then it went down again at like 15-16 weeks and now at almost 20 weeks I'm getting a legit bump

I was super bloated at the beginning too but once I got further into my first trimester my bloat went down a bit. Some days I'd have a bump and others I didn't. Your uterus doesn't rise above your pubic bone until about 12 weeks so depending on how far along you are it might still be bloat. But my opinion about bloating, it's because of the baby so it's a baby bump to me!! 

the size will just get bigger but you will have some days during pregnancy that it wouldnt just feel like "bloating" anymore but actual "baby". that happened for me when i started feeling popcorn like movements at 16weeks. regulated bowels also help with that feeling of relieving the bloating.