Headaches/migraines early pregnancy help!

Hi everyone! This is my first pregnancy and I’m pretty early around 6 weeks, I have been having on and off headaches since before I even knew I was pregnant but I’m wondering now if that was the reason for my headaches. I’ve been really trying to not take any medicine, dr told me I could take Tylenol if I needed to which I did have to take one day because it was so bad. These usually last for hours and days which is so frustrating they never go away. They told me I could try allergy medicine, my Ob has me on magnesium oxide supplement and who knows if that’s working...I’m just at a loss I’ve tried ice packs and resting. I’m taking the up & up brand prenatal gummy vitamins. My first trimester has been miserable so far not even because I’m nauseous just because of my headaches. Does anyone else have these? If so any advice on how to manage them I am miserable. Thank you!