Im almost never horny

Elle
So my boyfriend and I have been together for a year and a half. He's voiced some complaints about me not initiating sex. I don't know why but I almost never feel horny or anything. I've know it isn't him because I've always felt this way. Sometimes I am in the morning but we work different schedules so we can't really have sex many mornings. 
I just didn't know that it wasn't normal for me to not really have a sex drive. Right now I stopped taking my birth control to see if it was the problem since I've been on it since high school but it doesn't seem to have changed things. 
My boyfriend just constantly seems like he wants to touch me or something and I kinda get annoyed because I just want to lay with him without feeling like he wants to have sex every 5 seconds