is it wrong and should i feel ad

So when i was younger like (15) i met this boy and we had young love the way we looked at each and how happy we were to see each other i wasnt really allowed out but i make ways to see each other and that kinda ended our relationship tbh but for 5 year we werent on and off he was young so he would wonder off but suddenly we went our seperate ways made our own lives now were 21 and we met up and started talking again but he had a girlfriend and i had a boyfriend we instantly had that bond again now we were inlove at one point and i had everything he wanted but ijust couldnt be with him physically now the ex is starting problems saying he cheated was i was rong to just jump in ? They havent been together so long and i was always in his life  please give me advice me and him really do love each other deeply