I'm the other girl..

So there is this guy that I've had some.. interaction with. I've liked him for a very long time and we're just now starting to become more than friends.

Only problem is, he has a girlfriend. I feel absolutely guilty about it, because she's honestly very sweet. They haven't been together long but that doesn't make a difference.

I don't want to stop talking to him or stop anything, but I do (if that makes any sense.) I guess I just need some advice on what I should do in this whole situation.