Should I break up with him

Me and my boyfriend have been dating for three months we met at work and we still work together. It was so nice in the beginning. How it usually is but it already started going downhill. I love him, but I don't know if I should stay with him. The only time we see each other is when we're at Work and since we're keeping our relationship a secret we don't talk much at work. Whenever we don't work, I always try to make time for him, but he just doesn't really get the hint, even though sometimes I do tell him. Last Saturday we worked together, but he left early so I stayed behind and then he canceled the plans that we had together for Sunday because he was sick. Today he told me he went snowboarding even though he's still sick and I guess maybe today he feels better than yesterday, but it just feels weird with the timing. He tells me he loves me and he's such a sweet kind person, but I don't know what to make of his actions. I really do love him, but he's just adding stress as sleepless night thinking about him. Should I fix it? I don't even know how because I've already told him what bothers me before he hasn't fixed it or should I just end it?