Work and ttc

I missed work to go to appointment with my obgyn to have a check up since I wasn't ovulating on my own. My boss is now claiming he doesn't have to pay me sick leave for my visit even though I have 36 hours left to use. I filled out the request and submitted a doctor note saying I was seen. I never told my boss why I went to the doctor. Has anyone else ran into this problem? If so what was your result? Thank you!