Would this irritate you a little bit?

I ran into my son's biological dad yesterday. My son is 23 and his dad dipped out on me after he got me pregnant when I was 17 and he was 34. I later met my late wife when I was 19 and she legally adopted my son when he was 5. She died 6 years ago. His has met him and has been in and out of his life. They haven't talked since he was 18. His dad saw me at Walmart and waved me down. Asked how I was. I said I was good. I'm a grandma now because my son and his wife had a kid. He didn't really care about that but then I said our son graduated with his associates and is on his bachelor's now. And this man said "Really? I heard that but I didn't believe it." It bothers me that our son getting a degree is something shocking to him. He then started bragging about his genes and how now 7 of his kids have been to college and how great genes he has and then tried to get me to go home with him. I know I should have not even entertained a conversation with him and I'm not sure why I did because it just left me annoyed.