From Friends to FWB - need opinions!!

Ok. So crazy thing... a friend of mine just got divorced a while ago. (I was friends with him more than I was friends with her) they also no longer talk. 
He and I have hung out without flirting or anything once but we text (mostly) every day. We started talking about how he likes bjs and I said I liked when a guy went down on me. We toyed with the idea , and hadn't mentioned it since. Then we kind of brought it back onto the table and we are deciding to do it but very secretly since we also have a rather large number of friends in common (entertainment business) 
So we are wanting to do it because neither of us had any action for a while, and keep it as Fwb. But promise ourselves that we will keep our friendship how it is. 
I'm scared of being nervous etc. (It's been many years for me) 
Can I get some thoughts and opinions? 
I do have an IUD, and he will be using condoms.