Volume of sperm - can there be too little?

Ruth • 38, ttc #2
Hi girls
I am 33 and ttc no.1 for 2years. I am on my second cycle of letrozole now.
When me and my husband BD during fertile window (the only time we have sex so much!) we notice that by the final times in green window (so time 5-7 maybe) the volume of sperm is definitely a lot less. 
Can there be too little? In tests his sample is always fine but I just wondered. I know it only takes one swimmer obviously but does anyone else's quantity get less and less? 
Have tried every day and every other day but always the same by time it's a few BD's in 
Thanks, as always xx