baby not eating properly

My son is 3 months and one week. At his three months app he was good weight. But I feel that he hasn't been eating much since. I breastfeed him and he never stays more than 5 mins and only takes one breast. He doesn't feed more than 4-5 times during the day and 3 at night and non of them are proper feeds. Should I be concerned?