There is a time to fight back, and a time not to. As much as your instincts may tell you to fight back, in a situation like this, its best to just defend yourself as best you can. If your really itching to fight and wanna throw some punches, wait till after the fact, and catch them one on one where they don't have anyone to fall back on.
Thats one thing I've always noticed with people. MOST of the time, anyone who only challenges you to a fight, or tries to fight you while in a group, will refute a fight if they are alone with no 'back up', which means they're scared, and if they're scared, they have a reason to, and all you need to do is play on that reason.