Then what's probably happening is that you're concentrating too hard on your anchor(in this case, breathing). Do not actively count your breath, or even concentrate on it. Just have it at the back of your mind, always aknowledging it passively. Try to find your sweet spot of concentration.