Well.. In my opinion, that means your dream is unstable.. Try reminding yourself to stabilize your dream as soon as you get lucid.. By rubbing your hands, or focusing onto details, or even touching stuff around you and feeling with all your senses..
It depends form person to person, but stabilizing the dream will help for sure, and with it you can get the most out of your lucid dreams !

Good luck!