They call this losing lucidity. I seem to remember it being somewhat of an issue when I was new to being lucid, but as we speak it hasn't happened to me in ages. Although waking up is still an issue. It's possible that with experience you will find that this happens to you less and less. Or that could just be me.