Good Question:

My theory on that is that our minds tend to react to things that are new ie. the new techniques we try. If we try an old technique rather than a new one our mind (atleast mine) tends to see it as an ordinary routine, which makes the mind unintentionally reject it as used and uneffective, making us non-lucid. However when we try a new technique we're instantly more likely to dream lucid, simply because the mind thinks it will work and proceeds to do so.

If we were 100% sure of having a lucid dream every night, our minds would have a lucid dream, every night. We've had so many non-lucids that we simply made a habit of it. Its like that saying too, "the rich get richer the poorer get poorer" same is true with LDs, the more of a habit it is, the more you have, and the more the non-lucid dream habit is, the more of those you have instead.

As Gothlark said, its all in your confidence and the feeling of being able to do it without effort that truely will bring you LDs every night based on how confident you are. But for many of this, its the chicken and the egg. You can't really have confidence without alot of LDs to back it up, and you can't really have alot of LDs without alot of confidence... So we usually end up somewhere in between... A few LDs every once and a while. You have to build that up, and after all success breeds success.

Another theory is that our conscious mind is untrained in simply staying conscious as we fall asleep... if we could simply do this with ease, we'd be sure to be lucid the whole night, everynight.

Hope that helps.

-Daniel