I learned an interesting point about expectation from Robert Waggoners book, "Lucid Dreaming - gateway to the inner self" just recently.
He explained how he had not heard about the fact that spinning was supposed to change the dream scene and re-stabilise it, and when he tried it, he had a totally different experience with lights instead.
He also explained how someone had suggested that you could wave away the dream scene by waving your hand in front of your face, but when he tried it the dream went all dark, because he hadn't been told to expect a new dream scene in its place.
His point was that expectation is what drives everything in lucid dreaming.
If you expect something, it often happens.
That implies that we need to expel all negative thoughts about things not working and believe that it will work to progress in lucid dreaming.
For instance, if you think for a fraction of a second that you are getting too excited, and that could de-stabilise the dream, then it likely will happen.

It sounds obvious, but maybe the expectation thing is the real key to LD.
It's been said before, but I for one possibly have a hard time really believing at the unconscious level?