I think what ThreeCat said is pretty accurate and is a big component of it. As for the rest of it, here is my theory:

Lucidity in dreams, is simply a certain (high) level of awareness. It is a point on a continuum of awareness. Memory (and access to memory) are tied to awareness. When you intend to write down your dreams (prospective memory) and when you actually write them down, you are putting the focus of your conscious awareness on dream content. You actively increase the awareness of your dreams by putting your attention to it at the closest moment you can, which is right after waking up. Doing this on a regular basis increases dream recall the more you practice putting conscious attention into something that is usually an unconscious process for most (one that has a loose connection with memory for the most part), and in so increases your dream awareness.

This is why you remember more of a dream and more dreams the more you do this regularly. At some point dream awareness increases to a level that crosses the lucidity threshold. This happens both on a small scale, like writing down your dreams and going back to sleep and becoming lucid, as well as on the large scale when those who write down their dreams only in the morning, and over time start lucid dreaming (even without other methods, simply by writing down their dreams).