Well, you can ask yourself, why is it annoying? Because it is difficult? Why is it difficult? Maybe you have already asked yourself that question, over and over, until it made no sense or did not matter anymore.
Well the answer is as simple as it is cheesey: You made it difficult. Lucid dreaming is all in your head. You could be a total cripple with no muscles and no way of moving, yet with a normal brain you could master lucid dreaming. Inside your head is where all the dreams happen. If that same place knows that lucid dreaming is as hard as YOU know, then how do you expect to make it easy?

Lucid dreaming is alot like training, and then nothing like it at all. You can't train your mind endlessly to achieve a goal eventually. You can't just work mindlessly and basically not try, hoping to eventually stumble across your destination. However, you do need practice. You need to work at it, to try your hardest with it. You can't half-arse it all the way, you need focus, diligence, and patience.

Of course, some people have an easier time of it than others, much like how people are better at chess or running or writing books than others. However, since dreaming in universal, lucid dreaming is not a talent gifted to some, and unattainable to others.

Everyone can dream, everyone can lucid dream. What separates those you can, and those who do, is the belief that this knowledge brings. Everyone can, just not everyone knows it.

Silly speech over.