Self awareness is only half awareness, external awareness is just as vital (imho) though training both of these never quite seems enough to overcome that insidious semi-lucidity haha.

For myself, taking a break is rarely as successful as intent training, but I think it's a fantastic solution for certain lucid issues. When I do get stuck in that haze of semi-lucidity (night after night) what's worked for me in the past is taking on new and specific goals, something set to me by a friend or that I find online. I think the new angle shakes me out of that funk.

You're right though, banging your head against an obstacle for so long is only gonna result in a headache