This is a matter of expectations. In these dreams, you are expecting that the experiences are not normal, but you are not expecting this to mean that you are dreaming.

I find that changing an expectation is mostly a matter of being motivated to change it. When I don't experience my main dream sign for a while (false awakenings), over time I will tend to once again develop a strong expectation that I am not dreaming when I wake up, resulting in me consistently no longer catching the dream sign I used to consistently catch. When this happens, I remind myself to be more open to the possibility I might be dreaming upon waking up, which usually quickly gets me back to consistently catching the dream sign. Of course, my situation is very different from yours in that going back to an expectation that was already previously developed is usually a lot easier than developing one initially, but I think the idea of self-motivating to change expectations in order to change them is the same for both situations. Clearly, motivation influences expectations.

Simply tell yourself that next time you experience something like this, you will expect to be dreaming. Imagine it happening next time and try to make it happen next time. Don't be discouraged if It doesn't happen right away, though. The expectation you are trying to develop is not necessarily a natural one at first, so it can take some time to get over its awkwardness that might be discouraging you from fully adopting it.

Keep in mind it is possible to do reality checks without changing your expectations. These reality checks are completely ineffective because they won't change the expectations that need to be changed. So, if doing reality checks, keep in mind your intention to change the expectations you are trying to change like I mentioned before. In other words, use reality checks to study for the test that is the dream. During the dream, the clue will be something unusual that will happen and the answer is that the dream is a dream. Your answer, of course, will be based on what you expect it to be at the time. So, you should expect your surroundings to be a dream when practicing a reality check. Don't do anything that might do harm upon expecting this, though.

Also, when you're imagining and planning, include some other dream goals in mind you would like to try once you attain lucidity. These will help keep you lucid in the dream once you are.