Ive heard people say that doing things like smoking cannabis and drugs can make it easier to LD and when you dont have it it makes it hard to progress. I think the main reason for that is stress like you thought. I know that it affects my dreams when i dont have it. even if the thought of not having is in my mind a little bit, deep down it affects my subconscious which is enough to make me lose control of my dreams.