Is is noisy where you're trying to sleep? Any sudden noise brings me out of it.
Personally, I think SP is pretty scary anyway, so maybe you're afraid of that
feeling? Do you start to panic, or get nervous when you're in SP?