hopefully you're doing this after about 6 hours of sleep, otherwise you'll have some issues. if you are in sleep paralysis and you're seeing swirly colors then you're pretty darn close. keep watching the colors and try to recognize familiar things in the shapes they take, from there you can 'pull' yourself into a dream.

If you're not doing this after ~6 hours of sleep, i imagine you're not going to hit REM sleep very quickly, meaning you're less likely to fall right into a dream state.