The goal is to wake up during REM. You know you achieved that if you wake up and feel like you were just dreaming, and recall the dream exceptionally well. So basically, try to wake up during (preferably near the start of) one of the peaks on this chart...
http://www.end-your-sleep-deprivatio...-the-night.jpg
And the X-Axis is hours since you fell asleep, or at least close enough. You can see there is a monster REM period at around 6 hours, which is why you will often see the recommendation to set your alarm for 6 hours. The average person falls asleep pretty quickly so you can safely interpret that as "6 hours after I fall asleep". Even if that's not what they meant, you're probably only off by like ten minutes then. You just have to guesstimate how long it will take you to fall asleep and set the alarm for X hours after then. If you're off by twenty minutes it doesn't matter. If you're laying there in bed a lot longer than you expected though then consider pushing the alarm time back some.
Charts and stuff aside though, it sounds like you're overthinking it, this isn't an exact science by any means. You will experiment and find what works best for you. If you're not having luck with 6 hours, try 5.5 or 6.5 or something. I find I do best waking up after about 3.5 hours and staying up for 1 hour.