FryingMan's EILD (Halovision) experiments

[FryingMan]

Well I've finally gotten off my duff and taken action in my goal to learn about and create my own EILD systems. I've been thinking and dreaming (haha) of pursuing this for years, and I've finally taken the first major steps. First up is working with IAmCoder's Halovsion set-up. Massive thanks and shout-out to IAmCoder for hand-holding me extensively and patiently through the process of acquiring the equipment and setting it up to run with his LucidScribe system. This system works by using a night-vision camera (non-contact, off to the side of the bed) to detect tiny and large movements during sleep and to combine these detections with knowledge of how REM works to infer that the user is dreaming, and to send one of any number of possible signals (sound being easiest) to either wake for DEILD or to promote lucidity (DILD). I have the system up and running. It's not very good yet in detecting my eye movements in the dark, at least in bed. From what I've seen so far I seem to twitch my cheeks in REM more than move my eyes! RCM! Much more experimentation with placement, distance, sensitivity, etc. is required. I also end up playing with the system deep into the night so have not had any epic dreaming nights with REM cycles after 6 hours of sleep with it yet. In addition to the main REM notifications, I have plans for: WBTB notifications (helps you catch inter-sleep-cycle wakings, whether you're getting up for a full WBTB or just doing a micro-WBTB), sleep cycle count displays (which cycle of the night are you currently in), sleep clock (starts at 00:00 at bedtime), delay notifications (push a button, go to sleep, and a notification is signaled in some number of minutes/seconds, a button to toggle on/off all pending notifications (a problem with current EILD systems is having a way to easily and *quickly* turn the entire system on and off without having to get up and work at the computer or length settings, which can wake you up too much). I'm really excited to get going on all this! Here's my setup:

[FryingMan]

Moving the camera closer to my face and tweaking the Halovision LucidScribe plugin's threshold motion detection parameters made the eye movement detection much more accurate. Still more experiments to do. Also my own python-based RPi large and small ("eye") motion detection is coming along nicely. I will probably try a basic facial/eye recognition algorithm, or at least limit small movement detection to the general area of the eyes.

[fogelbise]

Very cool FM! This sounds like a fun project and I look forward to hearing more!

[IAmCoder]

Elite. Right back at you for the pointer to the video encoder motion vectors for the standalone version. I like the little guy holding your camera! This is what my setup looks like now: Lucid dreams.

[broth]

Wow guys this is awesome! I don't have much to add, except that I'd love to keep hearing updates as you continue working on it. Is any of the code up on github?

[FryingMan]

I've managed to go through my software/hardware career without diving into any numerical analysis. Looks like the need to analyze the motion vectors in more detail, and/or doing facial/eye recognition will take me deep into that territory. Time to start learning numpy to start with! Spent a long time tweaking my Raspberry Pi-based motion detection heuristics before bedtime. Managed to achieve a version where "false positives" in small movements (meaning non-eye movements) occur fairly infrequently compared to when eye motion is detected (very high frequency of incoming move signals in that case). Part of the problem of small movements is that the structure of my bed is such that I must place the camera on the bed with me, and my movements are thus probably magnifying the false readings. I need to build a little removable platform on the side of my headboard to hold the apparatus so it doesn't get jiggled by bed movements. I spent last night's data gathering session on the above python-based motion detector. I still need to write a program that will graph the raw data. Very surprising results: I made large movements far more frequently (like turning over) than I thought, it was a very restless night apparently! I need to add in capturing bursts of video at these times to confirm exactly what is happening.

[FryingMan]

Last night was back to halovision (still haven't gotten any graphs up from my own capture the night before, apparently a "simple plot" package is just not possible in python without installing large sets of libraries. I have bitten the bullet and downloaded anaconda [a masive python numerical analysis and visualization package], so hopefully I will get those graphs going). Positioning was off so I got no eye movement data. I did learn though that I am (at least now) apparently a very restless sleeper! Very frequent large movements all throughout the night. This is important data, too. To make the system very reliable I think I will need at least a basic facial/eye recognition component. It doesn't need to run frequently, only once after every sequence of large movements.

[FryingMan]

Last night I rotated the camera 90 degrees so that my face on the pillow would be upright in the captured images. I wanted to make sure this would work for trying traditional face/eye detection algorithms without the need for a rotate transform. Worked great. Again, I noticed in reviewing the video logs lots of frequent, short duration, large movement throughout the night. A lot of the night I spent facing away from the camera. Since my sleep schedule is still wacky as all getout I have no idea how well I'm sleeping. Bu I did manage to somehow end up facing the camera in the critical 5th hour region where I have a lot of my vivid dreams, and the system recorded clear eye-only movement in a few cases. There is still a lot of tweaking to be done but I'm pretty pleased with this. The system did actually send an audio notification in the following minute after the eye movement but I didn't hear it (it was the shorter of the two sound files I'm using). I plan to experiment with other notifications like vibration and light eventually, but first stage is reliable eye movement detection. Here's the key dreaming minute from LucidScribe (y'all don't want to see my sleepy face, so just the captured eye movement is included) : Spoiler for halovision graph: The blue spikes on the graph are detected movement, and the yellow staircase is the vREM detection algorithm showing the accumulated count of small movements that will lead up to a notification. A lot of tweeking for my personal sleep and dream style will be required, no doubt. Need to gather a lot more data. For now I've ordered a second camera (gotta love AliExpress, super cheap, the 2nd camera system cost 1/3 of what I paid for it local in a store) in order to have on hand for spare parts in case the first one fails, or to perhaps try setting up a 2nd system mounted above my face (for when sleeping on my back) or or on the left side. I'm 99% a side sleeper so the 2nd side is probably the better bet. Can't find a local or online (that ships to my address) flexible camera mount, and my current setup using the "helping hands" is suboptimal since the alligator clips are really strong, so I'm going to McGyver my own out of a bundle of wires tied together with tape. Update: found that using the combo of a spring-loaded women's hair claw holds the camera module nicely, then I can use a strong alligator clip on the handle of the hair claw, and voila, instance DIY camera mount. I still want to make a flexible, wire-based version so that the helping-hands device is no longer needed, and I can attach the camera to the side of the headboard structure so that bed movements do not introduce false movement detection on the camera.

[FryingMan]

Spent today researching and doing initial experiments with facial recognition systems. Tried OpenCV Haar cascade matching in python on the RPi, and it was OK but not stellar. I needed to train my own patterns because sleeping faces are just too different for the standard trained face detectors. So I switched to Dlib on linux using an example pattern training and feeding it stills of my sleeping face out of my LucidScribe logs, and the detector trained from those images works as advertised: for images close enough to the training images, they all detect accurately. Trying to put on-my-back-sleeping images together with the side-sleeping ones doesn't seem to work well, I think because they're too different. Instead, I'll try creating different detectors for different positions. The detector itself runs quite quickly. With good face detection, the system can get a lot more accurate: ignoring all small movements when no face is visible (avoiding false signaling), and when a face is present, ignoring (all?) movements outside the face region, and being much more sensitive to small movements within the face region.

[IAmCoder]

Pretty epic! Glad to see that you caught a dream already. If you don't come right with the camera mount, I'd be happy to send you one... I have some spare from when I was using them as mini selfie-sticks for the old headband version. Keep us posted on your findings!

[IAmCoder]

Still so stoked that it caught a dream already! If you ever summon the courage, you can right-click on the video recording replay in the minute view in Lucid Scribe and select "Save as GIF"... Got most of the Lucid Scribe code snippets on GitHub scattered around https://github.com/lucidcode... haven't committed anything in too long.

[FryingMan]

Camera installation complete! It's now mounted on the side headboard (due to the layout of my bed I can't put it on a bedside table, the angle and distance to my face would be wrong). This was an inspired improvisation! I built a camera mount from an alligator clip, 6 strands of sturdy 1.25mm enameled wire, cable ties, and a USB-A plug from an ancient web-cam. The RPi is mounted via a velcro-like strip (meant for hanging pictures, it has very high vertical hold strength), and the camera mount plugs in to one of the RPi's USB slot. A cable tie goes around one of the IR LEDs, and the alligator clip holds on to the locking housing of the cable tie, piece so the grip is firm and the camera does not jiggle and is oriented sideways so my face comes through upright in the video. The alligator clip is held to the copper wire bundle with cable ties, and the copper wire is bound to the USB adapter with cable ties. Cable ties also hold the copper wire bundle all together. Cable ties are AWESOME! Everything is oriented so that gravity helps. Really good test results with this. I can put my face in position and trigger a LucidScribe vREM notification with just a few closed eye movements. I think removing the camera from the surface of the bed is a huge improvement -- this eliminates jiggling of the camera (which produces false movements) due to my movements in bed. Spoiler for pics!:

[FryingMan]

Still so stoked that it caught a dream already! If you ever summon the courage, you can right-click on the video recording replay in the minute view in Lucid Scribe and select "Save as GIF"... Got most of the Lucid Scribe code snippets on GitHub scattered around https://github.com/lucidcode... haven't committed anything in too long. Yes that's great! Still waiting for my first lucid from it, though, or even a waking from a dream. Hopefully with the mount, things will improve. The dlib object (face) detection is a touch slow on the RPi (5-10 seconds), but within range of being usable. It may not be usable when I add in multiple detectors for different facial orientations. I may keep the laptop as a "face detection compute server"

[DoubleHelix]

Wow...what a huge undertaking. Congrats on the progress.

[FryingMan]

Well some small adjustments were necessary. The alligator clip holding the camera slipped off the mount cable somewhere after about 3 hours of sleep. Oops. In the morning I put it back on and wound a rubber band around the cable/clip junction to add more friction against slipping. Seems to hold tight now. I perhaps wacked the thing in the night when I turned over. I thought about adding a protective case around the camera. Cardboard will probably do for now but something "pretty" means ... another DIY "maker" project! I experimented for hours with tuning the RPi video motion-vector based eye movement detector last night with the newly mounted camera, I couldn't stop since I was making really great progress and went to bed super late (cough 4am cough) ruining the dreaming for the night. It's really exciting and I think it's going to make a really excellent REM detector! After the face detection and eye movement detection solidifies, and tweaking the REM detection heuristics, will come the fun part about creating signaling devices: lights, vibration alerts, sound of course, etc. For now it's all about gathering data, data, data, about large and small sleep movements, trying to discriminate between REM from non-REM facial twitches, and so on.

[FryingMan]

I've been taking a bit of a break from nightly monitoring for the last week. I've been investigating small microcontroller programming (ATTiny85, Arduino Nano) and acquiring and learning how to use their various programming environments (Arduino studio, Atmel AVR studio). The bedside Raspberry Pi IR camera approach has its advantages: nothing to wear, can distinguish between large movements and small, and if you put in the effort, I believe accurate face detection and isolation of small movements to the eye/face area can be achieved. Training the face detection to deal with face orientation takes gathering images from the night and feeding them in to the algorithm. But object/pattern detectors don't work it the orientation between objects is too wide-ranging, so you have to factor out different positions into different detectors, and call them all in a row, which takes more time (but should be more accurate). The nice thing about face detection is it only really needs to be done no more than probably a dozen times per night, after each major movement period which are likely fall between sleep cycles (which is also a great time for a WBTB alarm!). So it's OK if it runs or a minute or two. The disadvantages are: I'm always concerned about hitting the camera when I turn over, without face detection there are too many false notifications, and accuracy of face detection depends on face orientation: training the detector takes time and effort. That's OK for me, though, I have lots of time. As a turnkey product, probably less useful. But I'm not really thinking in that direction at this time. After working with the bedside camera for a bit I see the attraction of the sleep mask approach (which has its own issues, like: falling off, moving out of place, and strongly prefers a 100% wireless and very low power solution making the apparatus more complex/expensive). With a regular sleep schedule, this setup should allow a fairly detailed mapping of the typical night and sleep cycle behavior, allowing feedback of time into the REM detection algorithm. Ideally also I will produce a setup with buttons and/or voice activated commands that will make it really easy to indicate "I just woke up from a dream" on the night's record to additionally calibrate what dreaming looks like for me.

[FryingMan]

I'm still working on stuff. I've decided to name my system "The Night's Watch" . Working on (re)learning basic electronics and familiarizing myself with contemporary IoT devices. The ESP8266 modules are really astonishing: A fast, *wifi-included* MCU for under $4! I wrote a TCP client/server echo test from my mac to my windows laptop that ran through the wireless access point of this tiny board. Really astonishing. I may forego bluetooth in favor of this. For wearables BLE may still be king but this thing is really exciting.

[IAmCoder]

That is an epic name. And it is is awesome what you have done with all the projects. I think you just need to inch forward a little bit. From the screenshot I can vaguely tell that your eye is pretty far in the video feed: Spoiler for halovision graph: I get it to repeatedly trigger when I am this close: Spoiler for halovision graph: Can you give it a shot like that with the initial settings? It helps if the algorithm has lots of good contrast to work from. And lay still as a feather, that - in and of itself, helps focus the mind.

[fogelbise]

I've decided to name my system "The Night's Watch" . Love it Mr. Snow! You know, the level of talent on Dreamviews never ceases to amaze me and this thread is just one more reminder of that.

[FryingMan]

That is an epic name. And it is is awesome what you have done with all the projects. I think you just need to inch forward a little bit. From the screenshot I can vaguely tell that your eye is pretty far in the video feed: I get it to repeatedly trigger when I am this close: Spoiler for halovision graph: Can you give it a shot like that with the initial settings? It helps if the algorithm has lots of good contrast to work from. And lay still as a feather, that - in and of itself, helps focus the mind. Since I have the camera actually extending out into the sleeping area due to the positioning of my bed, I have this subconscious fear of whacking the camera when I turn over, which is why I don't want to move closer. So unless I can find a zoom lens for this thing, I'm probably not going to get much closer. Also, I would like the system to be able to work on a typical night, which means handling lots of re-positionings. High accuracy requires face detection I believe, at the very least to know when not to signal, and to limit all small movement analysis to the bounding box of the face area. Also, due to not being close enough, or just to the pattern of my blankets, I get too many false small movements when I'm facing away from the camera which trigger false notifications. I also need to customize the REM detection algorithm to my own sleep patterns and facial twitch behavior and desired data gathering system, so going forwards I'll probably stick to using TNW. Thanks for the advice, though!

[DoubleHelix]

Following this thread inspired me to get back to experimenting with my REM Dreamer mask. Will try it tonight.

[FryingMan]

Following this thread inspired me to get back to experimenting with my REM Dreamer mask. Will try it tonight. I got interested in looking again at eye movement detection masks and how to make them. They're pretty simple conceptually (the hardest part seems to be the physical construction of the mask and the placement of the circuit board into it). I looked into the REM Dreamer and from pictures of the circuit board I've determined that it only detects movement in the right eye! I guess the assumption is that both eyes move together, that's a pretty safe assumption I imagine, but if you're a side sleeper it might make more of a difference with the REM dreamer which side you sleep on. I can't guess which side would be better: on the right it would tend to stay in place, but might also be squished too close to the eye. Try it and see if you can discover which side works best for you!

[FryingMan]

Looking at this Novadreamer clone, it appears to also only have a movement detector for one eye. Kvasar It makes sense, since it makes the circuit simpler, smaller, and uses less power.

[Highlander]

Try using a mouse?

[FryingMan]

Lots of issues with the mouse hack approach, including the cable. I'm also in this for learning how to contruct DIY electronics, so I'd rather build my own system from individual components. I still think the IR camera is the best way, but for maximum REM detection needs at least 2 cameras. I've already ordered my Rasberry Pi Zero W with a camera cable, I've already received by backup IR Pi IR camera, so I'm all set for a stereo version of The Night's Watch! Gotta get back to the core system, next up is connecting to the external face detector, and saving video and photo snapshots of movement (large and small) throughout the night, and continuing to train the face detectors. I'm also building myself a beefy desktop compute system that I plan to use as detailed facial analysis server should the Pi prove too slow (which I think it will), which will be used for running through many facial orientation detectors, and then pinpointing the location of the eyes for targeted small movement detection. It's quite computationally intensive, and I want the results within about a second, so I will also eventually be adding a GPU that works with dlib and other deep-learning AI libraries to accelerate calculation.