Fast or slow, depends on your needs
I think it's important to understand that there's an old habit most of us have when we read--an internal voice. The philosophy behind speed reading states that most people read with an internal voice, which slows them down, and that the practice of speed reading includes "deprogramming" this voice and letting the visual cortext take in all the text. Speed reading books also encourage the reader to scan the text with a hand to guide the eye through sentences, and later chunks, of text. Moreover, many of the nonfiction books we read are filled with examples reinforcing what we already read; speed reading books encourage one to look for the meat of the text and to scan through the secondary stuff that merely backs up the main point.
That said, especially the last point of prioritizing what one focuses on during reading, Tommo made a good point in an earlier post. He wrote that while reading Zen and the Art of Motorcycle Maintenance he has to reflect on the philosophy of the text, and that he can't speed read it. I agree.
I'm going to read a book like "The Seven Ways to Success" (I'm just making that up but I think there might be a title like that out there) faster than Nietzsche's The Genealogy of Morals, simply because the former is going to have a clear, bullet-like format and the latter is a philosophical journey requiring analysis of translation and ideas that can't be summarized so easily.
We already speed read to some degree, and know when to turn it off or on. I think that the techniques of speed reading simply build upon what we already do.
However, getting rid of the internal voice might suit us all well. It's not true that slow reading equals stronger memory of the material. In fact, faster, visual reading corresponds with greater retention.--So states my speed reading book, which cites psychological studies to back this point. :)
--Tim