The problem was Windows's memory management
the issue specifically is memory fragmentation

When you load a webpage, it typically loads lots of images as well
these images are compressed (Jpeg, Gif, Png, etc.)

In order to use a webpage properly, these images have to be uncompressed, and sit in memory for the duration of the pages life

You have to store them uncompressed, else when you scroll a page (or any action that changes a page) the images would flicker as they are redrawn from their compressed states

A webpage can potentially take up many megs in memory due to large quantities of images being stored as uncompressed bitmaps

So, you have an initial "high memory usage" issue to start with

As you are navigating around a website, some images drop off (because they're only on one page) while some stick around (as they are present on all pages)

the various smilies and user avatars & sig pics are present on some pages, but not all
whereas the top banner on this site, and various style images are present on all pages

So, when you first view a page on a website, the memory is filled up with stuff

a bit solid block of used memory

then you move to a different page, and some memory is freed up as images drop off, while some cannot be as they're still needed

moving around a website will split up the memory block as new images may be too big to fit in the freed-up holes or the OS memory manager just chooses not to reuse previously freed memory

20 pages into the site, the memory block looks like a swiss cheese
there's a whole heap of "free'd" memory, but it's in small clumps dotted all over the place

An OS will report an application is "using" memory if it's been previously allocated but not taken back

Windows was reporting that Firefox was using more and more memory, purely because the Windows memory manager was not smart enough to deal with all these small scattered blocks of free memory

The OS did not reclaim the memory, so Firefox is still "using" it (even though it's been free'd)

All browsers on windows (as far as I know) suffered from this,
but Firefox seemed to be most affected due to the way it implimented it's page caching