Re: MIME-type indices

Albert Reiner <[email protected]> writes:

> speaking of MIME types: One thing I would like to see in wwwoffle some
> time is an index of lasttime / prevtime sorted by MIME types, as well
> as by whether things are redirects.  (Sorting by file type is close in
> that it allows me to skip tons of useless graphics files, but it is
> not sufficient.)
> 
> The downside compared to other indices is that every file has to be
> touched, which may be slow (but would not be a problem in my case).

Sorting by MIME type is possible but would be slow, like you say.
WWWOFFLE would have to open up each file and parse the headers to work
out which one is the MIME-Type and then use it for sorting.  At the
moment none of the indexes open up the data files, only the URL files
but already some people think they are too slow.

One thing that I have done recently is that I have started using the
lastout index after going online so that I get a list of the pages
that I requested when offline.  This is much better than the lasttime
index which contains everything fetched when online.  I hide the
obvious jpeg/gif/png files by using the IndexOptions section list-any
option for the appropriate file extensions.  But even with this there
will be lots of other files like redirections and images with strange
file extensions.

The other thing that makes it even better is a script that opens up
all of the files requested last time offline in tabs in Firefox.  This
way I can browse all the pages I requested when offline without
needing to use any indexes or click on any links.

-------------------- lastout-in-tabs.sh --------------------
#!/bin/sh

# Load the lastout pages into Firefox

urls=`wwwoffle-ls lastout | awk '{print $6}' | sort`

firefox -remote "openurl(http://localhost:8080/index/lastout/?sort=alpha,new-window)"

# Tune this depending on the speed of your computer
sleep 30

for url in $urls; do

    case $url in

        # Images, stylesheets and scripts

        *.ico)
            ;;

        *.gif|*.GIF)
            ;;

        *.jpg|*.JPG|*.jpeg|*.JPEG)
            ;;

        *.png|*.PNG)
            ;;

        *.css)
            ;;

        *.js)
            ;;

        # Any other urls

        *)
            echo Opening: $url
            firefox -remote "openurl($url)"
            ;;

    esac

done
-------------------- lastout-in-tabs.sh --------------------

-- 
Andrew.
----------------------------------------------------------------------
Andrew M. Bishop                             amb-Uxr6IM1mbv2TY6FTCsQk+9Bc4/[email protected]
                                      http://www.gedanken.demon.co.uk/

WWWOFFLE users page:
        http://www.gedanken.demon.co.uk/wwwoffle/version-2.8/user.html