Web Screenshots for Surftrackr
Feb 11th 2008
As part of my ongoing quest to give Surftrackr more visual goodies, I developed a system to capture web screenshots. They should provide quick visual clues to what your users have been looking at on the web. There are a few imperfections, but basically it works.
You'd think (wouldn't you?) that there would be a simple-to-use web screen-capture program which could run on any platform, was open source and (critically) would not need a web browser or X session (under Unix) to be running in order to do its magic. There are various solutions like khtml2png, webkit2png, WebShot and many others, but nothing which provides the holy grail of web-screen-grab www.example.com example.png. WebShot comes closest, but it's closed source, Windows only, and the command-line version costs extra.
So, the solution I've used is based on this one and requires:
- An X-session, which your Unix box might have running already. If not, and it's one of those "lives-in-a-server-room-and-never-has-a-monitor-attached" kind of server (which mine is) you can use Xvfb or VNC Server to fake it. Note that if you set up your PC or server with X (eg KDE, Gnome, etc) already, you don't need either of these programs.
- The Opera web browser. It's fast and does the job, with command-line control options. Firefox can be used in the same way, but you have to disable the session manager and make various other tweaks, so I prefer Opera.
- The import and convert programs which are provided as part of ImageMagick.
Having installed VNC server, I set up a couple of aliases in my ~/.bashrc file:
alias startvnc='vncserver :1 -geometry 1024x768 -depth 16'
alias stopvnc='vncserver -kill :1'
So just typing startvnc will start up a VNC server for me, on screen 1 (that's what the :1 means). If you're following along, you might want to use a VNC viewer to access your screen, just to see what's happening. I don't have any fancy window manager (like KDE) installed, so my desktop looks a little sparse.
But sparse is also fast in this context, which is a good thing. Incidentally, now is a good time to start up Opera and make sure its settings are OK - it asks a few questions first time it's started up which might snarl up the automated process. When you have it to your liking, quit Opera by pressing Ctrl-Q.
The screenshots themselves are taken using two programs in the surftrackr/utils directory, screenshots.sh and get-screenshots.py which should be run from the "utils" directory. With a clean install of Surftrackr, you can just start the process running by typing
python get-screenshots.py
If you have a VNC viewer running, you should see the magic begin at this point - frankly I enjoy watching this bit. If you have a slow link, and the browser repeatedly fails to load the page completely before the screenshot is taken, edit screenshots.sh and change line 8 ("sleep 10") to "sleep 15" or "sleep 20".
Some pages will be inaccessible, since at the moment the program takes screen-shots of only the website rather than specific URLs, and if that site happens to be something like cgi.ebay.com you'll probably get an error page, at best, or a "Directory listing denied," at worst.
To view the thumbnails on appropriate listings, go to the General Settings category in the Surftrackr admin interface, and add a check-mark to "Use thumbnails":
Visiting the Users -> Sites page should show something like this:
How you run the get-screenshots.py script is up to you - once or twice per day via cron, or (as I prefer) by hand when required. Just make sure two instances of the script don't go to work at the same time, or you'll get a nasty mess in your X or VNC session. If you don't want screenshots, don't run the script at all and leave the "use thumbnails" option unchecked in the general setting section of the admin. You should find everything works just as before.
Simon Burns
11 Feb 2008