Another, and probably final, update to Surftrackr has just been uploaded. It includes an AJAX-based facility to show web accesses as they occur, which relies on the previously-announce live-logging feature. This is still experimental but seems to work well in tests.
I've just uploaded the latest version of Surftrackr, with a few amendments and additions based on user feedback. The packages are available from the download site; read on for more details of the changes.
It's about time I put up some information about how to use Surftrackr, including its more advanced features and the Django admin interface. Here are some screenshots to give you an overview of how to use Surftrackr - stay tuned for details of using the admin interface.
A new version has been uploaded and is now available from the Surftrackr download site. This release includes several improvements to data acquisition and user management - read on for more details.
Grant (Surftrackr user) emailed to ask how to take data directly from the logfile, without the need to upload it or use the live-logging feature. Funny how the most obvious ideas are often the most elusive: I've made the change and the updated package is available from the download site.
A couple of bold users who tried installing Surftrackr under Windows reported problems running the scripts. I've uploaded an amended PDF describing how to fix the problem. Thanks to Diego for persisting through adversity and letting me know how to fix the problem!
Django has its own built-in authentication mechanism, which is documented on the Django website. Personally, I prefer to use Apache's authentication, simply because it means you can authenticate your Django users against any Apache auth mechanism.
If you've used the Django framework for your web scripts, you'll appreciate how powerful, flexible and easy-to-use it is. For me, it encapsulates the Unix ethos of simplicity combined with power, and I've been inclined to use it for every web project possible — including this website.
For ease-of-use, I've been experimenting with use of VMWare to create a complete, fully-working Surftrackr installation (with free Linux distro "thrown in").
The old-style blog format for Surftrackr.net had passed its sell-by date, and was starting to prove inadequate to the job. I spent today redesigning the site, and hope you like the results.
I've updated the documentation for the live-logging process.
Here's a fixed-up version of the views.py for the graphs app in Django. Just copy it to surftrackr/graphs/views.py. Thanks to Dave for the info.
It's quite feasible to run Surftrackr under Windows, without any modifications to the code. I've just uploaded a PDF with a walk-through of the necessary steps.
There was a small bug related to mime-types which will almost certainly affect you. New packages available from the download site, or just grab the two affected files and copy into your Surftrackr installation.
As promised, here's the latest Surftrackr with multi-user capabilities. During the testing process, I discovered a few little bugs which have been fixed. I've also added a feature to allow workstations to be mapped to user-names, for installations where your proxy is not using authentication.
I've spent the last few days converting Surftrackr to a multi-user application. This decision was based on user feedback, although it had occurred to me it might be a good idea. But I didn't want to get into it right from the start of the project, since I'd rather make Surftrackr work flawlessly for one user before adapting it for multiple users. Down the other road lies madness.
Don't panic! I'm not after money - although I reserve the option ;) and some people have even been kind enough to offer. But if you'd like to help, here are a few simple ideas which won't cost you anything and will help me promote Surftrackr as a contender in its field.
There was a bug in the JavaScript file (surftrackr.js), in the routines which flag users and URLs. It worked fine on Firefox 2 on Mac OS, but not Firefox 1.5 under Linux.
I've uploaded another PDF showing how to upload a logfile and search through it.
A description of how to capture screen-shots of the web pages in your Surftrackr database. It's not an elegant system, but it works and adds a little more information to otherwise featureless lists.
I've uploaded another walk-through showing how to set up real-time data logging for your Surftrackr installation.
Another visual guide (PDF format) which shows how to upload a squid logfile into Surftrackr.
I've uploaded a PDF file which shows, in words and pictures, how to install Django and Surftrackr on a newly-installed Centos Linux box. This is complementary to the installation docs, which you should also consult if you're about to install.
Just a small enhancement utilising MIME-types to show lists of images per user.
A bug has been found — thanks to Brandon for the info. Chances are, you weren't bitten by this if you've used Surftrackr according to the maker's recommendations. The bug caused duplicate entries to be created in the database where the MIME-type was unknown. But since uploaded logfiles are (a) locked when processing starts, and (b) moved somewhere else when processing is finished, you should find you were unaffected. But please upgrade now - read on for more details on how to avoid a full upgrade.
Another small enhancement allows designated users to be excluded from the reports, or omitted from the database itself. How you use these features is, of course, between you and your conscience ...
I spent a couple of hours this morning refining the search function to make its behaviour a little more intelligent when dealing with lists of words, and to allow exclusion of terms using a - sign in front of the term, a bit like Google does.
I added a search feature, in addition to the lexicons and hilites features. It's ridiculously easy with Django, and allows the Surftrackr user to put in words to search for in URLs (including the website name and the path to the resource).
Another day, another incremental Surftrackr improvement. One of the items on my to-do list is to enable live take-on of logfile data, rather than requiring a log upload. Although I haven't subjected the program to rigorous testing yet, I'm pleased to announce surftrackr-livelog.py just made it into the utils directory.
As part of my ongoing quest to give Surftrackr more visual goodies, I developed a system to capture web screenshots. They should provide quick visual clues to what your users have been looking at on the web. There are a few imperfections, but basically it works.
A few people have emailed me saying they get an error when trying to install Surftrackr. If your installation complains about an "unexpected keyword argument", you need to upgrade to the latest Django from their subversion repository.
I made my Surftrackr to-do list into a Django app. It took about an hour including HTML coding, has a 60-line models.py and <20 line views.py. You can view the items on the list, and if you want anything added to it, feel free to email me.
Matt from off the internets contacted me about a simple little bug in Surftrackr, which I would have caught if I'd just thought about it a bit more carefully. This is roughly equivalent to something a co-worker said to me once upon a time: "I pressed the wrong combination of keys." I did both. I failed to think, and I pressed the wrong combination of keys.
At first glance, it would appear complicated to install a Surftrackr app, but this is very far from the truth. Squidalyser, Surftrackr's predecessor, was written in Perl and required the installation of several modules, including DBD/DBI for database access, and the often difficult-to-install GD modules to produce the graphics. Most of the Squidalyser support emails I received were about installing, not using, Squidalyser. Compared to this, Surftrackr's installation requirements are, in fact, fairly modest, and shouldn't prove too difficult for most systems administrators. As a bonus, setting up Django on your system will allow you to use many other Django apps, and (if you're a developer) start using Python and Django to develop your own projects.
I've uploaded a slightly revised version of Surftrackr. As ever, you can get the latest (and previous) tar-balls of the software from the download site.
This is a list of improvements I hope to make to Surftrackr over the coming weeks. If you have anything you would like added, you can either contact me or send me any code you might have altered or added!
The admin interface is disabled in the demo, since it's a bit awkward to make it read-only to web visitors. It's the standard Django interface (with some HTML and CSS tweakage) but, in case you're not familiar with Django, here are some screenshots for your viewing pleasure.
In 1989, I almost invented the web. I like to think that, with a bit more time and more resources, I might have beaten Tim Berners-Lee to the prize by a good few years. Several years later, I nearly invented YouTube, and (with just a bit more luck) I might have invented web frameworks as well. But I'm not bitter: I had my moment, and it's good to see the kids doing well from my ideas.