In 1989, I almost invented the web. I like to think that, with a bit more time and more resources, I might have beaten Tim Berners-Lee to the prize by a good few years. Several years later, I nearly invented YouTube, and (with just a bit more luck) I might have invented web frameworks as well. But I'm not bitter: I had my moment, and it's good to see the kids doing well from my ideas.
The web thing nearly emerged from a conversation I had with a colleague about creating Microsoft Help files. We were in the education business at the time, for a certain well-known North American systems vendor, and multimedia was the Big Thing. There was this elusive goal of the Full Multimedia PC, and we thought MS Help files might be a way to facilitate the "media" part of that dream. But the resource kit required to create them was fairly expensive, the whole thing was proprietary, and there was no portability between operating systems.
But just imagine, we mused, how great it would be if you could just hack out a text file (a bit like SGML, say) and create hyperlinks to resources, and distribute those over the network. No need to compile, no need to be locked-in to another vendor's solution. Wouldn't that be cool? Yes it would, we agreed, and went off to get a cup of coffee, confident that our neat-o idea was impractical for reasons we couldn't quite put our collective finger on.
A few years later, I was working in a school and talking to my boss about creating a kind of multimedia diary of school life. By that time, we had the resources to create movie files, and we had been doing so for most of the term. Problem was, they were specific to one computer platform, the quality wasn't brilliant, and there was still that awkward issue of distribution. The best we could think of was copying a stack of CDs and mailing them out to people, but obviously this was a non-starter. More coffee, more certainty that it just couldn't be done.
The web framework thing nearly happened when I found myself working for the first time as a web programmer. It was all Perl and SQL back then, and it became obvious that I spent most of my time creating forms which were very similar to each other, then validating the input using very similar routines, and sending messages back to the users to congratulate or scold them. I started hacking up a system to pull form information from a text file, and format the page according to a few simple rules. The file would specify a Perl routine to validate each field, and the return text to use if the user got anything wrong.
I'm not saying any of this to point out how clever I am, but I do think that having a half-baked idea is, in itself, not difficult, and identifying a need (which is what I was really doing) even less so. What really talented people will do is take their ideas, iron out the wrinkles, and make them into something which other people get excited about as well. Some technologies emerge when people are ready for them, like telephones and televisions, both of which have numerous claims for their original invention. Probably, they have more than one inventor, because a bunch of people suddenly found that the technology of the time could support this dramatic new idea they'd had.
At the moment, in this small isolated spot of Internet time, web frameworks are an idea whose time has come. A few years ago, I was seriously considering giving up on web programming and finding something less frustrating instead. Even carefully-crafted code had a habit of becoming complex, messy and difficult to maintain, especially if more than one programmer was involved. Web design itself was an unhappy process, often leading to dozens of very similar files, all of which would need to be altered on the latest whim of the client or boss. Animated email icons and multi-coloured page dividers were "in", page elements were laid out in tables, and buttons were GIF files which also took ages to alter according to today's requirements. CSS was a fancy new idea all the smart kids were getting excited about, but it wasn't obvious how it could help me on a daily basis.
Which brings me finally to the Squidalyser, an idea I actually developed further than just the initial discussion, but which eventually fell apart under the onslaught of hubris, out-dated coding practices and (to some extent) its own success. But because other, more talented people have taken a lead in developing some exciting new tools and techniques, I hope to make a fresh start with it.
The project was initially born from the need to monitor web access within a school. The situation I wanted to arrive at would not rely to any great extent on blocking web access. I had a few filters in place, but overall I felt this was a flawed solution since there were always too many sites to block, and the few commercial systems which existed at the time were often less effective than my own home-brew system. The alternative was to monitor, as unobtrusively as possible, web accesses and follow up on anything which was contrary to our Acceptable Use Policy. It was hard work, but eventually people started to get the message: break the rules and you will be found out.
Initially, I used Unix's "grep" command to sift the logfiles for various words and phrases. This had the advantage that any accesses not flagged up as suspicious would be totally ignored, effectively allowing most users their privacy while letting me know who was breaking the rules. Later, I built a web-based front-end which did basically the same thing, and finally started developing Squidalyser to make it all easy and fancy, albeit in a very web-one-point-oh kind of way.
I developed it for about a year, but you can see from its website that it is, to all intents and purposes, an orphaned project. I haven't written a single line of code for it in over four years. Whenever I do crack open the tar-file and start looking at the code, I find it awkward to remember how it all works and not in the least bit exciting or interesting. After much thought, I've got some idea of the reasons:
1. Perl
I don't want to start a language war here, and I'm still using Perl almost every day. But when a project gets over a certain size, it can get very messy, and it's very messy-looking too. Eric Raymond has a thoughtful essay about this, and this one is interesting too. Eric puts it better than I can:
"Larger project size seemed to magnify some of Perl's annoyances into serious, continuing problems. The syntax that had seemed merely eccentric at a hundred lines began to seem like a nigh-impenetrable hedge of thorns at a thousand. 'More than one way to do it' lent flavor and expressiveness at a small scale, but made it significantly harder to maintain consistent style across a wider code base. And many of the features that were later patched into Perl to address the complexity-control needs of bigger programs (objects, lexical scoping, 'use strict', etc.) had a fragile, jerry-rigged feel about them."
2. Object Orientation
I started to rewrite Squidalyser in object-oriented Perl in the belief that it would allow me to create a more modular, extensible and maintainable system with the additional benefit of easier readability. This is the hubris thing I mentioned above. OOP does nothing to write better code for you, and I'm not convinced it's more readable either. I think it would have worked better if I'd concentrated on creating smaller modules to assist my otherwise procedural code, so it was entirely my mistake.
3. Second-System Effect
When I started the rewrite, I was aware that there would be a "kitchen sink" tendency when it came to improvements and new ideas. There always is, but I was also aware of the first version's deficiencies, and wanted to improve it. More hubris - I should have stuck to a rewrite, followed by bug-fixing, then improvements. One step at a time, one idea in my head at a time. Anything else and I start to lose track.
4. Squidalyser's Success
Yes, it was (in a limited but gratifying way) a product which found a niche with some people. It was used on home, school, corporate and public sector networks, and so I started to receive emails requesting support and assistance. Most of these were polite and friendly and I was happy to help, but it takes time and energy. After a while, this grinds you down in a relentless way, sapping enthusiasm and taking time away from the difficult task of ongoing development. Same questions, same issues. I should have been more hard-nosed about it and said "no" more often. But I was delighted that people were finding it useful, and that kind of feedback is one of the things which motivates open source programmers.
5. Lack of Time
My personal circumstances became more complicated at the end of 2003. No messy divorce, no sudden death in the family - it could have been much worse. But I was forced to move to find work, I was staying away from home all week (no Internet access) and I simply had no time for recreational programming. Much of the time, I didn't even have access to my own computer. I fear this will always be a problem with a single-developer project, but at least the source code was available to anyone with an inclination to modify the project - and someone even did.
So, The Future
Seeing that article on Linux Box Admin a couple of weeks ago (I'd done a "vanity-search" for Squidalyser) reminded me that it hadn't been such a bad project after all, and I feel I am now in a good position to make a newer and (maybe) better version. I have more programming experience, using more languages, and certain things in web development have become more formalised, better-delineated - things like semantic HTML and CSS, for example. There's less groping around for a solution, more adapting others' tools and techniques to the problem at hand. I also have the experience of Squidalyser to inform my plans in developing a new version. And then, most of all, there's the Django framework, an idea whose time truly has come.