The last few days I have been working on a small scraper script. I’m going to Canada soon and I wanted an email alert to warn me when on-line Visa applications open. I wanted the script to email me a nice message to show me what’s changed on the page and it to log what’s going on for my piece of mind.

It took a few days, it ended up getting far too complicated. Instead of writing a little script in Perl I ended up writing this Lisp monster that uses a library to parse the HTML into a tree, code to search the tree which got pretty hairy!

It works though, but it’s so horrible I’m too ashamed to even post it here. I’m going to rewrite it using CL-PPCRE (searching for a block of text with regexp instead of parsing the HTML as a tree (hoping it works) and then having to deal with a tree).

It was a good learning exercise but a pretty clear example of KISS. Oh well…

- Dave.