Here’s a tool that you might not have heard of, SiteCrawler from Lighthead. I have used a lot of crawlers over the years, but not one that is as lightweight, fast, and that downloads an entire website for you as easy as this.
With SiteCrawler, basically you put in a domain, hit “Start” and it crawls and downloads the entire thing for you. Let’s take a look.
This is the first tab of options for SiteCrawler, here you can specify what to crawl, where you want things downloaded, whether or not you want things categorized by sub-directories, and how many downloads at a time you want.
On the restrictions tab you have control over time, size, and number of files per session, along with time and size per file.
The rules panel allows you complete control with the ability to input regular expressions and specific handling commands for crawling and downloading.
The advanced tab gives you the ability to zip downloads, run applescripts and shell commands, as well as specify specific user-agent crawlers.
The fifth and final colum shows progress as the crawl and download takes place. When it’s finished you have a complete copy of the website in exact duplicate directory structure. I downloaded all of XKCD. 
If you’re looking for a tool that has this type of functionality, and you use a mac (sorry PC), I recommend this one at an affordable $20. I’ve tried several others and this is by far the best I’ve found yet.
Did we forget something?
Let us know in the comments what tools you use!
Popularity: 26%
