If you are looking to crawl your website for something like cache warming you can do so with wget quite easily. The following wget command will crawl a site and leave nothing behind on the local filesystem afterward. wget –mirror -q -e robots=off -p -r –delete-after -nd http://www.isleofmtv.com –mirror crawls the entire site -q prevents