If you are looking to crawl your website for something like cache warming you can do so with wget quite easily.

The following wget command will crawl a site and leave nothing behind on the local filesystem afterward.

wget --mirror -q -e robots=off -p -r --delete-after -nd http://www.isleofmtv.com
–mirror crawls the entire site
-q prevents wget from writing output to the buffer
-e robots=off ignores directives in robots.txt
-p get all page assets
-r recursively request pages
–delete-after clean up any local files after running wget
-nd prevents wget from writing a directory structure that gets left behind