If you are looking to crawl your website for something like cache warming you can do so with wget quite easily.
The following wget command will crawl a site and leave nothing behind on the local filesystem afterward.
wget --mirror -q -e robots=off -p -r --delete-after -nd http://www.isleofmtv.com
|crawls the entire site
|prevents wget from writing output to the buffer
|ignores directives in robots.txt
|get all page assets
|recursively request pages
|clean up any local files after running wget
|prevents wget from writing a directory structure that gets left behind