If you are looking to crawl your website for something like cache warming you can do so with wget quite easily.
The following wget command will crawl a site and leave nothing behind on the local filesystem afterward.
wget --mirror -q -e robots=off -p -r --delete-after -nd http://www.isleofmtv.com
|–mirror||crawls the entire site|
|-q||prevents wget from writing output to the buffer|
|-e robots=off||ignores directives in robots.txt|
|-p||get all page assets|
|-r||recursively request pages|
|–delete-after||clean up any local files after running wget|
|-nd||prevents wget from writing a directory structure that gets left behind|