If you are looking to crawl your website for something like cache warming you can do so with wget quite easily.
The following wget command will crawl a site and leave nothing behind on the local filesystem afterward.
wget --mirror -q -e robots=off -p -r --delete-after -nd http://www.isleofmtv.com
| –mirror | crawls the entire site |
| -q | prevents wget from writing output to the buffer |
| -e robots=off | ignores directives in robots.txt |
| -p | get all page assets |
| -r | recursively request pages |
| –delete-after | clean up any local files after running wget |
| -nd | prevents wget from writing a directory structure that gets left behind |